Intelligent Customer Service Knowledge Base Quality Inspection Method, Its Device, Equipment, Medium, and Product

Abstract

The present application discloses an intelligent customer service knowledge base quality inspection method, its device, equipment, medium, and product. The method includes: obtaining probability distribution data in the knowledge base, which is used to store the classification probabilities between each similar question and each intention in the knowledge base; performing confidence learning based on the probability distribution data to obtain joint distribution data, which is used to store the corresponding counts of each similar question subordinate to each noise label in the knowledge base being mapped to each true label; for each noise label, determining the filtering quantity of each similar question subordinate to the noise label according to the count of the noise label being mapped to the true label in the joint distribution data; and selecting multiple similar questions with the lowest classification probabilities of each noise label subordinate from the probability distribution data according to the filtering quantity. The present application can implement quality inspection on the knowledge base in the intelligent customer service system, screen out the mapping relationship data between suspicious similar questions and intentions, so as to correct the relevant data.

Description

Technical Field

[0001] This application relates to the field of e-commerce information technology, and particularly to an intelligent customer service knowledge base quality inspection method, its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] All intelligent customer service systems need to be associated with a knowledge base. In the application scenarios of e-commerce services, especially cross-border e-commerce services based on independent websites, a knowledge base is generally constructed for each merchant. This knowledge base can be customized through a standard template and usually includes multiple intents, each intent is associated with multiple similar questions, and one similar question only belongs to one intent. After the knowledge base is put into use in the intelligent customer service system, when a consumer user enters the intelligent customer service system associated with a merchant instance and sends a question, the intelligent customer service system matches the similar question according to this question, further determines the corresponding intent, and finds the corresponding reply text according to this intent, and feeds it back to the user in the form of an instant message, playing the role of a customer service robot.

[0003] In order to reflect personalization and expand the intelligence of the system, the system also allows users to customize similar questions and the association relationship between similar questions and intents. As the data records in the knowledge base grow, there are more and more similar questions, and the mapping relationship between similar questions and intents becomes more and more complex. Coupled with the subjective understanding differences of words when merchants manually label these data, the mapping relationship between similar questions and intents is often incorrect, or although its correctness can be understood, there are still ambiguities, etc. For this situation, there is a lack of a "referee" mechanism in the prior art.

[0004] For the background of providing an intelligent customer service system, there are more and more merchant instances of the e-commerce platform supported by it, and the volume of the knowledge base corresponding to each merchant instance is also getting larger and larger, and the overall data volume grows exponentially. How to technically improve the intelligence level of the intelligent customer service system so that it is suitable for providing accurate and effective response services is a problem that needs to be solved at a higher level. If there are too many incorrect labels in each knowledge base, affecting the correct response effect, it will affect the quality perception of the intelligent customer service system itself.

[0005] On the other hand, for a large number of knowledge bases with a huge amount of mapping relationship data between similar questions and intents, the prior art often uses a pre-trained model to achieve intelligent response of the customer service system. If the "referee" mechanism problem is not solved and the incorrect label set is not found, for the customer service system, it may seriously weaken the ability learned by its model, or at least make the model more difficult to be trained to converge, resulting in an inflated training cost and implementation cost of the intelligent customer service system.

[0006] As can be seen, in the e-commerce platform, the accuracy of the data in the knowledge base, or rather, the expression quality of the mapping relationship data between similar questions and intents, is related to the overall quality and implementation cost of the entire intelligent customer service system. Therefore, there is a large room for improvement in the technical improvement of the intelligent customer service system. Summary of the Invention

[0007] The primary objective of this application is to solve at least one of the above problems and provide an intelligent customer service knowledge base quality inspection method, its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0008] To meet the various objectives of this application, the following technical solutions are adopted:

[0009] An intelligent customer service knowledge base quality inspection method provided to meet one of the objectives of this application includes the following steps:

[0010] Obtain the probability distribution data in the knowledge base of the merchant instance; the probability distribution data is used to store the classification probabilities between each similar question and each intent in the knowledge base.

[0011] Perform confidence learning based on the probability distribution data to obtain joint distribution data; the joint distribution data is used to store the corresponding counts of similar questions under each noise label in the knowledge base being mapped to various true labels, where the true label is the intent redirected by the maximum classification probability of the similar question in the probability distribution data, and the noise label is the originally marked intent of the redirected similar question.

[0012] For each noise label, determine the filtering quantity of similar questions under each noise label according to the count of the noise label mapping to the true label in the joint distribution data, excluding the count of the true label that points to the same intent as the noise label.

[0013] Select multiple similar questions with the lowest classification probabilities under each noise label from the probability distribution data according to the filtering quantity to form the mislabeled set of similar questions.

[0014] In an extended embodiment, before obtaining the probability distribution data in the knowledge base of the merchant instance, the following steps are included:

[0015] Start multiple parallel training tasks, and each training task uses the knowledge base of a corresponding merchant instance as the data set to jointly train the intent classification model.

[0016] After each training task is started, initialize its corresponding intent classification model instance, and set the classification number of the intent classification model instance according to the total number of intents in the knowledge base of its corresponding merchant instance.

[0017] For each training task, the cross-validation method is applied to alternately divide the dataset of the corresponding merchant instances into a training set and a validation set, and the intent classification model is trained until it converges;

[0018] The intent classification model trained to convergence is used to classify each similar question from the knowledge bases of each merchant instance, and the classification probabilities obtained from the classification are constructed into the probability distribution data of the knowledge base corresponding to this training task.

[0019] In a further embodiment, during the process of training the intent classification model, the following steps are executed:

[0020] The text feature vectors of the corresponding training samples are extracted in each training task, and the training samples are the similar questions in the training set;

[0021] Classification is performed according to the text feature information to adapt to each training task, and the classification probabilities of mapping each text feature vector to each set classification are determined;

[0022] Adapting to each training task, using the intent corresponding to the similar question in the training set as the supervision label, calculating the loss function value corresponding to each intent classification model instance, and performing gradient update on this intent classification model instance with this loss function value.

[0023] In a further embodiment, confidence learning is performed according to the probability distribution data to obtain joint distribution data, including the following steps:

[0024] The probability distribution data is converted into a probability distribution matrix, where each element represents the classification probability that the similar question pointed to by its row coordinate is mapped to the intent pointed to by its column coordinate;

[0025] According to this probability distribution matrix, the confidence threshold for each intent is calculated, and the confidence threshold is the average classification probability that the similar questions in the corresponding intent are predicted as this intent;

[0026] The intent with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix is determined as the true label, and the originally marked intent of the similar question corresponding to this true label is determined as the noise label;

[0027] The data distributions of the noise label and the true label are counted to obtain a count matrix, where each element represents the total number of similar questions with the true label being the intent pointed to by the column coordinate of this element among all similar questions corresponding to the intent of the noise label pointed to by its row coordinate;

[0028] Convert the counting matrix into joint distribution data, such that the joint distribution data corresponds to the scale of the total number of similar questions in the corresponding knowledge base, and store the corresponding counts of the similar questions subordinate to each noise label in the knowledge base mapped to each true label.

[0029] In a specific embodiment, converting the counting matrix into joint distribution data includes the following steps:

[0030] Scale the counting matrix to obtain a scaled intermediate matrix, such that the count of each noise label is based on the total number of similar questions subordinate to that noise label in the knowledge base;

[0031] Normalize the scaled intermediate matrix to obtain a joint distribution matrix as the joint distribution data.

[0032] In an extended embodiment, according to the filtering quantity, select multiple similar questions with the lowest classification probability for each noise label from the probability distribution data to form a mislabeled set of the similar questions. Then, the following steps are included:

[0033] In response to the error correction start request of a merchant instance, push its corresponding question dataset to it, where the question dataset includes each similar question in the mislabeled set and its corresponding noise label;

[0034] Obtain the correction data of the question dataset submitted by the merchant instance, where the correction data includes the mapping relationship data between the similar questions in the question dataset and their corrected intents;

[0035] Modify the corresponding relationship between the similar questions and intents in the knowledge base of the merchant instance according to the mapping relationship data.

[0036] An intelligent customer service knowledge base quality inspection device provided to meet one of the purposes of the present application includes: a probability acquisition module, a distribution learning module, a filtering calculation module, and a set selection module, where:

[0037] The probability acquisition module is used to obtain the probability distribution data in the knowledge base of a merchant instance; the probability distribution data is used to store the classification probabilities between each similar question and each intent in the knowledge base;

[0038] The distribution learning module is used to perform confidence learning according to the probability distribution data to learn joint distribution data; the joint distribution data is used to store the corresponding counts of the similar questions subordinate to each noise label in the knowledge base mapped to each true label, the true label is the intent redirected by the maximum classification probability of the similar question in the probability distribution data, and the noise label is the original marked intent of the redirected similar question;

[0039] A filtering calculation module, which is used to correspond to each noise label, and determine the filtering quantity of similar questions under each noise label according to the count of the noise label mapped to the true label in the joint distribution data, excluding the count of the true label pointing to the same intention as the noise label;

[0040] A set selection module, which is used to select multiple similar questions with the lowest classification probability under each noise label from the probability distribution data according to the filtering quantity, and form an incorrect annotation set of the similar questions.

[0041] In an extended embodiment, the intelligent customer service knowledge base quality inspection device of the present application further includes the following modules that run in advance:

[0042] A task start module, which is used to start multiple parallel training tasks. Each training task uses the knowledge base of a corresponding merchant instance as a data set to jointly train an intention classification model;

[0043] An initialization module, which is configured to initialize the corresponding intention classification model instance after each training task starts, and set the number of classifications of the intention classification model instance according to the total number of intentions in the knowledge base of its corresponding merchant instance;

[0044] A model training module, which is configured to apply the cross-validation method to alternately divide the data set of its corresponding merchant instance into a training set and a validation set for each training task, and train the intention classification model until it converges;

[0045] A data generation module, which is used to classify each similar question from the knowledge bases of each merchant instance by using the intention classification model trained to convergence, and construct the classification probability obtained by the classification into the probability distribution data of the knowledge base corresponding to the training task.

[0046] In a further embodiment, the intention classification model is configured to implement the following functions during its training process: extract the text feature vectors of the corresponding training samples in each training task, where the training samples are similar questions in the training set; classify according to the text feature information to adapt to each training task, and determine the classification probability of each text feature vector mapped to each set classification; adapt to each training task, calculate the loss function value corresponding to each intention classification model instance with the intention corresponding to the similar question in the training set as the supervision label, and perform gradient update on the intention classification model instance with the loss function value.

[0047] In a deepened embodiment, the distribution learning module includes:

[0048] A data representation sub-module, configured to convert the probability distribution data into a probability distribution matrix, where each element represents the classification probability that the similar question pointed to by its row coordinate is mapped to the intent pointed to by its column coordinate;

[0049] A threshold determination sub-module, configured to calculate and determine the confidence threshold for each intent according to the probability distribution matrix, where the confidence threshold is the average classification probability that the similar questions in the corresponding intent are predicted to be this intent;

[0050] A label determination sub-module, configured to determine the intent with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix as the true label, and determine the intent of the original label corresponding to the similar question of this true label as the noise label;

[0051] A distribution counting sub-module, configured to count the data distributions of the noise label and the true label to obtain a counting matrix, where each element represents the total number of similar questions corresponding to the noise label pointed to by its row coordinate and the true label is the intent pointed to by the column coordinate of this element;

[0052] A counting conversion sub-module, configured to convert the counting matrix into joint distribution data, so that the joint distribution data corresponds to the scale of the total number of similar questions in the corresponding knowledge base, and store the corresponding counts of the similar questions under each noise label in the knowledge base being mapped to each true label.

[0053] In a specific embodiment, the counting conversion sub-module includes:

[0054] A scaling module, configured to scale the counting matrix to obtain a scaled intermediate matrix, so that the counts of each noise label are all based on the total number of similar questions under this noise label in the knowledge base;

[0055] A data normalization module, configured to normalize the scaled intermediate matrix to obtain a joint distribution matrix as the joint distribution data.

[0056] In an extended embodiment, the intelligent customer service knowledge base quality inspection device of the present application further includes:

[0057] A mislabeled push module, configured to push the corresponding question data set to a merchant instance in response to an error correction start request of the merchant instance, where the question data set includes each similar question in the error annotation set and its corresponding noise label;

[0058] An update acquisition module, configured to acquire the correction data of the question data set submitted by the merchant instance, where the correction data includes the mapping relationship data between the similar questions in the question data set and their corrected intents;

[0059] An update and correction module is used to modify the corresponding relationship between similar questions and intents in the knowledge base of the merchant instance according to the mapping relationship data.

[0060] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory. The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the intelligent customer service knowledge base quality inspection method described in the present application.

[0061] A computer-readable storage medium provided to meet another purpose of the present application stores a computer program implemented according to the intelligent customer service knowledge base quality inspection method in the form of computer-readable instructions. When the computer program is called and run by a computer, it executes the steps included in the method.

[0062] A computer program product provided to meet another purpose of the present application includes a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in any embodiment of the present application.

[0063] Compared with the prior art, the advantages of the present application are as follows:

[0064] First, the present application uses the probability distribution data corresponding to the knowledge base of the merchant instance for confidence learning to obtain a joint distribution matrix representing the count of similar questions under the noise labels in the knowledge base being mapped to each true label. The true label usually refers to the intent corresponding to the maximum classification probability presented by each similar question after classification in the probability distribution data, and the noise label is the originally marked intent of the similar question. Then, using the counts in the joint distribution matrix, the filtering amount regarded as mislabeled is calculated for each similar question corresponding to the noise label. According to the filtering amount, multiple similar questions with relatively low classification probabilities corresponding to each noise label are determined, thereby constructing a corresponding mislabeled set in the knowledge base of the merchant instance, realizing a "referee" mechanism, using big data in the e-commerce platform to conduct quality inspection and investigation of mislabeled in the knowledge base of the merchant instance, laying a foundation for subsequent improvement of labeling, helping to improve the accuracy of the mapping relationship between data in the knowledge base, so as to improve the intelligence level of the intelligent customer service system.

[0065] Furthermore, for the back-end of an e-commerce platform that centrally maintains an intelligent customer service system, identifying the mapping relationship between similar questions and intents with errors based on probability distribution data is essentially to detect error data according to the classification probabilities of each similar question. As for the classification probabilities of each similar question mapped to each intent, they can be obtained incidentally in the relevant model for intent classification used for classifying them. Therefore, the back-end can centrally and efficiently identify error labels in its large number of merchant instance knowledge bases, quickly and timely detect the error labels in each knowledge base on the platform with a relatively low computational workload, and greatly improve the back-end maintenance efficiency at a relatively low implementation cost. Similarly, after the error labels are identified, the correctly labeled data is more suitable for training the aforementioned model, enabling the aforementioned model to be trained more efficiently until convergence.

[0066] Subsequently, after identifying the error label set, the intelligent customer service system obtains more accurate data. These data can improve the intelligence level of the system during the model training stage, and after the system is put into use, it can also more accurately identify the intent corresponding to the user's question, so as to accurately match the response text and improve the accuracy of the response, thereby comprehensively improving the user experience of the intelligent customer service system. Brief Description of the Drawings

[0067] The above and / or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0068] Figure 1 is a schematic flowchart of a typical embodiment of the intelligent customer service knowledge base quality inspection method of the present application;

[0069] Figure 2 is a schematic flowchart of the multi-task joint training process of the intent classification model in the embodiment of the present application;

[0070] Figure 3 is a principle block diagram of the multi-task training logic implementation architecture in the embodiment of the present application;

[0071] Figure 4 is a schematic flowchart of the working process of a single training task in the embodiment of the present application;

[0072] Figure 5 is a schematic structural diagram of the intent classification model in the case of a single instance of the present application;

[0073] Figure 6 is a schematic flowchart of the confidence learning process in the embodiment of the present application;

[0074] Figure 7 is a schematic flowchart of the process of correcting error labels in the embodiment of the present application;

[0075] Figure 8It is a schematic block diagram of the intelligent customer service knowledge base quality inspection device of the present application;

[0076] Figure 9 It is a schematic structural diagram of a computer device adopted by the present application. Specific implementation manners

[0077] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and should not be construed as a limitation to the present application.

[0078] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0079] Those skilled in the art of the present technology can understand that unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0080] Those skilled in the art can understand that the "client", "terminal", and "terminal device" used herein include both devices with a wireless signal receiver that only has the ability to receive without transmitting, and devices with receiving and transmitting hardware that have the receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, etc., which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service), which can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; conventional laptop and / or palm-held computers or other devices, which are conventional laptop and / or palm-held computers or other devices with and / or including a radio frequency receiver. The "client", "terminal", and "terminal device" used herein can be portable, transportable, installed in a vehicle (air, sea, and / or land), or suitable for and / or configured to run locally, and / or run in a distributed manner at any other location on the earth and / or in space. The "client", "terminal", and "terminal device" used herein can also be a communication terminal, an Internet access terminal, a music / video playback terminal, such as a PDA, MID (Mobile Internet Device), and / or a mobile phone with music / video playback function, or can also be a smart TV, a set-top box, etc.

[0081] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer, which is a hardware device with the necessary components disclosed by the von Neumann principle, including a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input / output devices to complete specific functions.

[0082] It should be noted that the concept of "server" in this application can similarly be extended to the case of server clusters. According to the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can either be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility and should not be restricted by this when implementing the network deployment method of this application.

[0083] One or several technical features of this application, unless expressly specified, can either be deployed on the server and accessed by the client remotely calling the online service interface provided by the server, or directly deployed and run on the client for access.

[0084] The neural network models cited or possibly cited in this application, unless expressly specified, can either be deployed on a remote server and remotely called by the client, or deployed on a client capable of handling the device for direct calling. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid over-occupying the client's hardware operating resources.

[0085] All kinds of data involved in this application, unless expressly specified, can either be remotely stored on the server or stored on the local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0086] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus show commonality with each other, unless otherwise specified, these methods can be executed independently. Similarly, for the various embodiments disclosed in this application, they are all proposed based on the same inventive concept. Therefore, for the same concept expressed, as well as the concepts that are only appropriately transformed for convenience although the concept expressions are different, they should be equivalently understood.

[0087] For the various embodiments to be disclosed in this application, unless expressly pointed out that there is a mutually exclusive relationship between them, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct new embodiments, as long as this combination does not deviate from the creative spirit of this application and can meet the needs in the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0088] An intelligent customer service knowledge base quality inspection method of this application can be programmed into a computer program product and deployed to run in the server. Thus, the client can access the interface opened after the computer program product runs in the form of a web program or an application program, and achieve human-computer interaction with the process of the computer program product through the graphical user interface.

[0089] An application scenario of the present application is an intelligent customer service system in an e-commerce platform. The intelligent customer service system matches a knowledge base for each merchant instance. Each knowledge base includes intents and response texts, and each intent corresponds to multiple similar questions. When a user configures their knowledge base, they can either reference a template provided by the system or customize the mapping relationship data between the similar questions and intents. The management user of the merchant instance is responsible for defining, updating, and controlling the on / off of the knowledge base. After a user accessing the merchant instance enters their store and the consultation interface of the intelligent customer service system, they can input a question message into the consultation interface. Then, the intelligent customer service system matches the intent based on the question message and replies to the user with the response text corresponding to the intent in the form of an instant messaging message. The intelligent customer service knowledge base quality inspection method of the present application is suitable for processing the data in the knowledge base used by such an intelligent customer service system, referring to big data to check the accuracy of the mapping relationship between the similar questions and intents, so as to optimize the data in the knowledge base and further improve the intelligence level of the intelligent customer service system.

[0090] Please refer to Figure 1 , in a typical embodiment of the intelligent customer service knowledge base quality inspection method of the present application, it includes the following steps:

[0091] Step S2100: Obtain the probability distribution data in the knowledge base of the merchant instance; the probability distribution data is used to store the classification probabilities between each similar question and each intent in the knowledge base:

[0092] For the e-commerce platform of the present application, the background server can perform centralized statistical processing on the knowledge bases of each merchant instance to obtain the probability distribution data corresponding to the knowledge base of each merchant instance.

[0093] The probability distribution data can be represented in various structural forms. For example, it is stored in an array structure in the form of a matrix for convenient and efficient operation. The probability distribution data of a merchant instance is used to store the classification probabilities between each similar question and all the intents mapped to the knowledge base of the merchant instance, that is, to store one by one the classification probabilities of each similar question being classified into any one of the intents.

[0094] It is not difficult to understand that each similar question in the knowledge base has been originally marked with the intention to which it should be classified. Whether this intention is accurate can be determined by centrally statistically processing the knowledge base. Correspondingly, a classification probability corresponding to each intention to which a similar question is mapped to a merchant instance can be calculated. By comparing the classification probabilities corresponding to the mapped intentions, it can be known whether the originally marked intention of the similar question is correct, playing the role of a "referee". Usually, the intention with the largest classification probability is the intention that highly probably matches the similar question. Thus, it can be understood that according to the probability distribution data, each similar question can be redirected from its originally marked intention to the intention corresponding to the maximum classification probability. In this application, the originally marked intention of the similar question is defined as a noise label, and the intention to which the similar question will be redirected is defined as a true label. These concepts will be referred to later.

[0095] Step S2200: Perform confidence learning based on the probability distribution data to obtain joint distribution data; the joint distribution data is used to store the corresponding counts of similar questions subordinate to each noise label in the knowledge base being mapped to each true label, where the true label is the intention redirected by the maximum classification probability of the similar question in the probability distribution data, and the noise label is the originally marked intention of the redirected similar question:

[0096] There is rich classification information in the probability distribution data. Based on this classification information, by applying the principle of confidence learning, data mining can be achieved, thereby obtaining joint distribution data that reflects the association information between the noise label and the true label.

[0097] The joint distribution data is statistically obtained based on the probability distribution data and is used to represent the corresponding statistical times when the similar question set subordinate to each noise label is mapped to each true label. For example, there are 100 similar questions originally marked with intention A in the knowledge base. Among them, 40 similar questions obtain the maximum classification probability in other intentions outside intention A. Among them, 10 similar questions obtain the maximum classification probability in intention B, and another 30 similar questions obtain the maximum classification probability in intention C. In this case, these 40 rewritten similar questions, whose originally marked intention A, should be regarded as a noise label, or an incorrect label, and the specific intentions pointed to by the maximum classification probability of their corresponding redirection, namely intention B and intention C, are the true labels for these 40 redirected similar questions. The joint distribution data is the count statistically obtained for each such noise label, and essentially, it statistically counts the number of times the noise label is rewritten. In this example, since 40 similar questions in the noise label (intention A) are redirected to two true labels (intention B and intention C), accordingly, the counts of the noise label (intention A) corresponding to the two true labels (intention B and intention C) are 10 and 30 respectively.

[0098] Of course, since there may be cases where some similar questions may not be redirected, not all similar questions will obtain true labels. Therefore, the counts obtained through statistics are preferably unified to the same statistical sample scale, so that each count is characterized based on the same scale to achieve normalization. Regarding this, subsequent embodiments of this application will further disclose this, and this example is temporarily omitted.

[0099] Thus, the joint distribution data obtains the corresponding counts of similar questions under each noisy label being mapped to each true label. For example, for a noisy label, among all the similar questions labeled with this noisy label (intention A), the counts of being redirected to each true label (such as intention B and intention C) are determined. It represents how many mislabelings exist among the similar questions labeled with this noisy label (intention A) in the knowledge base. Specifically, corresponding breakdown counts are given for each true label (such as intention B and intention C).

[0100] Step S2300: For each noisy label, determine the filtering quantity of similar questions under each noisy label according to the count of this noisy label mapped to true labels in the joint distribution data, excluding the count of true labels that point to the same intention as this noisy label:

[0101] Based on the joint distribution data, it is already known the corresponding counts of similar questions under each noisy label that may be redirected to each true label, that is, the total number of mislabelings existing among the similar questions under each noisy label is known. Based on this, the filtering quantity of the set of similar questions under each noisy label can be calculated according to this count. In this calculation process, the situation where the noisy label overlaps with the true label should be excluded. For example, among the 100 similar questions of intention A, the remaining 60 similar questions are not redirected, and their true labels still point to intention A itself, so they should not be included.

[0102] It is not difficult to understand that through this calculation, the total quantity of similar questions with highly probable mislabelings corresponding to each noisy label, that is, each intention, can be obtained, which is the so-called filtering quantity.

[0103] Step S2400: Select multiple similar questions with the lowest classification probabilities under each noisy label from the probability distribution data to form the mislabeling set of the similar questions:

[0104] Re - call the probability distribution data mentioned above. As described before, the probability distribution data stores the classification probabilities of each similar question being redirected to all intents. Among them, for an intent, the lower the classification probability of the similar questions under it, the more likely it is a similar question that has been redirected. Selecting these similar questions constitutes the mislabeled set. Thus, by querying the classification probabilities of all similar questions under the intent corresponding to the noise label in the probability distribution data, the screening of similar questions under the noise label can be realized, and the preparation of the mislabeled set can be completed.

[0105] The mislabeled set can be constructed by marking similar questions in the knowledge base, or an error annotation list pointing to similar questions for each mislabel can be independently constructed for future use. In this regard, those skilled in the art can implement it flexibly.

[0106] It can be seen from this typical embodiment that this application discloses a process for quality - detecting the mapping relationship between similar questions and intents in the knowledge base of an intelligent customer service system, which can realize the quality screening of the Q&A knowledge in the knowledge base and lay a foundation for improving the quality of the knowledge base. More specifically, the implementation of this application can at least achieve the following positive effects:

[0107] First, this application uses the probability distribution data corresponding to the knowledge base of the merchant instance for confidence learning to obtain a joint distribution matrix representing the count of similar questions under the noise label in the knowledge base being mapped to each true label. The true label usually refers to the intent corresponding to the maximum classification probability presented after each similar question in the probability distribution data is classified, and the noise label is the originally marked intent of this similar question. Then, using the counts in the joint distribution matrix, calculate the filtering amount regarded as mislabeled among the similar questions corresponding to each noise label, and determine multiple similar questions with relatively low classification probabilities corresponding to each noise label according to the filtering amount. Thus, construct the corresponding mislabeled set in the merchant instance's knowledge base, realizing a "referee" mechanism, using big data in the e - commerce platform to conduct quality inspection and screening of mislabeled knowledge bases of merchant instances, laying a foundation for subsequent improvement of annotations, helping to improve the accuracy of the mapping relationship between data in the knowledge base, and thus improving the intelligence level of the intelligent customer service system.

[0108] Furthermore, for the back-end of an e-commerce platform that centrally maintains an intelligent customer service system, identifying the mapping relationship between similar questions and intents with errors based on probability distribution data is essentially to detect incorrect data according to the classification probabilities of each similar question. As for the classification probabilities of each similar question mapped to each intent, they can be obtained incidentally in the relevant model for intent classification used for classifying them. Therefore, the back-end can centrally and efficiently identify incorrect labels for its large number of merchant instance knowledge bases, quickly and timely detect incorrect labels in each knowledge base of the platform with a relatively low computational workload, and greatly improve the back-end maintenance efficiency at a relatively low implementation cost. Similarly, after the incorrect labels are identified, the correctly labeled data is more suitable for training the said model, enabling the said model to be trained more efficiently until convergence.

[0109] Subsequently, after identifying the incorrect label set, the intelligent customer service system obtains more accurate data. These data can improve the intelligence level of the system during the model training stage, and after the system is put into use, it can also more accurately identify the intent corresponding to the user's question, so as to accurately match the response text and improve the accuracy of the response, thereby comprehensively improving the user experience of the intelligent customer service system.

[0110] The probability distribution data of the present application can be obtained through an intent classification model. After training the intent classification model, this model can be used to obtain the probability distribution data corresponding to each knowledge base. For this purpose, please refer to Figure 2 the process shown in Figure 3 and the training task logic implementation architecture shown in

[0111] In an extended embodiment, before step S2100, obtaining the probability distribution data in the knowledge base of the merchant instance, the following steps are included:

[0112] Step S1100: Start multiple parallel training tasks. Each training task uses the knowledge base of a corresponding merchant instance as the data set to jointly train the intent classification model:

[0113] Therefore, in this embodiment, the idea of multi-task training is applied to adapt to the knowledge base of each merchant instance, and a corresponding intent classification model instance is provided. Through this instance, the text feature information of the training samples in a knowledge base is represented and learned. On this basis, a classifier is used for classification, so as to realize the joint training of the intent classification model in a multi-task manner. In such an idea, each training task shares the underlying structure and parameters of the intent classification model. Subsequently, each merchant instance calls this intent classification model to implement its intelligent customer service system, which is equivalent to deriving a logically relatively independent intelligent customer service instance. This intelligent customer service instance has learned the association information contained in the mapping relationship between similar questions and intents in its knowledge base and can effectively answer user questions based on the knowledge in this knowledge base.

[0114] When the training tasks corresponding to multiple merchant instances are started, each training task uses the knowledge base of its corresponding merchant instance as the data set required for training, so as to realize the representation learning of the association information between similar questions and intents therein for classification.

[0115] Step S1200: After each training task is started, initialize its corresponding intent classification model instance, and set the number of classifications of this intent classification model instance according to the total number of intents in the knowledge base of its corresponding merchant instance:

[0116] After each training task is started, it can start to initialize its intent classification model instance, mainly by initializing its classification layer. Specifically, according to the knowledge base of the merchant instance corresponding to this training task, determine the total number of intents in this knowledge base, and set it as the total number of classifications N of the classifier of the model, so that the model finally inputs the classification probabilities corresponding to N categories.

[0117] Step S1300: Each training task applies the cross-validation method to alternately divide the data set of its corresponding merchant instance into a training set and a validation set, and train the intent classification model until it converges:

[0118] When each training task inputs the data set of its corresponding merchant instance into training, it can apply the K-fold cross-validation method to randomly divide this data set into K parts, and each time select a fixed number of parts as the validation set, and the remaining parts as the training set. Preferably, the leave-one-out method is adopted, that is, each time select 1 part as the validation set, and the remaining K - 1 parts as the training set. On the basis of dividing the training set and the validation set, alternately replace different shares as the validation set to complete K batches of training. It can be understood that in the data set, each similar question is a training sample, and the intent corresponding to this training sample constitutes the supervision label.

[0119] Suppose there are M merchants, and each merchant has (N1, N2,..., NM ) intentions, then for the shared underlying parameters, we can obtain updates, where C ij represents the number of similar questions included in the j-th intention of the i-th merchant. For the classification layer parameters of each merchant, we can obtain updates. In this way, the parameters of the shared text feature extraction model in the model can be fully trained to play the role of feature extraction. Finally, only through a simple classification layer can we obtain the prediction result.

[0120] It can be seen from this that by applying the K-fold cross-validation method for training, aiming at the characteristic of a small total sample size in the merchant instance knowledge base, in the case of a small sample size, the training efficiency can be improved by reusing the training samples, enabling the model to converge faster.

[0121] Step S1400: Use the intention classification model trained to convergence to classify each similar question from the knowledge bases of each merchant instance, and construct the classification probabilities obtained from the classification as the probability distribution data of the knowledge base corresponding to this training task:

[0122] After the above training process, the intention classification model can be trained to a convergent state. Therefore, this intention classification model can be put into use in the intelligent customer service system. By calling this intention classification model to extract features and classify the knowledge base of each merchant instance, the classification probabilities of each similar question in this knowledge base corresponding to each intention in this knowledge base can be obtained, and these classification probabilities constitute the aforementioned probability distribution data.

[0123] In this embodiment, aiming at the situation where the knowledge base sample size of each merchant instance is small and the distribution is discrete, and there is a large amount of knowledge base training in the background, the K-fold cross-validation method is used to enable the multi-task mechanism to classify the intention classification model, which is more likely to make the intention classification model converge quickly, with low training cost and high training efficiency.

[0124] Please refer to Figure 4 , in a further embodiment, during the process of training the intention classification model, the following steps are executed:

[0125] Step S3100: Extract the text feature vectors of the corresponding training samples in each training task, where the training samples are the similar questions in the training set:

[0126] Figure 5It is the structural schematic diagram of a single intent classification model instance. It can be understood that when each training task is implemented for training, the training is carried out according to the structure schematic diagram shown. This model instance includes a text feature extraction model and a classifier. The text feature extraction model can be implemented by using Bert or a similar neural network model suitable for extracting text feature information. The classifier can be a multi-classifier based on the Softmax function.

[0127] Each training task extracts a training sample from the training set through each such instance, performs text feature extraction on it, and obtains the corresponding text feature vector, also known as text feature information. As mentioned above, the training sample is a similar question in the training set.

[0128] Step S3200: Classify according to the text feature information to adapt to each training task, and determine the classification probabilities of each text feature vector mapped to its set various classifications:

[0129] After obtaining the text feature information, the model corresponds to each instance and performs classification through a classifier that has been initialized with the total number of classifications N set by the corresponding training task, generating the classification probabilities corresponding to each classification of the training sample mapped.

[0130] Step S3300: Adapt to each training task, calculate the loss function value corresponding to each intent classification model instance with the intent corresponding to the similar question in the training set as the supervision label, and perform gradient update on this intent classification model instance with this loss function value:

[0131] Each intent classification model instance, in its corresponding training task, uses the intent corresponding to the training sample as the supervision label to verify the classification result of the classifier, calculates the relative Poisson loss function value according to this supervision label, and performs gradient update on this intent classification model instance with this loss function value.

[0132] It can be understood that through multiple iterative trainings by the training tasks, the multiple tasks jointly update the parameters of the text feature extraction model, and the intent classification model is easily trained to convergence. Although the amount of data in the knowledge base of each merchant instance itself is not large, with the help of the multi-task sharing training mechanism, the advantage of the coexistence of a large number of knowledge bases on the e-commerce platform can be utilized to enable the model to learn the representation learning ability. Therefore, for the e-commerce platform providing the background support service for the intelligent customer service system, the progressiveness of this embodiment is particularly significant.

[0133] Please refer to Figure 6, in the in-depth embodiment, based on cross-validation, probability distribution data of all similar questions is obtained, and confidence learning can be carried out on the knowledge base of each merchant one by one to assist the merchant in discovering incorrect labels. Taking a certain merchant as an example, assume that the knowledge base held by the merchant has m intents, and the total number of similar questions as samples is n. In this regard, the step S2200, performing confidence learning according to the probability distribution data to obtain joint distribution data, includes the following steps:

[0134] Step S2210, converting the probability distribution data into a probability distribution matrix, where each element represents the classification probability that the similar question pointed to by its row coordinate is mapped to the intent pointed to by its column coordinate:

[0135] In this embodiment, to facilitate improving the operation efficiency, the probability distribution data is constructed into a vector matrix, that is, a probability distribution matrix, denoted as matrix P, where P[i][j] represents the probability that the i-th similar question is predicted to be the j-th intent. That is: each row vector in the matrix is used to represent the classification probability that a similar question in the knowledge base of a merchant instance is mapped to each intent in the knowledge base, that is, the classification probability obtained by classifying a similar question is constructed into a row vector. Thus, each column vector is used to represent the corresponding classification probability that an intent in the knowledge base of a merchant instance is mapped to each similar question in the knowledge base. Therefore, in the probability distribution matrix corresponding to the probability distribution data, each element represents the classification probability that the similar question corresponding to its row coordinate is mapped to the intent corresponding to its column coordinate.

[0136] Step S2220, calculating and determining the confidence threshold for each intent, where the confidence threshold is the average classification probability that the similar questions in the corresponding intent are predicted to be this intent:

[0137] To determine the confidence threshold, calculate the confidence threshold for each intent through the following formula: where D j represents the set of similar questions in the j-th intent, and the meaning of this threshold is the average classification probability that all similar questions in this intent are predicted to be this intent.

[0138] Step S2230, determining the intent with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix as the true label, and determining the intent with the original label of the similar question corresponding to this true label as the noise label:

[0139] The confidence threshold is used to determine whether the classification probability of the intent mapped by a similar question constitutes a sufficiently credible intent redirection result. Only the intent with a classification probability greater than this confidence threshold constitutes the redirection intent of the corresponding similar question, that is, the true label. Correspondingly, the intent with the original label of this similar question constitutes the noise label in this case.

[0140] In this case, the tags originally labeled for similar questions (i.e., the intentions labeled by merchants) are regarded as "noisy tags", denoted as The tag (intention) with the highest classification probability predicted by the model is regarded as the true tag The estimation of, that is And it is required that P[i][j]>t[j], that is, the classification probability is greater than the confidence threshold. Therefore, it can be seen that not all samples have true tags. When necessary, the processing of other embodiments of this application can be combined to unify the statistical scale.

[0141] Step S2240: Count the data distributions of the noisy tags and the true tags to obtain a count matrix, where each element represents the total number of all similar questions with the intention corresponding to the noisy tag pointed to by its row coordinate and the true tag being the intention pointed to by the column coordinate of this element:

[0142] For Count the distribution of to obtain a count matrix For example, represents the number of similar questions with the true tag being the k-th intention among all similar questions with the intention corresponding to the j-th intention (noisy tag) originally labeled. Thus, for the count matrix, each element represents the total number of all similar questions with the intention corresponding to the noisy tag pointed to by its row coordinate and the true tag being the intention pointed to by the column coordinate of this element.

[0143] Step S2250: Convert the count matrix into joint distribution data, so that the joint distribution data corresponds to the scale of the total number of similar questions in the corresponding knowledge base, and store the corresponding counts of each noisy tag in the knowledge base where the subordinate similar questions are mapped to each true tag.

[0144] In a variant embodiment, step S2250 can be decomposed into the following steps for execution:

[0145] Step S2251: Scale the count matrix to obtain a scaled intermediate matrix, so that the count of each noisy tag is based on the total number of similar questions subordinate to this noisy tag in the knowledge base:

[0146] As can be seen from step S2240, not all samples have true tags, and the count matrix only counts the distributions of the noisy tags and the true tags. Therefore, the statistical values can be further scaled so that the total count of each noisy tag is equal to the total number of actual samples. Denote as the total number of samples of the noisy tag Then the scaling formula is as follows:

[0147]

[0148] In step S2252, the scaled intermediate matrix is normalized to obtain a joint distribution matrix as the joint distribution data:

[0149] After the above processing and then normalization, the joint distribution data of the noise labels and the true labels can be obtained, denoted as a matrix:

[0150]

[0151] It can be seen that in this embodiment, by revealing the detailed process of confidence learning, the probability distribution matrix is converted into a joint distribution matrix, and relevant mislabelings can be retrieved more quickly and efficiently through the joint distribution matrix. For example, in specific applications, in combination with the embodiment of step S1300 herein, the filtering amount corresponding to each intent can be determined based on this joint distribution matrix:

[0152] There are various different methods that can be adopted for filtering mislabels based on the joint distribution matrix. In this application, the Prune by Class (PBC) method in confidence learning is taken as an example for illustration. Specifically, for each intent i ∈ 1, 2, …, m, the number of samples that need to be filtered is calculated as The operation result of this formula is the filtering amount of the samples corresponding to this intent.

[0153] Furthermore, in step S1400, for each intent, similar questions corresponding to its filtering amount are selected. The selection method is to select, under the constraint of , the samples with the lowest prediction probability. To understand vividly, that is, from each column vector corresponding to an intent in the probability distribution matrix, multiple similar questions corresponding to the elements with the lowest classification probability are selected, and the number of selected elements is determined by the filtering amount corresponding to this intent. These selected similar questions are the mislabeled similar questions.

[0154] Thus, all mislabeled similar questions corresponding to each intent are selected to form a mislabeled set, and this mislabeled set can be pushed to the user for correction so as to make the labeling more accurate, obtain the result after modified labeling, and be used for the secondary training of the intent classification model to improve the response accuracy of the intelligent customer service system.

[0155] Please refer to Figure 7 , in the extended embodiment, in step S2400, according to the filtering amount, multiple similar questions with the lowest classification probability under each noise label are selected from the probability distribution data to form the mislabeled set of the similar questions. After that, the following steps are included:

[0156] Step S4100: In response to the error correction start request of the merchant instance, push the corresponding problem data set to it. The problem data set includes each similar question in the error annotation set and its corresponding noise label:

[0157] For the management user corresponding to the merchant instance, when entering the knowledge base maintenance page of this merchant instance, an error correction start request can be triggered. The background server responds to this request and can push the problem data set screened for the knowledge base of this merchant instance to this management user for editing. The problem data set is formatted and generated based on the error annotation set, which includes multiple similar questions extracted from the error annotation set and their corresponding noise labels, that is, the originally marked intent. Further, the intent with the maximum classification probability mapped by this similar question can also be obtained by querying the probability distribution matrix to provide it to the management user for reference to achieve the purpose of assisting in error correction.

[0158] Step S4200: Obtain the correction data of the problem data set submitted by this merchant instance. The correction data includes the mapping relationship data between the similar questions in the problem data set and their corrected intents:

[0159] After the management user of the merchant instance obtains the problem data set, the mark of the similar question in it can be modified to remap it to a new intent, such as the intent corresponding to the maximum classification probability recommended by the background server, and then the correction data is submitted to the server. It can be understood that the correction data includes the mapping relationship data between the similar questions in the problem data set and their corrected intents.

[0160] Step S4300: Modify the corresponding relationship between the similar questions and intents in the knowledge base of the merchant instance according to the mapping relationship data:

[0161] After the server obtains the correction data of the merchant instance, it performs corresponding parsing on it, obtains the mapping relationship data between the similar questions and their re-labeled intents, and modifies the corresponding data records in the corresponding knowledge base, so that the intents marked by the relevant similar questions are updated to complete error correction.

[0162] In this embodiment, by remotely interacting with the merchant instance based on the error annotation set, error correction of the knowledge base is realized. After error correction, the expression of the mapping relationship between similar questions and intents in the knowledge base will be more in line with semantic relevance. The server can further put the corrected knowledge base back into the training of the intent classification model. Through training, the intent classification model can more accurately match intents for similar questions, thereby reflecting higher response intelligence in the intelligent customer service system and realizing the upgrade of the knowledge base of the intelligent customer service system. By continuously iterating in this way, the intelligence level of the intelligent customer service system can be continuously improved.

[0163] Please refer toFigure 8 , an intelligent customer service knowledge base quality inspection device provided to meet one of the purposes of the present application, is a functional embodiment of the intelligent customer service knowledge base quality inspection method of the present application. The device includes: a probability acquisition module 2100, a distribution learning module 2200, a filtering calculation module 2300, and a set selection module 2400, where:

[0164] The probability acquisition module 2100 is used to acquire the probability distribution data in the knowledge base of the merchant instance; the probability distribution data is used to store the classification probabilities between each similar question and each intention in the knowledge base;

[0165] The distribution learning module 2200 is used to perform confidence learning according to the probability distribution data to obtain joint distribution data; the joint distribution data is used to store the corresponding counts of each similar question under each noise label in the knowledge base being mapped to each true label, where the true label is the intention redirected by the maximum classification probability of the similar question in the probability distribution data, and the noise label is the originally marked intention of the redirected similar question;

[0166] The filtering calculation module 2300 is used to, for each noise label, determine the filtering amount of each similar question under the noise label according to the count of the noise label mapping to the true label in the joint distribution data, excluding the count of the true label pointing to the same intention as the noise label;

[0167] The set selection module 2400 is used to select multiple similar questions with the lowest classification probabilities under each noise label from the probability distribution data according to the filtering amount to form the mislabeled set of the similar questions.

[0168] In an extended embodiment, the intelligent customer service knowledge base quality inspection device of the present application further includes the following pre - running modules:

[0169] The task start module is used to start multiple parallel training tasks, and each training task uses the knowledge base of a corresponding merchant instance as the data set to jointly train the intention classification model;

[0170] The initialization module is configured to initialize the corresponding intention classification model instance after each training task starts, and set the classification number of the intention classification model instance according to the total number of intentions in the knowledge base of its corresponding merchant instance;

[0171] The model training module is configured to, for each training task, alternately divide the data set of its corresponding merchant instance into a training set and a validation set by using the cross - validation method, and train the intention classification model until it reaches the convergence state;

[0172] A data generation module, configured to classify each similar question from the knowledge bases of each merchant instance by using the intent classification model trained to convergence, and construct the classification probabilities obtained from the classification into probability distribution data of the knowledge base corresponding to the training task.

[0173] In a further embodiment, the intent classification model is configured to implement the following functions during its training process: extract text feature vectors of corresponding training samples in each training task respectively, where the training samples are similar questions in the training set; classify according to the text feature information to adapt to each training task, and determine the classification probabilities of mapping each text feature vector to each set classification; adapt to each training task, calculate the loss function value corresponding to each intent classification model instance by using the intent corresponding to the similar question in the training set as the supervision label, and perform gradient update on the intent classification model instance with the loss function value.

[0174] In a further embodiment, the distribution learning module 2200 includes:

[0175] A data representation sub-module, configured to convert the probability distribution data into a probability distribution matrix, where each element represents the classification probability that the similar question pointed to by its row coordinate is mapped to the intent pointed to by its column coordinate;

[0176] A threshold determination sub-module, configured to calculate and determine the confidence threshold for each intent according to the probability distribution matrix, where the confidence threshold is the average classification probability that the similar question in the corresponding intent is predicted as this intent;

[0177] A label determination sub-module, configured to determine the intent with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix as the true label, and determine the intent of the original label corresponding to the similar question of this true label as the noise label;

[0178] A distribution counting sub-module, configured to count the data distributions of the noise label and the true label to obtain a counting matrix, where each element represents the total number of similar questions corresponding to the noise label pointed to by its row coordinate and with the true label being the intent pointed to by the column coordinate of this element;

[0179] A counting conversion sub-module, configured to convert the counting matrix into joint distribution data, scale the joint distribution data to correspond to the total amount of similar questions in the corresponding knowledge base, and store the corresponding counts of similar questions under each noise label in the knowledge base being mapped to each true label.

[0180] In a specific embodiment, the counting conversion sub-module includes:

[0181] A scaling module for scaling the counting matrix to obtain a scaled intermediate matrix, such that the count of each noise label is based on the total number of similar questions under the noise label in the knowledge base;

[0182] A data normalization module for normalizing the scaled intermediate matrix to obtain a joint distribution matrix as the joint distribution data.

[0183] In an extended embodiment, the intelligent customer service knowledge base quality inspection device of the present application further includes:

[0184] A mislabeled push module for pushing a corresponding question data set to a merchant instance in response to an error correction start request of the merchant instance, where the question data set includes each similar question in the error annotation set and its corresponding noise label;

[0185] An update acquisition module for acquiring correction data of the question data set submitted by the merchant instance, where the correction data includes mapping relationship data between the similar questions in the question data set and their corrected intents;

[0186] An update correction module for modifying the corresponding relationship between the similar questions and intents in the knowledge base of the merchant instance according to the mapping relationship data.

[0187] To solve the above technical problems, an embodiment of the present application further provides a computer device. As Figure 9 shown, it is a schematic internal structure diagram of the computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database can store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement an intelligent customer service knowledge base quality inspection method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device can store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the intelligent customer service knowledge base quality inspection method of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand that Figure 9 the structure shown in

[0188] is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout. Figure 8The specific functions of each module and its sub-modules therein, the memory stores program codes and various types of data required to execute the above modules or sub-modules. The network interface is used for data transmission between user terminals or servers. The memory in this embodiment stores the program codes and data required to execute all modules / sub-modules in the intelligent customer service knowledge base quality inspection device of the present application, and the server can call the program codes and data of the server to execute the functions of all sub-modules.

[0189] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the intelligent customer service knowledge base quality inspection method according to any embodiment of the present application.

[0190] The present application also provides a computer program product, including computer programs / instructions, which when executed by one or more processors, implement the steps of the method according to any embodiment of the present application.

[0191] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments of the present application can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disc, a read-only memory (ROM), etc., or a random access memory (RAM), etc.

[0192] In summary, the present application can implement quality inspection of the knowledge base in the intelligent customer service system, screen out the mapping relationship data between suspicious similar questions and intents, so as to correct the relevant data, thereby improving the intelligence level of the intelligent customer service system.

[0193] Those skilled in the art of the present technology can understand that the steps, measures, and solutions in the various operations, methods, and processes discussed in the present application can be alternated, changed, combined, or deleted. Further, the other steps, measures, and solutions in the various operations, methods, and processes discussed in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and solutions in the prior art that are the same as those disclosed in the various operations, methods, and processes in the present application can also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0194] The above are only some embodiments of the present application. It should be noted that for those of ordinary skill in the art of the present technology, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An intelligent customer service knowledge base quality inspection method, characterized in that, The steps are as follows: Obtain the probability distribution data in the knowledge base of the merchant instance, including: converting the probability distribution data into a probability distribution matrix, where each element represents the classification probability of mapping the similar question pointed to by its row coordinate to the intention pointed to by its column coordinate; calculating and determining the confidence threshold for each intention according to the probability distribution matrix, and the confidence threshold is the average classification probability that the similar question in the corresponding intention is predicted as this intention; determining the intention with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix as the true label, and determining the intention of the original label of the similar question corresponding to this true label as the noise label; counting the data distribution of the noise label and the true label to obtain a count matrix, where each element represents the total number of similar questions with the true label being the intention pointed to by the column coordinate of this element among all similar questions corresponding to the intention corresponding to the noise label pointed to by its row coordinate; converting the count matrix into joint distribution data so that the joint distribution data corresponds to the scale of the total number of similar questions in the corresponding knowledge base, and storing the corresponding count of each noise label in the knowledge base where the subordinate similar questions are mapped to each true label; the probability distribution data is used to store the classification probability between each similar question and each intention in the knowledge base. Perform confidence learning based on the probability distribution data to learn the joint distribution data; the joint distribution data is used to store the corresponding count of each noise label in the knowledge base where the subordinate similar questions are mapped to each true label, the true label is the intention redirected by the maximum classification probability of the similar question in the probability distribution data, and the noise label is the intention of the original label of the redirected similar question. For each noise label, determine the filtering amount of the subordinate similar questions of each noise label according to the count in the joint distribution data where this noise label is mapped to the true label, excluding the count of the true label pointing to the same intention as this noise label. Select multiple similar questions with the lowest classification probability of each noise label from the probability distribution data according to the filtering amount to form the mislabeled set of the similar questions.

2. The intelligent customer service knowledge base quality inspection method according to claim 1, wherein Before obtaining the probability distribution data in the knowledge base of the merchant instance, the steps are as follows: Start multiple parallel training tasks, and each training task uses the knowledge base of a corresponding merchant instance as the data set to jointly train the intention classification model. After each training task is started, initialize the corresponding intention classification model instance, and set the classification number of this intention classification model instance according to the total number of intentions in the corresponding merchant instance's knowledge base. Each training task applies the cross-validation method to alternately divide the data set of its corresponding merchant instance into a training set and a validation set, and train the intention classification model until it converges. Use the intention classification model trained to convergence to classify each similar question from the knowledge bases of each merchant instance, and construct the classification probability obtained from the classification into the probability distribution data of the knowledge base corresponding to this training task.

3. The intelligent customer service knowledge base quality inspection method according to claim 2, wherein During the training process of the intention classification model, the following steps are executed: Extract the text feature vectors of the corresponding training samples in each training task, where the training samples are similar questions in the training set; Classify according to the text feature vectors to adapt to each training task, and determine the classification probabilities of mapping each text feature vector to each set classification; Adapt to each training task, use the intention corresponding to the similar question in the training set as the supervision label, calculate the loss function values corresponding to each intention classification model instance, and perform gradient update on the intention classification model instance with the loss function value.

4. The intelligent customer service knowledge base quality inspection method according to claim 1, wherein Convert the count matrix into joint distribution data, including the following steps: Scale the count matrix to obtain a scaled intermediate matrix, so that the count of each noise label is based on the total number of similar questions under the noise label in the knowledge base; Normalize the scaled intermediate matrix to obtain a joint distribution matrix as the joint distribution data.

5. The intelligent customer service knowledge base quality inspection method according to any one of claims 1 to 4, characterized in that According to the filtering quantity, select multiple similar questions with the lowest classification probabilities under each noise label from the probability distribution data to form the mislabeled set of the similar questions, and then, include the following steps: In response to the error correction start request of the merchant instance, push the corresponding question data set to it, where the question data set includes each similar question in the mislabeled set and its corresponding noise label; Obtain the correction data of the question data set submitted by the merchant instance, where the correction data includes the mapping relationship data between the similar questions in the question data set and their corrected intentions; Modify the corresponding relationship between the similar questions and intentions in the knowledge base of the merchant instance according to the mapping relationship data.

6. An intelligent customer service knowledge base quality inspection device, characterized in that, Include: A probability acquisition module for acquiring the probability distribution data in the knowledge base of the merchant instance, including: converting the probability distribution data into a probability distribution matrix, where each element represents the classification probability that the similar question pointed to by its row coordinate is mapped to the intention pointed to by its column coordinate; calculating and determining the confidence threshold for each intention according to the probability distribution matrix, where the confidence threshold is the average classification probability that the similar question in the corresponding intention is predicted as the intention; determining the intention with the maximum classification probability greater than the confidence threshold mapped by the similar question in the probability distribution matrix as the true label, and determining the originally marked intention of the similar question corresponding to the true label as the noise label; counting the data distribution of the noise label and the true label to obtain a count matrix, where each element represents the total number of similar questions corresponding to the intention of the noise label pointed to by its row coordinate and the true label is the intention pointed to by its column coordinate; converting the count matrix into joint distribution data to make the joint distribution data correspond to the scale of the total number of similar questions in the corresponding knowledge base, and storing the corresponding counts of each noise label subordinate similar question in the knowledge base mapped to each true label; the probability distribution data is used to store the classification probabilities between each similar question and each intention in the knowledge base; A distribution learning module for performing confidence learning based on the probability distribution data to acquire joint distribution data; the joint distribution data is used to store the corresponding counts of similar questions under each noisy label in the knowledge base being mapped to various true labels, where the true label is the intent redirected by the maximum classification probability of the similar questions in the probability distribution data, and the noisy label is the intent of the original label of the redirected similar questions; A filtering calculation module for, corresponding to each noisy label, determining the filtering quantity of similar questions under each noisy label according to the count of the noisy label mapped to the true label in the joint distribution data, excluding the count of the true label pointing to the same intent as the noisy label; A set selection module for selecting, according to the filtering quantity, multiple similar questions with the lowest classification probabilities under each noisy label from the probability distribution data to form the mislabeled set of the similar questions.

7. The intelligent customer service knowledge base quality inspection device according to claim 6, wherein It further includes the following modules that run in advance: A task startup module for starting multiple parallel training tasks, each training task using the knowledge base of a corresponding merchant instance as the data set to jointly train the intent classification model; An initialization module configured to initialize the corresponding intent classification model instance after each training task is started, and set the number of classifications of the intent classification model instance according to the total number of intents in the knowledge base of its corresponding merchant instance; A model training module configured to, for each training task, alternately divide the data set of its corresponding merchant instance into a training set and a validation set using the cross-validation method, and perform training on the intent classification model until it is trained to a converged state; A data generation module for classifying each similar question from the knowledge bases of each merchant instance using the intent classification model trained to convergence, and constructing the classification probabilities obtained from the classification as the probability distribution data of the knowledge base corresponding to this training task.

8. A computer device, comprising a central processing unit and a memory, characterized in that, The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to the method according to any one of claims 1 to 6. When the computer program is called and run by the computer, it executes the steps included in the corresponding method.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images