Question and answer library construction method, scoring method, electronic device and storage medium

By processing the conversation data set, calculating the point mutual information of candidate Q&A data, and automatically building the Q&A library, the problem of low efficiency in the construction of Q&A library in the existing technology is solved, and efficient Q&A library construction and manual customer service training are realized.

CN115017284BActive Publication Date: 2024-11-15ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210617144.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-01
Publication Date
2024-11-15
Estimated Expiration
2042-06-01

AI Technical Summary

Technical Problem

In the prior art, the method of building a question and answer library by manually writing questions and answers is inefficient, which leads to a long time and manpower in the customer service industry, and it is impossible to effectively reduce the cost of manual customer service training.

Method used

By obtaining the conversation data set, processing and extracting candidate question and answer data, computing the point mutual information of each candidate question and answer data to indicate its probability of occurrence in the target scenario, and building a question and answer library based on high-probability question and answer data without manual writing of questions and answers.

Benefits of technology

It improves the construction efficiency of the Q&A library, reduces the need for manual writing, and can automatically mine Q&A data suitable for target scenarios for manual customer service training, improving training efficiency and effectiveness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115017284B_ABST
    Figure CN115017284B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a method for constructing a question-and-answer library, a scoring method, an electronic device, and a storage medium. The method for constructing a question-and-answer library includes: obtaining a conversation data set, wherein the conversation data set includes multiple conversation data in a target scenario; processing the conversation data included in the conversation data set to obtain multiple candidate question-and-answer data; respectively determining the point mutual information of each candidate question-and-answer data, wherein the point mutual information is used to indicate the probability of the corresponding candidate question-and-answer data appearing in the target scenario; determining at least one question-and-answer data from each candidate question-and-answer data according to the point mutual information of each candidate question-and-answer data; and constructing a question-and-answer library corresponding to the target scenario through the at least one question-and-answer data. This solution can improve the efficiency of constructing a question-and-answer library.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a question and answer library construction method, a scoring method, an electronic device, and a storage medium. Background Art

[0002] In the customer service industry, since it is necessary to learn complex industry knowledge, manual customer service staff need to undergo a long period of training before they can take up their posts. The training robot is used for training manual customer service staff to reduce the training cost of manual customer service staff. The training robot is an intelligent dialogue system that can simulate a conversation between a user and a manual customer service staff, and train and score the service skills of the manual customer service staff. When the training robot trains and scores the service of the manual customer service staff, it is necessary to build a question and answer library including questions and answers. The training robot asks questions to the manual customer service staff based on the questions in the question and answer library, and scores the manual customer service staff based on the answers of the manual customer service staff and the answers in the question and answer library.

[0003] Currently, a question-and-answer library consisting of multiple question-and-answer pairs is constructed by manually writing industry-related questions and answers.

[0004] However, due to the complexity of industry knowledge, manually writing questions and answers to build a question-and-answer database requires a lot of time and manpower, resulting in low efficiency in building the question-and-answer database. Summary of the invention

[0005] In view of this, embodiments of the present application provide a question-and-answer database construction method, a scoring method, an electronic device, and a storage medium to at least solve or alleviate the above-mentioned problems.

[0006] According to a first aspect of an embodiment of the present application, a question and answer library file method is provided, comprising: obtaining a conversation data set, wherein the conversation data set includes multiple conversation data in a target scenario; processing the conversation data included in the conversation data set to obtain multiple candidate question and answer data; respectively determining the point mutual information of each of the candidate question and answer data, wherein the point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scenario; determining at least one question and answer data from each of the candidate question and answer data based on the point mutual information of each of the candidate question and answer data; and constructing a question and answer library corresponding to the target scenario through the at least one question and answer data.

[0007] According to a second aspect of an embodiment of the present application, a scoring method is provided, comprising: determining target question and answer data from a plurality of question and answer data included in a question and answer library; playing the question data included in the target question and answer data to an artificial customer service voice; obtaining the answer data generated by the artificial customer service for the question data; determining the service score of the artificial customer service according to the degree of match between the answer data and the answer data included in the target question and answer data; wherein the question and answer library is constructed in the following manner: obtaining a conversation data set, wherein the conversation data set includes a plurality of conversation data between an artificial agent and an incoming user; processing the conversation data included in the conversation data set to obtain a plurality of candidate question and answer data; respectively determining the point mutual information of each of the candidate question and answer data, wherein the point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scenario; determining at least one question and answer data from each of the candidate question and answer data according to the point mutual information of each of the candidate question and answer data; and constructing the question and answer library through the at least one question and answer data.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the question and answer database construction method described in the first aspect above or operations corresponding to the scoring method described in the second aspect above.

[0009] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the question and answer library construction method as described in the first aspect or the scoring method as described in the second aspect is implemented.

[0010] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to execute the question and answer library construction method as described in the first aspect above or the scoring method as described in the second aspect above.

[0011] According to the above technical solution, the dialogue data set includes multiple dialogue data in the target scenario, and multiple candidate question and answer data can be obtained by processing the dialogue data, and the point mutual information of the candidate question and answer data can indicate the probability of the candidate question and answer data appearing in the target scenario, so the candidate question and answer data with a higher probability of appearing in the target scenario can be determined as the question and answer data according to the point mutual information, and then the question and answer library corresponding to the target scenario is constructed based on the determined question and answer data. Since the dialogue data includes the question and answer mode dialogue between the dialoguers in the target scenario, the question and answer data including the question data and the answer data can be mined from the dialogue data, so that the question and answer data is suitable for the target scenario, and then the question and answer library is automatically constructed based on the question and answer data, without manually writing questions and answers to construct the question and answer library, thereby improving the construction efficiency of the question and answer library. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] Figure 1 is a schematic diagram of an exemplary system used in one embodiment of the present application;

[0014] Figure 2 is a flowchart of a method for constructing a question-and-answer database according to an embodiment of the present application;

[0015] Figure 3 is a flow chart of a method for determining point mutual information according to an embodiment of the present application;

[0016] Figure 4 is a flow chart of a scoring method according to an embodiment of the present application;

[0017] Figure 5 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0019] Exemplary Systems

[0020] Figure 1 An exemplary system for the question-answer library construction method applicable to the embodiment of the present application is shown. Figure 1 As shown, the system 100 may include a server 102, a communication network 104 and / or one or more user devices 106. Figure 1 The example in FIG. 1 is a plurality of user devices.

[0021] Server 102 may be any suitable server for storing information, data, programs, and / or any other suitable type of content. In some embodiments, server 102 may perform any suitable function. For example, in some embodiments, server 102 may be used to build a question-and-answer library. As an optional example, in some embodiments, server 102 may be used to build a question-and-answer library and score the results of the manual customer service's question responses.

[0022] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the server 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link or any suitable combination of such links.

[0023] User device 106 may include any one or more user devices suitable for receiving voice data and sending voice data. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, and / or any other suitable type of user device.

[0024] Although server 102 is illustrated as one device, in some embodiments, any suitable number of devices may be used to perform the functions performed by server 102. For example, in some embodiments, multiple devices may be used to implement the functions performed by server 102. Alternatively, the functions of server 102 may be implemented using cloud services.

[0025] Question and answer database construction method

[0026] Based on the above system, the present application embodiment provides a method for constructing a question and answer library, such as Figure 2 As shown, the question-answer database construction method includes the following steps:

[0027] Step 201: Obtain a conversation dataset.

[0028] The conversation data set includes multiple conversation data in the target scenario, each conversation data corresponds to a conversation between at least two interlocutors, a conversation includes at least one speech by each speaker, and each speech of the interlocutor corresponds to one speech data.

[0029] The target scenario is a scenario that requires question-answering and scoring. For example, when training and scoring human customer service through a training robot, the target scenario is the intelligent customer service scenario, and the conversation data set includes the conversation data between human customer service and online users in the intelligent customer service scenario.

[0030] Step 202: Process the dialogue data included in the dialogue data set to obtain a plurality of candidate question and answer data.

[0031] The candidate question and answer data includes question data and answer data, where the answer data is a reply to the question data.

[0032] Since the dialogue data set includes multiple dialogue data, the dialogue data includes the dialogue content between the interlocutors, and the interlocutors will have a question-and-answer mode dialogue, so part or all of the dialogue data in the dialogue data set includes corresponding question data and answer data. Therefore, by processing the dialogue data, multiple candidate question and answer data can be obtained.

[0033] Step 203: Determine the point mutual information of each candidate question and answer data respectively.

[0034] Point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scene. Since the candidate question and answer data includes question data and answer data, the probability of the candidate question and answer data appearing in the target scene is the probability of the question data and answer data included in the candidate question and answer data appearing in the target scene in combination.

[0035] If a candidate question and answer data has a high probability of appearing in the target scenario, it means that in the target scenario, the interlocutors often conduct question and answer dialogues corresponding to the question data and answer data in the candidate question and answer data, that is, the question data in the candidate question and answer data often appears as questions in the target scenario. If a candidate question and answer data has a low probability of appearing in the target scenario, it means that the question data and answer data included in the candidate question and answer data do not often appear in the dialogue of the target scenario, that is, the question data in the candidate question and answer data does not often appear as questions in the target scenario, or the question data and answer data in the candidate question and answer data do not match.

[0036] Step 204: Determine at least one question and answer data from each candidate question and answer data according to the point mutual information of each candidate question and answer data.

[0037] Since the point mutual information can indicate the probability of candidate question and answer data appearing in the target scenario, and the candidate question and answer data with a higher probability of appearing in the target scenario includes question data, which corresponds to the user questions that manual customer service personnel often encounter in their work, and is the content that requires training for manual customer service personnel. Therefore, based on the point mutual information of each candidate question and answer data, the candidate question and answer data with a higher probability of appearing in the target scenario can be determined as question and answer data, and then training can be conducted on the user questions that manual customer service personnel often encounter in their work.

[0038] Step 205: construct a question and answer library corresponding to the target scenario through the determined question and answer data.

[0039] After determining one or more question and answer data from each candidate question and answer data, a question and answer library corresponding to the target scenario is constructed through the determined question and answer data. Question and answer data can be obtained from the question and answer library, and manual customer service can be trained through the question data in the obtained question and answer data to improve the service capabilities of manual customer service.

[0040] In an embodiment of the present application, the conversation data set includes multiple conversation data in a target scenario. Multiple candidate question and answer data can be obtained by processing the conversation data, and the point mutual information of the candidate question and answer data can indicate the probability of occurrence of the candidate question and answer data in the target scenario. Therefore, the candidate question and answer data with a higher probability of occurrence in the target scenario can be determined as the question and answer data based on the point mutual information, and then a question and answer library corresponding to the target scenario is constructed based on the determined question and answer data. Since the conversation data includes a question and answer mode conversation between the interlocutors in the target scenario, the question and answer data including the question data and the answer data can be mined from the conversation data, so that the question and answer data is suitable for the target scenario, and then the question and answer library is automatically constructed based on the question and answer data, without the need to manually write questions and answers to construct the question and answer library, thereby improving the efficiency of constructing the question and answer library.

[0041] In one possible implementation, when processing the conversation data included in the conversation data set to obtain candidate question and answer data, speech merging processing can be performed on each conversation data in the conversation data set, so that the processed conversation data includes multiple speech data arranged in time sequence, and adjacent speech data correspond to different interlocutors, and then at least one conversation data pair in each conversation data can be respectively determined as candidate question and answer data based on the role information of the conversation in the target scenario.

[0042] Each conversation data corresponds to a conversation between the interlocutors. A conversation includes at least one speech by each interlocutor, and each speech corresponds to a voice data. However, the same interlocutor may make multiple consecutive speeches on the same topic. By performing speech merging processing on the conversation data, multiple voice data corresponding to the same interlocutor's multiple consecutive speeches can be merged into one speech data, so that the processed conversation data includes multiple speech data arranged in time sequence, and adjacent speech data correspond to different interlocutors.

[0043] For example, a conversation data includes speech data of interlocutor A and interlocutor B, and the speaking order of interlocutor A and interlocutor B is as follows:

[0044] Interviewer A: Voice data A1;

[0045] Interviewer B: voice data B1;

[0046] Interviewer A: Voice data A2;

[0047] Interviewer A: Voice data A3;

[0048] Interviewer B: Voice data B2.

[0049] When the conversation data is processed for speech merging, since voice data A2 and voice data A3 are continuous speeches of interlocutor A, voice data A2 and voice data A3 are merged into speech data A2, voice data A1 is determined as speech data A1, voice data B1 is determined as speech data B1, and voice data B2 is determined as speech data B2. The conversation data after the speech merging process is: speech data A1-speech data B1-speech data A2-speech data B2.

[0050] After the speech merging process is performed on the dialogue data, at least one speech data pair in each dialogue data can be determined as candidate question and answer data according to the role information of the dialogue in the target scenario. Since the dialogue role determines the order of the question data and the answer data in the dialogue data, for example, in the scenario of a two-person dialogue, the dialogue person A asks a question and then the dialogue person B answers, while in the scenario of a three-person dialogue, the dialogue person A asks a question and then the dialogue person B and the dialogue person C answer in sequence. In the scenario of a two-person dialogue, according to the role information of the dialogue person A and the dialogue person B, the speech data pair consisting of the dialogue person A's question data and the dialogue person B's answer data in the dialogue data is determined as candidate question and answer data. In the scenario of a three-person dialogue, according to the role information of the dialogue person A, the dialogue person B and the dialogue person C, the speech data pair consisting of the dialogue person A's question data and the dialogue person B's answer data in the dialogue data is determined as candidate question and answer data, and the speech data pair consisting of the dialogue person A's question data and the dialogue person C's answer data in the dialogue data is determined as candidate question and answer data.

[0051] In the embodiment of the present application, by processing the speech merging of the dialogue data, the processed dialogue data includes multiple speech data arranged in time sequence, and adjacent speech data correspond to different interlocutors, which facilitates the subsequent clustering of the speech data to determine the probability of question data and answer data appearing in the target scene, thereby accurately determining the question and answer library suitable for the target scene. According to the role information of the dialogue in the target scene, the speech data pairs in the dialogue data are determined as candidate question and answer data, so that the determined candidate question and answer data include the corresponding question data and answer data, thereby ensuring the accuracy of the constructed question and answer library.

[0052] In a possible implementation, when the speech merging process is performed on the dialogue data in the dialogue data set, the dialogue data summarized in the dialogue data set may also be normalized and stop word removed.

[0053] In the embodiment of the present application, through normalization processing, the name, telephone number, date of birth and other information in the dialogue data can be converted into corresponding field names, which is convenient for subsequent clustering of speech data in the dialogue data to identify speech data representing the same semantics, and then determine the point mutual information of candidate question and answer data. Through stop word removal processing, stop words in the dialogue data can be removed, reducing the data required to be processed when the dialogue data is subsequently processed, thereby improving the efficiency of question and answer database construction.

[0054] In a possible implementation, when the dialogue data includes speech data of two interlocutors, the speech data included in each dialogue data can be divided into multiple speech data pairs, each speech data pair includes two adjacent speech data, and then each speech data is determined as a candidate question and answer data.

[0055] For example, a dialogue data includes speech data 1 to speech data 6 arranged in time sequence, speech data 1, speech data 3 and speech data 5 are question data of interlocutor A, speech data 2 is the answer data of interlocutor B to speech data 1, speech data 4 is the answer data of interlocutor B to speech data 3, and speech data 6 is the answer data of interlocutor B to speech data 5. Then, the combination of speech data 1 and speech data 2 is used as a candidate question and answer data, the combination of speech data 3 and speech data 4 is used as a candidate question and answer data, and the combination of speech data 5 and speech data 6 is used as a candidate question and answer data.

[0056] In an embodiment of the present application, when the conversation data includes speech data of two interlocutors, the speech data arranged in time sequence in the conversation data are arranged alternately with question data and answer data, so that the speech data arranged in time sequence can be divided into multiple speech data pairs, each speech data pair includes two adjacent speech data, so that each speech data pair is a combination of "question data + answer data", ensuring that the candidate question data includes question data and answer data, and further ensuring the accuracy of the constructed question and answer library, so that manual customer service can be trained and assessed through the constructed question and answer library.

[0057] In a possible implementation manner, the point mutual information of the candidate question and answer data may be determined according to the semantics of the candidate question and answer data. Figure 3 is a flow chart of a method for determining point mutual information according to an embodiment of the present application. Figure 3 As shown, the method for determining the point mutual information includes the following steps:

[0058] Step 301: extract features from each speech data included in the dialogue data set to obtain semantic representations corresponding to each speech data.

[0059] For each speech data in the dialogue dataset, the pre-trained deep neural network encoder can be used to extract features of the speech data to obtain the semantic representation corresponding to the speech data. The semantic representation can indicate the semantics of the speech data. Speech data with different semantics correspond to different semantic representations. The semantic representation can be in the form of a multi-dimensional vector.

[0060] The deep neural network encoder used for feature extraction of speech data can be a convolutional neural network (CNN), a recurrent neural network (RNN) or a pre-trained language model, etc., or it can be a conversational sentence vector representation learning model, such as DialogueCSE, etc., which is not limited in this application.

[0061] Step 302: Cluster the semantic representations of each speech data to obtain a cluster label corresponding to each speech data.

[0062] Cluster the semantic representations corresponding to the speech data included in the dialogue dataset, and determine the cluster label corresponding to each speech data. The cluster label is used to indicate the type of semantics expressed by the speech data. Speech data expressing the same semantics correspond to the same cluster label, and speech data expressing different semantics correspond to different cluster labels.

[0063] When the semantic representations of the speech data are clustered, the semantic representations may be clustered using various standard clustering methods, for example, clustering methods such as KMeans and DBScan may be used to cluster the semantic representations.

[0064] It should be understood that clustering labels can indicate whether the semantics expressed by different speech data are the same, but cannot indicate the specific semantics expressed by the speech data. Clustering labels can be represented by natural numbers, and different natural numbers represent different clustering labels. For example, the clustering labels corresponding to speech data 1 and speech data 2 are both natural number 1, and the clustering label corresponding to speech data 3 is natural number 2. Then speech data 1 and speech data 2 express the same semantics, such as speech data 1 and speech data 2 both express the semantics of "do you have a mortgage", and speech data 3 expresses different semantics from speech data 1 and speech data 2, such as speech data 3 expresses the semantics of "do you have a child".

[0065] Step 303: Determine the point mutual information of each candidate question and answer data according to the clustering label corresponding to each speech data.

[0066] Since the speech data is generated based on the conversation between the interlocutors in the target scenario, the clustering label is used to indicate the type of semantics expressed by the speech data. Speech data expressing the same semantics corresponds to the same clustering label. Therefore, according to the clustering label corresponding to each speech data, the occurrence probability of speech data with various semantics in the target scenario can be determined, thereby determining the occurrence probability of candidate question and answer data composed of two speech data in the target scenario, that is, determining the point mutual information of each candidate question and answer data.

[0067] In an embodiment of the present application, after extracting the semantic representation of each speech data, by clustering the semantic representation of each speech data, the speech data expressing the same semantics can be determined, so that the number of speech data expressing each semantics in the dialogue data set can be determined, and then the occurrence probability of speech data with various semantics in the target scene can be obtained. Since the candidate question and answer data is composed of two speech data, the occurrence probability of speech data with various semantics in the target scene can be determined according to the occurrence probability of speech data with various semantics in the target scene, that is, the point mutual information of each candidate question and answer data, to ensure that the determined point mutual information can accurately reflect the occurrence probability of the candidate question and answer data in the target scene, and then ensure that the determined question and answer data is a commonly used question and answer dialogue in the target scene.

[0068] In one possible implementation, when determining the point mutual information of candidate question and answer data based on the cluster labels corresponding to each speech data, the cluster label pairs corresponding to each candidate question and answer data can be determined separately based on the cluster labels corresponding to each speech data, and then the point mutual information of each candidate question and answer data can be calculated separately based on the probability of occurrence of the cluster label pairs corresponding to each candidate question and answer data in each cluster label pair.

[0069] In an embodiment of the present application, the candidate question and answer data consists of two speech data. The combination of the clustering labels corresponding to the two speech data included in the candidate question and answer data is used as the clustering label pair corresponding to the candidate question and answer data. According to the clustering label pairs corresponding to each candidate question and answer data, the occurrence probability of each clustering label pair in each clustering label pair can be determined, and this probability is positively correlated with the occurrence probability of the corresponding candidate question and answer data in the target scene. Therefore, the point mutual information of each candidate question and answer data can be determined according to the occurrence probability of each clustering label pair in each clustering label pair, thereby ensuring the accuracy of the determined point mutual information.

[0070] It should be understood that the clustering label pairs corresponding to different candidate question and answer data have the same format. For example, the format of the clustering label pairs is (M, N), where M is the clustering label of the speech data that serves as question data in the candidate question and answer data, and N is the clustering label of the speech data that serves as answer data in the candidate question and answer data.

[0071] In one possible implementation, when calculating the point mutual information of candidate question and answer data based on the occurrence probability of each cluster label pair, for each candidate question and answer data, the point mutual information of the candidate question and answer data can be calculated by the following formula based on the occurrence probability of the cluster label pair corresponding to the candidate question and answer data in each cluster label pair.

[0072]

[0073] PMI(M,N) is used to represent the point mutual information of candidate question and answer data, (M,N) is used to represent the cluster label pairs corresponding to the candidate question and answer data, P(M,N) is used to represent the probability of occurrence of cluster label pairs (M,N) in each cluster label pair, P(M) is used to represent the probability of occurrence of cluster label M in each cluster label, and P(N) is used to represent the probability of occurrence of cluster label N in each cluster label.

[0074] In an embodiment of the present application, for each candidate question and answer data, the point mutual information of the candidate question and answer data is calculated by the above formula based on the occurrence probability of the clustering label pair corresponding to the candidate question and answer data and the occurrence probability of the clustering labels corresponding to the two speech data included in the candidate question and answer data. The calculated point mutual information can accurately reflect the probability of the occurrence of the combination of the two speech data included in the candidate question and answer data, and then the point mutual information can accurately characterize the occurrence probability of the candidate question and answer data in the target scene. When determining the candidate data from the candidate question and answer data based on the point mutual information, the question and answer dialogues that frequently appear in the target scene can be screened out, so that the constructed question and answer library includes common question and answer data in the target scene, and then when training manual customer service based on the question and answer library, the coverage of training knowledge and the training effect can be guaranteed.

[0075] In one possible implementation, when determining the question and answer data based on the point mutual information of each candidate question and answer data, at least one target cluster label pair with a larger corresponding point mutual information is determined from each cluster label pair, and then for each target cluster label pair, word segmentation is performed on each candidate question and answer data corresponding to the target cluster label pair, and word frequency statistics are performed on the multiple word segments obtained by the word segmentation processing. According to the word frequency statistics results, multiple word segments whose word frequencies are greater than a preset threshold are sequentially combined, and the combined result is used as the question and answer data corresponding to the target cluster label.

[0076] For example, among each candidate question and answer data, the candidate question and answer data corresponding to the clustering label pair (M, N) include candidate question and answer data 1, candidate question and answer data 2, and candidate question and answer data 3. After candidate question and answer data 1, candidate question and answer data 2, and candidate question and answer data 3 are segmented respectively, word frequency statistics are performed on each segmentation obtained by the segmentation processing. The word frequency statistics result is that the word frequency of segmentation 1, segmentation 2, segmentation 3, and segmentation 4 is greater than a preset threshold, and then segmentation 1, segmentation 2, segmentation 3, and segmentation 4 are combined according to the natural language structure as the question and answer data corresponding to the clustering label pair (M, N).

[0077] It should be understood that when segmenting the candidate question and answer data, it is necessary to segment the question data and answer data included in the candidate question and answer data separately. Accordingly, when combining the segmented words according to word frequency, the segmented words of the question data and the segmented words of the answer data are combined separately.

[0078] In an embodiment of the present application, word segmentation is performed on each candidate question and answer data corresponding to the same cluster label pair, and word frequency statistics are performed. According to the word frequency statistics results, the word segmentations with higher frequencies are combined to obtain the question and answer data corresponding to the cluster label pair, thereby ensuring that the generated question and answer data has a standard format, which can improve the readability of the question and answer data and thus improve the user experience.

[0079] Scoring Methodology

[0080] Based on the above system, an embodiment of the present application provides a scoring method, which is described in detail below through multiple embodiments.

[0081] Figure 4 Schematic diagram of a scoring method provided in the embodiment of the present application. Figure 4 As shown, the scoring method includes the following steps:

[0082] Step 401: Determine target question and answer data from a plurality of question and answer data included in the question and answer database.

[0083] The question-and-answer database includes multiple question-and-answer data, each of which includes question data and answer data, where the answer data is the answer corresponding to the question data. Each question-and-answer data has corresponding point mutual information, and each question-and-answer data in the question-and-answer database can be determined as the target question-and-answer data in descending order of corresponding point mutual information, or a question-and-answer data can be randomly selected from the question-and-answer database as the target question-and-answer data.

[0084] Step 402: Play the question data included in the target question and answer data to the human customer service voice.

[0085] After determining the target question and answer data, the target question and answer data are configured to the training robot, and the training robot plays the question data included in the target question and answer data in voice, simulating the customer asking questions to the manual customer service.

[0086] Step 403: Obtain answer data generated by the manual customer service in response to the question data.

[0087] After the question data included in the target question and answer data is played by voice, the answer data of the manual customer service for the question data is obtained. The voice data of the manual customer service can be collected by an audio collection device, and then the collected voice data is converted into answer data by a pre-trained voice recognition model.

[0088] Step 404: Determine the service score of the manual customer service based on the matching degree between the answer data and the answer data included in the target question and answer data.

[0089] After the answer data is obtained, the service score of the manual customer service is determined based on the matching degree between the answer data and the answer data included in the target question and answer data, wherein the higher the matching degree between the answer data and the answer data, the higher the service score of the manual customer service.

[0090] The question-and-answer database can be constructed through the following steps:

[0091] S1. Obtain a conversation data set, wherein the conversation data set includes multiple conversation data between a human agent and an incoming user;

[0092] S2. Processing the dialogue data included in the dialogue data set to obtain multiple candidate question and answer data;

[0093] S3. Determine the point mutual information of each candidate question and answer data respectively, wherein the point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scene;

[0094] S4. Determine at least one question and answer data from each candidate question and answer data according to the point mutual information of each candidate question and answer data;

[0095] S5. Build a question and answer database using at least one question and answer data.

[0096] In an embodiment of the present application, when constructing a question-and-answer library, the conversation data set includes multiple conversation data under the target scenario. By processing the conversation data, multiple candidate question-and-answer data can be obtained, and the point mutual information of the candidate question-and-answer data can indicate the probability of occurrence of the candidate question-and-answer data in the target scenario, so the candidate question-and-answer data with a higher probability of occurrence in the target scenario can be determined as the question-and-answer data according to the point mutual information, and then the question-and-answer library corresponding to the target scenario is constructed based on the determined question-and-answer data. Since the conversation data includes the question-and-answer mode conversation between the interlocutors under the target scenario, the question-and-answer data including the question data and the answer data can be mined from the conversation data, so that the question-and-answer data is suitable for the target scenario, and then the question-and-answer library is automatically constructed based on the question-and-answer data, without manually writing questions and answers to construct the question-and-answer library, thereby improving the construction efficiency of the question-and-answer library. When training and service scoring are performed on artificial customer service based on the constructed question-and-answer library, since the question-and-answer data in the question-and-answer library has a strong correlation with the work content of the artificial customer service, the effectiveness and effect of training the artificial customer service can be guaranteed.

[0097] It should be noted that Figure 4 The illustrated embodiment is a specific application of the question-and-answer library construction solution in the embodiment of the present application. The specific question-and-answer library construction process can be found in the description in the aforementioned embodiment, which will not be described again here.

[0098] Electronic devices

[0099] Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Figure 5 As shown, the electronic device may include: a processor (processor) 502, a communication interface (Communications Interface) 504, a memory (memory) 506, and a communication bus 508. Among them:

[0100] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0101] The communication interface 504 is used to communicate with other electronic devices or servers.

[0102] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in any of the aforementioned question and answer database construction method embodiments or any of the scoring method embodiments.

[0103] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0104] The processor 502 may be a CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0105] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0106] The program 510 may be specifically configured to enable the processor 502 to execute the question-and-answer database construction method or the scoring method in any of the aforementioned embodiments.

[0107] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in any of the above-mentioned question-answer database construction methods or scoring method embodiments, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiments, which will not be repeated here.

[0108] Through the electronic device of the embodiment of the present application, the dialogue data set includes multiple dialogue data in the target scene, and multiple candidate question and answer data can be obtained by processing the dialogue data, and the point mutual information of the candidate question and answer data can indicate the probability of occurrence of the candidate question and answer data in the target scene, so the candidate question and answer data with a higher probability of occurrence in the target scene can be determined as the question and answer data according to the point mutual information, and then the question and answer library corresponding to the target scene is constructed based on the determined question and answer data. Since the dialogue data includes the question and answer mode dialogue between the dialoguers in the target scene, the question and answer data including the question data and the answer data can be mined from the dialogue data, so that the question and answer data is suitable for the target scene, and then the question and answer library is automatically constructed based on the question and answer data, without manually writing questions and answers to construct the question and answer library, thereby improving the construction efficiency of the question and answer library.

[0109] Computer storage media

[0110] The present application also provides a computer-readable storage medium storing instructions for causing a machine to execute the question-answer library construction method or scoring method as described herein. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0111] In this case, the program code read from the storage medium itself can implement the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.

[0112] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0113] Computer program product

[0114] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.

[0115] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0116] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0117] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0118] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A method for constructing a question-answer database. include: Acquire a dialogue data set, wherein the dialogue data set includes a plurality of dialogue data in a target scenario; Processing the dialogue data included in the dialogue data set to obtain a plurality of candidate question and answer data; Determine the point mutual information of each candidate question and answer data respectively, wherein the point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scene; Determine at least one question and answer data from each of the candidate question and answer data according to the point mutual information of each of the candidate question and answer data; Building a question and answer library corresponding to the target scenario through the at least one question and answer data; The step of respectively determining the point mutual information of each candidate question and answer data comprises: respectively determining the point mutual information of each candidate question and answer data according to the semantic representation corresponding to each speech data included in the processed dialogue data.

2. The method for constructing a question-answer database according to claim 1, in, The processing of the dialogue data included in the dialogue data set to obtain a plurality of candidate question and answer data includes: Performing continuous speech merging processing on each of the dialogue data in the dialogue data set, so that the processed dialogue data includes at least two speech data arranged in time sequence, and adjacent speech data correspond to different interlocutors; According to the role information of the dialogue in the target scenario, at least one speech data pair in each of the dialogue data is determined as the candidate question and answer data.

3. The method for constructing a question-answer database according to claim 2, in, The dialogue data includes speech data of two dialoguers; The step of determining at least one speech data pair in each of the dialogue data as the candidate question-answer data comprises: The speech data included in each of the dialogue data is divided into at least one speech data pair, each of the speech data pairs includes two adjacent speech data, and each of the speech data pairs is determined as the candidate question and answer data.

4. The method for constructing a question-answer database according to claim 2, in, Determining the point mutual information of each candidate question and answer data respectively according to the semantic representation corresponding to each speech data included in the processed dialogue data, including: Performing feature extraction on each of the speech data included in the processed dialogue data to obtain a semantic representation corresponding to each of the speech data; Clustering the semantic representations of each of the speech data to obtain a cluster label corresponding to each of the speech data; According to the clustering labels corresponding to the speech data, the point mutual information of each candidate question and answer data is determined respectively.

5. The method for constructing a question-answer database according to claim 4, in, Determining the point mutual information of each candidate question and answer data according to the clustering labels corresponding to the speech data includes: According to the cluster labels corresponding to the speech data, respectively determine the cluster label pairs corresponding to each candidate question and answer data; According to the occurrence probability of the cluster label pair corresponding to each of the candidate question and answer data in each of the cluster label pairs, the point mutual information of each of the candidate question and answer data is calculated respectively.

6. The method for constructing a question-answer database according to claim 5, in, The step of calculating the point mutual information of each candidate question and answer data according to the occurrence probability of the cluster label pair corresponding to each candidate question and answer data in each cluster label pair comprises: For each candidate question and answer data, the point mutual information of the candidate question and answer data is calculated by the following formula according to the occurrence probability of the cluster label pair corresponding to the candidate question and answer data in each cluster label pair; PMI(M,N) is used to characterize the point mutual information of the candidate question and answer data, (M,N) is used to characterize the cluster label pairs corresponding to the candidate question and answer data, P(M,N) is used to characterize the probability of occurrence of the cluster label pair (M,N) in each cluster label pair, P(M) is used to characterize the probability of occurrence of cluster label M in each cluster label, and P(N) is used to characterize the probability of occurrence of cluster label N in each cluster label.

7. The method for constructing a question-answer database according to claim 6, in, The step of determining at least one question and answer data from each of the candidate question and answer data according to the point mutual information of each of the candidate question and answer data comprises: Determining at least one target cluster label pair from each of the cluster label pairs based on the point mutual information; For each of the target cluster label pairs, word segmentation processing is performed on each candidate question and answer data corresponding to the target cluster label pair, and word frequency statistics are performed on the word segmentations obtained by the word segmentation processing to obtain word frequency statistics results, and at least two word segmentations whose word frequencies are greater than a preset threshold are sequentially combined according to the word frequency statistics results to obtain the question and answer data corresponding to the target cluster label pair.

8. The method for constructing a question-answer database according to any one of claims 1 to 7, in, The method further comprises: The conversation data in the conversation data set is normalized and stop word removed.

9. A scoring method, include: Determining target question and answer data from a plurality of question and answer data included in the question and answer library; Playing the question data included in the target question and answer data to the human customer service voice; Obtaining answer data generated by the manual customer service in response to the question data; Determining a service score of the manual customer service according to a degree of matching between the answer data and the answer data included in the target question and answer data; The question-answer database is constructed in the following way: Acquire a conversation data set, wherein the conversation data set includes a plurality of conversation data between a human agent and an incoming user; Processing the dialogue data included in the dialogue data set to obtain a plurality of candidate question and answer data; Determine the point mutual information of each candidate question and answer data respectively, wherein the point mutual information is used to indicate the probability of the corresponding candidate question and answer data appearing in the target scene; Determine at least one question and answer data from each of the candidate question and answer data according to the point mutual information of each of the candidate question and answer data; Constructing the question and answer library through the at least one question and answer data; The step of respectively determining the point mutual information of each candidate question and answer data comprises: respectively determining the point mutual information of each candidate question and answer data according to the semantic representation corresponding to each speech data included in the processed dialogue data.

10. An electronic device, include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the question and answer database construction method according to any one of claims 1 to 8 or an operation corresponding to the scoring method according to claim 9.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the question-answer database construction method according to any one of claims 1 to 8 or the scoring method according to claim 9.

12. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the question-answer database construction method according to any one of claims 1 to 8 or operations corresponding to the scoring method according to claim 9.