Training method of quality evaluation model, multi-round dialogue quality evaluation method and device

Abstract

The disclosure provides a quality evaluation model training method and device and a multi-round dialogue quality evaluation method, relates to the fields of artificial intelligence such as natural language processing and deep learning, and comprises the following steps: obtaining an initial quality evaluation model, training the initial quality evaluation model, obtaining a trained candidate quality evaluation model, labeling a first sample multi-round dialogue to obtain a sample label vector set, training the candidate quality evaluation model according to the sample label vector set, and obtaining a trained target quality evaluation model. The semantic learning effect of the candidate quality evaluation model on the first sample multi-round dialogue is optimized, the quality evaluation capability of the candidate quality evaluation model for the first sample multi-round dialogue is improved, the quality evaluation efficiency and precision of the multi-round dialogue are improved, the degree of artificial dependence and the artificial cost are reduced compared with the quality evaluation of the multi-round dialogue relying on artificial implementation, and accurate data support is provided for a downstream task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and in particular to the field of artificial intelligence such as natural language processing and deep learning. BACKGROUND

[0002] With the development of technology, a dialogue service has become a popular application, in which a user can obtain required information through a real person multi-round dialogue. In this scenario, quality of the multi-round dialogue can be evaluated, so as to measure the quality of the service provided by the multi-round dialogue.

[0003] In the related art, the quality of the multi-round dialogue can be evaluated by a human, and the correctness and professionalism of an answer in the multi-round dialogue and other related parameters can be evaluated by the human, so as to realize the quality evaluation of the multi-round dialogue. However, the cost of the human is high. SUMMARY

[0004] The present disclosure provides a training method of a quality evaluation model, a multi-round dialogue quality evaluation method and device.

[0005] According to a first aspect of the present disclosure, a training method of a quality evaluation model is provided. The method comprises: obtaining an initial quality evaluation model to be trained, and performing model training on the initial quality evaluation model to obtain a trained candidate quality evaluation model; obtaining a first sample multi-round dialogue and labeling the first sample multi-round dialogue to obtain a sample label vector set of the first sample multi-round dialogue; and training the candidate quality evaluation model according to the sample label vector set until the training is completed, to obtain a trained target quality evaluation model.

[0006] According to a second aspect of the present disclosure, a multi-round dialogue quality evaluation method is provided. The method comprises: obtaining a trained target multi-round dialogue quality evaluation model, wherein the target multi-round dialogue quality evaluation model is obtained by the training method of the quality evaluation model according to the first aspect; obtaining a multi-round dialogue to be evaluated, and inputting the multi-round dialogue into the target multi-round dialogue quality evaluation model to obtain a quality evaluation result output by the target multi-round dialogue quality evaluation model.

[0007] According to a third aspect of the present disclosure, a training device of a quality evaluation model is provided. The device comprises: a first training module configured to obtain an initial quality evaluation model to be trained, and perform model training on the initial quality evaluation model to obtain a trained candidate quality evaluation model; a first obtaining module configured to obtain a first sample multi-round dialogue and label the first sample multi-round dialogue to obtain a sample label vector set of the first sample multi-round dialogue; and a second training module configured to train the candidate quality evaluation model according to the sample label vector set until the training is completed, to obtain a trained target quality evaluation model.

[0008] According to a fourth aspect of the present disclosure, a multi-turn dialogue quality evaluation device is provided, the device comprising: a second acquisition module configured to acquire a trained target multi-turn dialogue quality evaluation model, wherein the target multi-turn dialogue quality evaluation model is obtained by the quality evaluation model training device according to the third aspect of the present disclosure; and an evaluation module configured to acquire a multi-turn dialogue to be evaluated, and input the multi-turn dialogue into the target multi-turn dialogue quality evaluation model to obtain a quality evaluation result output by the target multi-turn dialogue quality evaluation model.

[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the quality evaluation model training method according to the first aspect of the present disclosure and / or the multi-turn dialogue quality evaluation method according to the second aspect of the present disclosure.

[0010] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the quality evaluation model training method according to the first aspect of the present disclosure and / or the multi-turn dialogue quality evaluation method according to the second aspect of the present disclosure.

[0011] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the quality evaluation model training method according to the first aspect of the present disclosure and / or the multi-turn dialogue quality evaluation method according to the second aspect of the present disclosure.

[0012] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0014] Figure 1 a flowchart of a quality evaluation model training method according to an embodiment of the present disclosure;

[0015] Figure 2 a flowchart of a quality evaluation model training method according to another embodiment of the present disclosure;

[0016] Figure 3 a flowchart of a quality evaluation model training method according to another embodiment of the present disclosure;

[0017] Figure 4 a flowchart of a training method of a quality evaluation model according to another embodiment of the present disclosure;

[0018] Figure 5 a flowchart of a training method of a quality evaluation model according to another embodiment of the present disclosure;

[0019] Figure 6 a flowchart of a multi-turn dialogue quality evaluation method according to an embodiment of the present disclosure;

[0020] Figure 7 a structural diagram of a training device of a quality evaluation model according to an embodiment of the present disclosure;

[0021] Figure 8 a structural diagram of a multi-turn dialogue quality evaluation device according to an embodiment of the present disclosure;

[0022] Figure 9 a schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a thorough understanding of embodiments of the present disclosure by a person of ordinary skill in the art, and should be considered in connection with the following detailed description, but do not limit the scope of the present disclosure. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, in the following description, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

[0024] Data processing is a basic link of system engineering and automatic control. Data is an expression form of facts, concepts or instructions, which can be processed by artificial or automatic devices. After data is interpreted and given certain meaning, it becomes information. Data processing is the collection, storage, retrieval, processing, transformation and transmission of data. The basic purpose of data processing is to extract and deduce data that is valuable and meaningful to certain people from a large amount of data that is possibly disorganized and difficult to understand.

[0025] Deep learning (DL) is a new research direction in the field of machine learning. Deep learning is to learn the internal rules and representation levels of sample data. The information obtained in the learning process is very helpful for the interpretation of data such as text, images and sound. The ultimate goal of deep learning is to enable machines to have analysis and learning ability like people, and to recognize text, images and sound data.

[0026] Natural Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not generally studying natural language, but developing computer systems that can effectively implement natural language communication.

[0027] Artificial Intelligence (AI) is a new technical science that studies, develops and extends human intelligence. Artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Since the birth of artificial intelligence, the theory and technology have become more mature, and the application field has also expanded. It can be imagined that the technology products brought by artificial intelligence in the future will be a "container" of human intelligence. Artificial intelligence can simulate the information process of human consciousness and thinking.

[0028] Figure 1 The flowchart of the training method of the quality evaluation model of an embodiment of the present disclosure is shown in FIG. 1, which comprises the following steps. Figure 1

[0029] In S101, an initial quality evaluation model to be trained is obtained, and the initial quality evaluation model is trained to obtain a trained candidate quality evaluation model.

[0030] In the implementation, people are used to obtaining information through multiple rounds of real human dialogue. In this scenario, the quality of the dialogue can be evaluated to determine whether the answers obtained by the questioner in the dialogue process meet the user's needs.

[0031] In the embodiments of the present disclosure, the trained model used for multi-round dialogue quality evaluation can be determined as the initial quality evaluation model.

[0032] Optionally, the historical multi-round dialogue of the sample user and the quality evaluation label of the historical multi-round dialogue can be obtained, and then the corresponding training sample can be obtained based on the historical multi-round dialogue and the quality evaluation label of the historical multi-round dialogue.

[0033] ​Further, the initial quality evaluation model is trained according to the training sample, and the trained model is determined as the candidate quality evaluation model.

[0034] S102, a first sample multi-turn dialogue is obtained and labeled, and a sample label vector set of the first sample multi-turn dialogue is obtained.

[0035] In the embodiments of the present disclosure, the candidate quality evaluation model can be subjected to secondary model training, so as to realize further iterative optimization of the trained candidate quality evaluation model.

[0036] The sample multi-turn dialogue subjected to the secondary training of the candidate quality evaluation model can be marked as the first sample multi-turn dialogue.

[0037] Optionally, the first sample multi-turn dialogue can be labeled multiple times, and a set composed of vectors obtained by each labeling is determined as the sample label vector set.

[0038] The labeling strategy of the pre-set first sample multi-turn dialogue can be obtained, the element to be labeled in the first sample multi-turn dialogue can be obtained based on the labeling strategy, and the element to be labeled is encoded, so as to obtain the sample label vector set of the first sample multi-turn dialogue.

[0039] S103, the candidate quality evaluation model is trained according to the sample label vector set until the training is completed, and a trained target quality evaluation model is obtained.

[0040] In the embodiments of the present disclosure, the sample label vector set can be input into the candidate quality evaluation model, the sample label vector is subjected to feature extraction by the candidate quality evaluation model, and the first sample multi-turn dialogue is subjected to quality evaluation according to the extracted features.

[0041] As an example, as shown in Figure 2 the sample label set can be input into the candidate quality evaluation model shown in Figure 2 The language model (ERNIE 2.0) in the candidate quality evaluation model is used to extract features of each vector in the sample label vector set, so as to obtain feature expressions in the dimensions of morphology, syntax and semantics of the first sample multi-turn dialogue.

[0042] Further, the features extracted by the language model shown in Figure 2 are input into the fully connected layer (Fully connected layer, FC) and the normalization layer (softmax) shown in Figure 2 The features extracted by the language model are subjected to quality evaluation by the FC and the softmax, so as to obtain Figure 2The candidate quality evaluation model shown evaluates the quality of the first sample multi-turn dialogue.

[0043] It should be noted that the quality evaluation result can be a three-classification result output by the candidate quality evaluation model, where the three classifications can include high-quality resolution, resolution, and non-resolution, which are not limited here.

[0044] Optionally, the training loss of the candidate quality evaluation model can be obtained according to the quality evaluation result output by the candidate quality evaluation model, and the candidate quality evaluation model can be iteratively optimized according to the training loss until the training is completed. The candidate quality evaluation model obtained after the training is completed is determined as the trained target quality evaluation model.

[0045] The training method of the multi-turn dialogue quality evaluation model provided by the present disclosure trains the initial quality evaluation model to be trained to obtain a trained candidate quality evaluation model, obtains a first sample multi-turn dialogue, and labels the first sample multi-turn dialogue to obtain a sample label vector set. The candidate quality evaluation model is trained again according to the sample label vector set until the training is completed, and a trained target quality evaluation model is obtained. In the present disclosure, the candidate quality evaluation model is obtained by training the initial quality evaluation model, and the candidate quality evaluation model is trained again by labeling the first sample multi-turn dialogue to obtain a sample label vector. The semantic learning effect of the candidate quality evaluation model on the first sample multi-turn dialogue is optimized, the quality evaluation capability of the candidate quality evaluation model for the first sample multi-turn dialogue is improved, and the training method and training effect of the quality evaluation model are optimized. In the scenario of quality evaluation of multi-turn dialogue by the target quality evaluation model, the quality evaluation efficiency and accuracy of the multi-turn dialogue are improved, the degree of dependence on artificial and the cost of artificial are reduced compared with the quality evaluation of multi-turn dialogue relying on artificial, and accurate data support is provided for downstream tasks.

[0046] In the above embodiments, regarding the labeling of the first sample multi-turn dialogue, the following can be combined Figure 3 understand, Figure 3 The flowchart of the training method of the multi-turn dialogue quality evaluation model of another embodiment of the present disclosure is shown in Figure 3 as shown, the method comprises:

[0047] S301, obtaining the question text in the first sample multi-turn dialogue and the first reduction information of the question text, and the answer text in the first sample multi-turn dialogue and the second reduction information of the question and answer text.

[0048] In the embodiments of the present disclosure, the first sample multi-turn dialogue includes question text and answer text, wherein the text corresponding to the question part can be marked as the question text in the first sample multi-turn dialogue, and the text answering the question text in the question part can be marked as the answer text in the first sample multi-turn dialogue.

[0049] Optionally, a first reduction item of the pre-set question text and a first feature value of the question text under the first reduction item can be obtained, and the first feature value can be binned to obtain a first reduction identifier of the first reduction item.

[0050] The summary item of the pre-set feature of the question text can be marked as the first reduction item of the question text. For example, the question text can be evaluated and summarized based on three dimensions of question classification, user score and question value, and the first reduction item of the question text is the question classification item, the user score item and the question value item.

[0051] In this scenario, the first feature value of the question text under the first reduction item can be obtained, and the first feature value can be binned to obtain the first reduction identifier of the question text under the first reduction item.

[0052] As an example, the first reduction item is set as the question value item, and the first feature value of the question text under the question value item can be binned, wherein the question value item is provided with three buckets of low, medium and high. In this example, the feature value interval of each of the low, medium and high buckets can be obtained, and the corresponding bucket of the feature value interval to which the first feature value of the question value item belongs is determined as the bucket to which the first feature value belongs.

[0053] When the bucket to which the first feature value belongs is medium, the first reduction identifier of the question text under the question value item is medium.

[0054] Optionally, the first reduction information of the question text can be obtained according to the first reduction item and the first reduction identifier, wherein the first reduction item and the first reduction identifier can be combined, and the information obtained by the combination is determined as the first reduction information of the question text.

[0055] As an example, based on the above example, the first reduction identifier of the question text under the question value item as the first reduction item is medium, and the question value item and medium can be combined, and the combination of the question value item and medium is determined as the first reduction information of the question text.

[0056] Optionally, a second reduction item of the pre-set answer text and a second feature value of the answer text under the second reduction item can be obtained, and the second feature value can be binned to obtain a second reduction identifier of the second reduction item.

[0057] The pre-set summary item that can evaluate the features of the answer text is marked as a second summary item of the answer text. For example, the answer text can be evaluated based on two dimensions of the answerer authentication field and user feedback, and the second summary item of the answer text is the answerer authentication field item and the user feedback item.

[0058] In this scenario, the second feature value of the answer text under the second summary item can be obtained, and the second feature value is binned, thereby obtaining the second summary identifier of the answer text under the second summary item.

[0059] As an example, the second summary item is set as the user feedback item, and the second feature value of the answer text under the user feedback item can be binned. The user feedback item is provided with five buckets of very low, low, medium, high, and very high. In this example, the feature value interval of each of the five buckets of very low, low, medium, high, and very high can be obtained, and the corresponding bucket of the feature value interval to which the second feature value under the user feedback item belongs is determined as the bucket to which the second feature value belongs.

[0060] When the bucket to which the second feature value belongs is very high, the second summary identifier of the answer text under the user feedback item is very high.

[0061] Optionally, the second summary information of the answer text is obtained according to the second summary item and the second summary identifier. The second summary item and the second summary identifier can be combined, and the information obtained by the combination is determined as the second summary information of the answer text.

[0062] As an example, based on the above example, the second summary identifier of the answer text under the user feedback item as the second summary item is very high, the user feedback item and very high can be combined, and the combination of “user feedback item: very high” is determined as the second summary information of the answer text.

[0063] S302, the question text, the answer text, the first summary information and the second summary information are spliced to obtain the first sample label vector of the first sample multi-round dialogue.

[0064] Optionally, the question text can be obtained. The question text belongs to the asking round, and the first label symbol of the question text is generated according to the asking round. The first label symbol includes the question text asking round serial number.

[0065] In the embodiment of the present disclosure, the first sample multi-round dialogue includes a plurality of question texts and a plurality of answer texts. In this scenario, the question text has a corresponding asking round.

[0066] In this scenario, the corresponding annotation symbol can be generated according to the question round to which the question text belongs, as the first annotation symbol, wherein the first annotation symbol includes the question round serial number corresponding to the question text.

[0067] For example, the first question text in the first sample multi-round dialogue can correspond to a question round serial number of 1, and in this scenario, the first annotation symbol of the first question text can be generated based on the serial number, that is, "Q1".

[0068] For example, the second question text in the first sample multi-round dialogue can correspond to a question round serial number of 2, and in this scenario, the first annotation symbol of the second question text can be generated based on the serial number, that is, "Q2".

[0069] Optionally, the answer round to which the answer text belongs is obtained, and the second annotation symbol of the answer text is generated according to the answer round, wherein the second annotation symbol includes the answer round serial number of the answer text.

[0070] In the embodiment of the present disclosure, the first sample multi-round dialogue includes multiple question texts and multiple answer texts, and in this scenario, the answer text has a corresponding answer round.

[0071] In this scenario, the corresponding annotation symbol can be generated according to the answer round to which the answer text belongs, as the second annotation symbol, wherein the second annotation symbol includes the question round serial number corresponding to the answer text.

[0072] For example, the first answer text in the first sample multi-round dialogue can correspond to an answer round serial number of 1, and in this scenario, the second annotation symbol of the first answer text can be generated based on the serial number, that is, "R1".

[0073] For example, the second answer text in the first sample multi-round dialogue can correspond to an answer round serial number of 2, and in this scenario, the second annotation symbol of the second answer text can be generated based on the serial number, that is, "R2".

[0074] Optionally, the third annotation symbol of the first and second reduction information can be obtained.

[0075] In the embodiment of the present disclosure, the corresponding annotation symbol can be set for the first and second reduction information, as the third annotation symbol.

[0076] For example, the third annotation symbol of the first reduction information can be set as "QF", and the third annotation symbol of the second reduction information can be set as "RF".

[0077] Optionally, based on the first annotation symbol, the second annotation symbol and the third annotation symbol, the question text, the answer text, the first resolution information and the second resolution information are spliced to obtain a first sample annotation vector of the first sample multi-turn dialogue.

[0078] In the first sample multi-turn dialogue, the splicing order between each question text, each answer text, the first resolution information and the second resolution information can be obtained.

[0079] In the first sample multi-turn dialogue, the first question text and the first answer text are interleaved, the first answer text and the second answer text are interleaved, and the second answer text and the third question text are interleaved, and so on.

[0080] In this scenario, the question text and the answer text of the first sample multi-turn dialogue can be recognized to obtain the splicing order between each question text and each answer text.

[0081] Correspondingly, the corresponding positions of the first resolution information and the second resolution information can be set to obtain the splicing order of the first resolution information and the second resolution information.

[0082] Optionally, the question text can be annotated by the first annotation symbol to obtain a first annotation segment, and the answer text can be annotated by the second annotation symbol to obtain a second annotation segment, and the first resolution information and / or the second resolution information can be annotated by the third annotation symbol suffix pair to obtain a third annotation segment.

[0083] In this scenario, the pre-set splicing symbol can be obtained, and the first annotation segment, the second annotation segment and the third annotation segment can be spliced according to the splicing order by the splicing symbol to obtain the first sample annotation vector.

[0084] As an example, it is assumed that the first sample multi-turn dialogue includes two question texts and three answer texts, wherein the first annotation symbol of the first question text is Q1, the first annotation symbol of the second question text is Q2, the second annotation symbol of the first answer text is R1, the second annotation symbol of the second answer text is R2, the second annotation symbol of the third answer text is R3, the third annotation symbol of the first resolution information is QF, and the third annotation symbol of the second resolution information is RF.

[0085] In this scenario, the first question text can be annotated by Q1 to obtain a first annotation segment Q1, and the second question text can be annotated by Q2 to obtain a first annotation segment Q2.

[0086] Correspondingly, the first answer text is labeled with R1 to obtain a second labeled segment R1, the second answer text is labeled with R2 to obtain a second labeled segment R2, and the first answer text is labeled with R3 to obtain a second labeled segment R3.

[0087] In addition, the first resolution information is labeled with QF to obtain a third labeled segment QF, and the second resolution information is labeled with RF to obtain a third labeled segment RF.

[0088] Further, a preset splicing symbol is obtained.

[0089] As an example, the splicing symbol between the first labeled segment and the second labeled segment is set as TURN, the splicing symbol between the first labeled segment or the second labeled segment and the third labeled segment is set as SEP, and the splicing symbol between the third labeled segment of the first resolution information and the third labeled segment of the second resolution information is set as SEP.

[0090] In this example, the first labeled segment, the second labeled segment, and the third labeled segment are spliced according to the obtained splicing order through the part of the splicing symbol, so as to obtain Figure 3 The first sample label vector CLS, Q1, TURN, R1, TURN, Q2, TURN, R2, PARA, R3, SEP, QF, SEP, RF, and SEP are shown.

[0091] The CLS is a starting label value of the first sample label vector.

[0092] It should be noted that in the first sample label vector, R3 is a follow-up answer text after R2. In this scenario, in response to the two segments adjacent to each other being the first labeled segment of the two question texts and / or the second labeled segment of the two answer texts, a fourth label symbol is obtained and added between the first labeled segment of the two question texts and / or the second labeled segment of the two answer texts to obtain an added first sample label vector.

[0093] In the embodiments of the present disclosure, there is no answer text between the possible question text and the question text in the multi-turn dialogue, which can be understood as that the questioner can ask follow-up questions for the questions he has asked without receiving answers. In this scenario, the possible question text and the question text are adjacent in the first sample multi-turn dialogue.

[0094] Correspondingly, there is no question text between the possible answer text and the answer text in the multi-turn dialogue, which can be understood as that the respondent can reply to the content that has been answered by the questioner without the questioner putting forward the next question, and in this scenario, the possible answer text and the answer text are adjacent in the first sample multi-turn dialogue.

[0095] In this scenario, the pre-configured fourth annotation symbol can be inserted between the first annotation segment of each of the two adjacent question-answer texts and / or between the second annotation segment of each of the two adjacent answer texts.

[0096] As an example, as shown in Figure 4 , the fourth annotation symbol can be PARA, and as shown in Figure 4 , in the first sample annotation vector, R2 and R3 are the second annotation segments of the two adjacent answer texts respectively, and in this scenario, the fourth annotation symbol PARA can be inserted between R2 and R3, thereby obtaining the first sample annotation vector shown in Figure 4 .

[0097] S303, annotating the first sample annotation vector to obtain a second sample annotation vector of the first sample annotation vector.

[0098] In the embodiments of the present disclosure, the first sample annotation vector can be divided into a question text and answer text part, a first reduction information part and a second reduction information part, and the three parts are annotated respectively to obtain the annotated second sample annotation vector.

[0099] Optionally, the first region annotation value of the question text and the answer text, the second region annotation value of the first reduction information and the third region annotation value of the second reduction information in the first sample annotation vector are obtained.

[0100] Among them, the annotation value of annotating the question text and the answer text part can be determined as the first region annotation value, the annotation value of annotating the first reduction information can be determined as the second region annotation value, and the annotation value of annotating the second reduction information can be determined as the third region annotation value.

[0101] Optionally, the first sample annotation vector can be encoded according to the first region annotation value, the second region annotation value and the third region annotation value to obtain the second sample annotation vector.

[0102] Among them, the region order between the question text and the answer text, the first reduction information and the second reduction information in the first sample annotation vector can be obtained, and the first region annotation value, the second region annotation value and the third region annotation value can be sorted according to the region order to obtain the second sample annotation vector.

[0103] As an example, as shown inFigure 3 As shown in Figure 3 In the first sample annotation vector shown, the sequence of regions between the question text and the answer text, the first resolution information and the second resolution information is CLS→Q1→TURN→R1→TURN→Q2→TURN→R2→PARA→R3→SEP→QF→SEP→RF→SEP.

[0104] Wherein, CLS→Q1→TURN→R1→TURN→Q2→TURN→R2→PARA→R3→SEP belongs to the part corresponding to the question text and the answer text pair, then the annotation values of CLS, Q1, TURN, R1, TURN, Q2, TURN, R2, PARA, R3 and SEP are respectively the first region annotation values.

[0105] And QF→SEP is the part corresponding to the first resolution information, then the annotation values of QF and SEP are respectively the second annotation values.

[0106] And RF→SEP is the part corresponding to the second resolution information, then the annotation values of RF and SEP are respectively the third annotation values.

[0107] In this scenario, set the first region annotation value as 0, the second region annotation value as 1, and the third region annotation value as 2, then encode the first sample annotation vector according to the first region annotation value, the second region annotation value and the third region annotation value to obtain the vector Figure 3 The second sample annotation vector in the region separation dimension shown is 000000000001122. Figure 3

[0108] In the embodiments of the present disclosure, the order of each token in the first sample annotation vector can be obtained, and the encoding of the first sample annotation vector is performed based on the order of each token to obtain the second sample annotation vector.

[0109] Optionally, the token order of each token in the token sequence of the first sample annotation vector can be obtained, and the token positions of each token are sequentially encoded according to the token order to obtain the second sample annotation vector.

[0110] Taking the first sample annotation vector shown as an example, as shown in Figure 3 The token sequence in the first sample annotation vector is Figure 3 Figure 3 ​​The sequence of word pieces corresponding to the word piece of CLS, the word piece of Q1, the word piece of TURN, the word piece of R1, the word piece of TURN, the word piece of Q2, the word piece of TURN, the word piece of R2, the word piece of PARA, the word piece of R3, the word piece of SEP, the word piece of QF, the word piece of SEP, the word piece of RF, and the word piece of SEP, as shown in Figure 3 The order of the word pieces in the sequence of word pieces is CLS→Q1→TURN→R1→TURN→Q2→TURN→R2→PARA→R3→SEP→QF→SEP→RF→SEP, as shown in

[0111] In this scenario, the first sample label vector can be encoded according to the order of the word pieces, where a first position label value of a word piece position of a first word piece in the sequence of word pieces can be obtained according to the order of the word pieces.

[0112] As an example, in the scenario shown in Figure 3 In the first sample label vector shown, the first word piece in the sequence of word pieces is the word piece corresponding to CLS, and the label value of the word piece position to which the word piece belongs can be determined as the first position label value.

[0113] Further, a position label difference value between any two adjacent word piece positions in the word piece position of each word piece is obtained, and a second position label value of each of the remaining word piece positions except the first word piece position is obtained according to the first position label value and the position label difference value.

[0114] As an example, the first position label value of the word piece position of the first word piece is set to 0, and the position label difference value between any two adjacent word piece positions is set to 1. Starting from 0, the second position label values of the remaining word piece positions except the first word piece position can be obtained by sequentially adding 1.

[0115] For example, Figure 3The first sample label vector shown is an example. The first position label value of the first token corresponding to CLS is 0. The second position label value of the token corresponding to Q1 in the remaining token positions except the first token position is 1. The second position label value of the token corresponding to TURN is 2. The second position label value of the token corresponding to R1 is 3. The second position label value of the token corresponding to TURN is 4. The second position label value of the token corresponding to Q2 is 5. The second position label value of the token corresponding to TURN is 6. The second position label value of the token corresponding to R2 is 7. The second position label value of the token corresponding to PARA is 8. The second position label value of the token corresponding to R3 is 9. The second position label value of the token corresponding to SEP is 10. The second position label value of the token corresponding to QF is 11. The second position label value of the token corresponding to SEP is 12. The second position label value of the token corresponding to RF is 13. And the second position label value of the token corresponding to SEP is 14.

[0116] Optionally, the first position label value and each second position label value are sorted according to the token order to obtain a second sample label vector.

[0117] Based on the above example, the first position label value and each second position label value can be sorted according to the token order of each token to obtain Figure 3 The second sample label vector under the token dimension shown is 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14.

[0118] In the embodiments of the present disclosure, there are a question-asking role, an answering role, and a role corresponding to the first and second resolution information in the first sample multi-turn dialogue. In this scenario, the first sample label vector can be encoded according to the roles in the first sample multi-turn dialogue.

[0119] Among them, the first role label value of the question text, the second role label value of the answer text, and the third role label value of the first and second resolution information in the first sample label vector can be obtained, and the first sample label vector can be encoded according to the first role label value, the second role label value, and the third role label value to obtain a second sample label vector.

[0120] In the embodiments of the present disclosure, the role corresponding to the question text can be determined as the question-asking role, and the standard value used when labeling the question-asking role can be determined as the first role label value of the question text. Correspondingly, the role corresponding to the answer text can be determined as the answering role, and the standard value used when labeling the answering role can be determined as the second role label value of the answer text.

[0121] Correspondingly, the annotation value used when annotating the role corresponding to the first and second summary information is determined as the third role annotation value.

[0122] Optionally, the elements in the first sample annotation vector can be divided into roles, so as to identify the elements belonging to the question role, the elements belonging to the answer role, and the elements belonging to the summary role in the first sample annotation vector, and the elements belonging to the question role are annotated and encoded based on the first role annotation value, the elements belonging to the answer role are annotated and encoded based on the second role annotation value, and the elements belonging to the summary role are annotated and encoded based on the third role annotation value, thereby obtaining the encoded second sample annotation vector.

[0123] Among them, the role order between the question text, the answer text, the first summary information and the second summary information in the first sample annotation vector can be obtained, and the first role annotation value, the second role annotation value and the third role annotation value are sorted according to the role order, thereby obtaining the second sample annotation vector.

[0124] For example, as shown in the first sample annotation vector, Figure 3 The first role annotation value is set to 0, the second role annotation value is set to 1, and the third role annotation value is set to 2.

[0125] As shown in the first sample annotation vector, Figure 3 The elements belonging to the question role include CLS, Q1, TURN, Q2, TURN, which can be annotated and encoded based on the first role annotation value 0. The elements belonging to the answer role include R1, TURN, R2, PARA, R3, SEP, which can be annotated and encoded based on the second role annotation value 1. And the elements belonging to the summary role include QF, SEP, RF, SEP, which can be annotated and encoded based on the third role annotation value 2.

[0126] As shown in the first sample annotation vector, Figure 4 The role order between the question text, the answer text, the first summary information and the second summary information is CLS→Q1→TURN→R1→TURN→Q2→TURN→R2→PARA→R3→SEP→QF→SEP→RF→SEP, so the first role annotation value 0, the second role annotation value 1 and the third role annotation value 3 can be sorted based on the order, thereby obtaining Figure 4 The second sample annotation vector under the role dimension shown in the figure is 0,0,0,1,1,0,0,1,1,1,1,2,2,2,2.

[0127] S304, according to the first sample annotation vector and the second sample annotation vector, a sample annotation vector set is obtained.

[0128] In the embodiments of the present disclosure, the second sample annotation vector is a plurality of vectors obtained by performing multi-round annotation encoding on the first sample annotation vector.

[0129] In this scenario, the first sample annotation vector and the second sample annotation vector can be combined to obtain a sample annotation vector set for training the candidate quality evaluation model.

[0130] S305, according to the sample annotation vector set, the candidate quality evaluation model is trained until the training is completed, and a trained target quality evaluation model is obtained.

[0131] Optionally, the sample annotation vector set can be input into the candidate quality evaluation model to obtain a first quality evaluation result output by the candidate quality evaluation model.

[0132] As an example, as shown in Figure 4 , the sample annotation vector set can be input into the candidate quality evaluation model as shown in Figure 4 , and features of relevant dimensions such as morphology, syntax and semantics in the sample annotation vector are extracted by a language model in the candidate quality evaluation model.

[0133] Further, based on the features proposed by the language model, the FC layer and the softmax layer as shown in Figure 5 perform quality evaluation of the first sample multi-round dialogue, thereby outputting the first quality evaluation result as shown in Figure 5 .

[0134] It should be noted that the first quality evaluation result can be a three-classification result, wherein the classification items in the three classification can include high-quality solution, solved and unsolved, or can include other quality evaluation items of multi-round dialogue, which is not limited here.

[0135] Optionally, a first evaluation label of the first sample multi-round dialogue is obtained, and based on the first quality evaluation result and the first evaluation label, a first training loss of the candidate quality evaluation model is obtained.

[0136] The label of the first sample multi-round dialogue can be determined as the first evaluation label, and in this scenario, the first evaluation label and the first quality evaluation result of the first sample multi-round dialogue can be algorithmically processed based on the loss value obtaining algorithm in the related art, thereby obtaining the loss value of the first quality evaluation result based on the first evaluation label, and the loss value is determined as the first training loss.

[0137] Optionally, the candidate quality evaluation model is adjusted according to the first training loss, and the candidate quality evaluation model after parameter adjustment is returned to obtain the next first sample multi-round dialogue for continuous training until the training is completed, and a trained target quality evaluation model is obtained.

[0138] The training end condition of the candidate quality evaluation model can be set based on the training rounds. For the model training of the current round, if the current round meets the preset training end condition, the training of the candidate quality evaluation model can be ended, and the model obtained at the end of the last round of training is determined as the trained target quality evaluation model.

[0139] Correspondingly, the training end condition of the candidate quality evaluation model can be set based on the training output. For the model training of the current round, if the model output result of the current round meets the preset training end condition, the training of the candidate quality evaluation model can be ended, and the model obtained at the end of the last round of training is determined as the trained target quality evaluation model.

[0140] The training method of the multi-round dialogue quality evaluation model provided by the present disclosure obtains question text and answer text in a first sample multi-round dialogue, as well as first reduction information of the question text and second reduction information of the answer text. The question text, the answer text, the first reduction information, and the second reduction information are spliced to obtain a first sample annotation vector. The first sample annotation vector is annotated to obtain a second sample annotation vector, so as to obtain a sample annotation vector set composed of the first sample annotation vector and the second sample annotation vector. The candidate quality evaluation model is trained through the sample annotation vector set to obtain a trained target quality evaluation model. In the present disclosure, the first sample multi-round dialogue is finely annotated through the first sample annotation vector and the second sample annotation vector, the learning difficulty of the candidate quality evaluation model in learning the expression of related dimensions such as morphology, syntax, and semantics in the first sample multi-round dialogue is reduced, the learning effect of the candidate quality evaluation model on the first sample multi-round dialogue is improved, and the training effect of the candidate quality evaluation model is optimized, thereby improving the precision and accuracy of the quality evaluation of the multi-round dialogue by the target quality evaluation model based on the candidate quality evaluation model.

[0141] In the above embodiments, regarding the training of the initial quality evaluation model, the training of the initial quality evaluation model can be combined with Figure 5 It is further understood that Figure 6 The flowchart of the training method of the multi-round dialogue quality evaluation model of another embodiment of the present disclosure is shown in FIG. 5B. Figure 6 The method comprises the following steps.

[0142] S501, obtaining a second sample multi-round dialogue of an initial quality evaluation model, and a second evaluation label of the second sample multi-round dialogue.

[0143] In the embodiments of the present disclosure, the historical multi-round dialogue of a sample user can be obtained, and the historical multi-round dialogue is taken as the second sample multi-round dialogue of the initial quality evaluation model.

[0144] Optionally, the second sample multi-turn dialogue can be subjected to quality evaluation, and the quality evaluation result can be taken as a second evaluation label of the second sample multi-turn dialogue.

[0145] The second evaluation label can be obtained by performing multi-dimensional quality evaluation on the second sample multi-turn dialogue. For example, the degree of solving the question raised by the question text to the answer text in the second sample multi-turn dialogue can be evaluated, and the correctness and professionalism of the second sample multi-turn dialogue can also be evaluated. Here, no specific limitation is given.

[0146] S502, inputting the second sample multi-turn dialogue into the initial quality evaluation model to obtain a second quality evaluation result output by the initial quality evaluation model.

[0147] Optionally, the second sample multi-turn dialogue can be input into the initial quality evaluation model to extract features such as lexical, syntactic and semantic related features in the second sample multi-turn dialogue by the initial quality evaluation model.

[0148] Further, based on the extracted features, quality evaluation of the second sample multi-turn dialogue is performed, and the quality evaluation result of the second sample multi-turn dialogue output by the initial quality evaluation model is obtained through the FC layer and the softmax layer of the initial quality evaluation model, and the result is marked as the second quality evaluation result.

[0149] S503, obtaining a second training loss of the initial quality evaluation model based on the second quality evaluation result and the second evaluation result label.

[0150] Optionally, the second quality evaluation result and the second evaluation result label can be subjected to loss value algorithm processing based on the loss value algorithm in the related art, and then the training loss of the second quality evaluation result based on the second evaluation result label is obtained as the second training loss according to the result of the algorithm processing.

[0151] The loss value algorithm can be a classification loss value algorithm or other types of loss value algorithms, and no specific limitation is given here.

[0152] S504, adjusting parameters of the initial quality evaluation model according to the second training loss, and returning to obtain a next second sample multi-turn dialogue to continue training the adjusted initial quality evaluation model until the training is completed, and obtaining a trained candidate quality evaluation model.

[0153] Optionally, the training end condition of the initial quality evaluation model can be set based on the training round. For the model training of the current round, if the round meets the preset training end condition, the training of the initial quality evaluation model can be ended, and the model obtained at the end of the last round of training is determined as the trained candidate quality evaluation model.

[0154] Optionally, the initial quality evaluation model can be set based on the training output, and the training end condition of the initial quality evaluation model can be set. For the model training of the current round, if the model output result of the round meets the preset training end condition, the training of the initial quality evaluation model can be ended, and the model obtained at the end of the last round of training is determined as the trained candidate quality evaluation model.

[0155] The training method of the quality evaluation model provided by the present disclosure obtains an initial quality evaluation model to be trained, and trains the initial quality evaluation model through a second sample multi-round dialogue until the training is completed, thereby obtaining a trained candidate quality evaluation model. In the present disclosure, the initial quality evaluation model is trained through the second sample multi-round dialogue, so that the trained candidate quality evaluation model has the quality evaluation capability of the multi-round dialogue, thereby providing a basis for the secondary training of the subsequent model.

[0156] The present disclosure also provides a quality evaluation method for a multi-round dialogue, which can be combined with Figure 6 understanding, Figures 1 to 5 The flowchart of the quality evaluation method for a multi-round dialogue according to an embodiment of the present disclosure is shown in Figure 7 as shown, the method comprises the following steps:

[0157] S601, obtaining a trained target multi-round dialogue quality evaluation model.

[0158] In the embodiment of the present disclosure, the model for quality evaluation of a multi-round dialogue can be determined as the trained target multi-round dialogue quality evaluation model.

[0159] The target multi-round dialogue quality evaluation model is obtained by Figure 7 the training method of the quality evaluation model according to the embodiment.

[0160] S602, obtaining a multi-round dialogue to be evaluated, and inputting the multi-round dialogue into the target multi-round dialogue quality evaluation model to obtain a quality evaluation result output by the target multi-round dialogue quality evaluation model.

[0161] Optionally, the multi-round dialogue to be evaluated can be input into the trained target multi-round dialogue quality evaluation model, and the expression features of the lexical, syntactic and semantic dimensions in the multi-round dialogue can be extracted by the language model in the target multi-round dialogue quality model.

[0162] Further, the extracted features are quality evaluated based on the FC layer and the softmax layer in the target multi-round dialogue quality evaluation model, thereby obtaining the quality evaluation result of the multi-round dialogue to be evaluated.

[0163] The quality evaluation method of the multi-turn dialogue provided in the present disclosure comprises the following steps: obtaining a trained target multi-turn dialogue quality evaluation model, inputting a multi-turn dialogue to be evaluated into the target multi-turn dialogue quality evaluation model, and then obtaining a quality evaluation result output by the target multi-turn dialogue quality evaluation model. In the present disclosure, the quality evaluation of the multi-turn dialogue is realized through the target multi-turn dialogue quality evaluation model. Compared with the quality evaluation of the multi-turn dialogue realized by relying on artificial means, the degree of artificial dependence and the artificial cost are reduced, the quality evaluation precision and the quality evaluation efficiency of the multi-turn dialogue are improved, the quality evaluation method of the multi-turn dialogue is optimized, and accurate data support is provided for downstream tasks.

[0164] Corresponding to the training method of the quality evaluation model proposed in the above several embodiments, one embodiment of the present disclosure further proposes a training device of a quality evaluation model. Since the training device of the quality evaluation model proposed in the embodiment of the present disclosure corresponds to the training method of the quality evaluation model proposed in the above several embodiments, the implementation modes of the training method of the quality evaluation model described above are also applicable to the training device of the quality evaluation model proposed in the embodiment of the present disclosure, which will not be described in detail in the following embodiments.

[0165] Figure 8 The structure diagram of the training device of the quality evaluation model of one embodiment of the present disclosure is shown in FIG. 7. Figure 8 As shown in FIG. 7, the training device 700 of the quality evaluation model comprises a first training module 71, a first acquisition module 72 and a second training module 73, wherein:

[0166] The first training module 71 is configured to obtain an initial quality evaluation model to be trained, and perform model training on the initial quality evaluation model to obtain a trained candidate quality evaluation model.

[0167] The first acquisition module 72 is configured to obtain a first sample multi-turn dialogue and label the first sample multi-turn dialogue to obtain a sample label vector set of the first sample multi-turn dialogue.

[0168] The second training module 73 is configured to train the candidate quality evaluation model according to the sample label vector set until the training is completed, and obtain a trained target quality evaluation model.

[0169] In the embodiment of the present disclosure, the first acquisition module 72 is further configured to obtain question text in the first sample multi-turn dialogue and first reduction information of the question text, and obtain answer text in the first sample multi-turn dialogue and second reduction information of the question and answer text. The question text, the answer text, the first reduction information and the second reduction information are spliced to obtain a first sample label vector of the first sample multi-turn dialogue. The first sample label vector is labeled to obtain a second sample label vector of the first sample label vector. The sample label vector set is obtained according to the first sample label vector and the second sample label vector.

[0170] In the embodiments of the present disclosure, the first obtaining module 72 is further configured to: obtain a first reduction item of the pre-set question text and a first feature value of the question text under the first reduction item. The first feature value is binned to obtain a first reduction identifier of the first reduction item. The first reduction information of the question text is obtained according to the first reduction item and the first reduction identifier.

[0171] In the embodiments of the present disclosure, the first obtaining module 72 is further configured to: obtain a second reduction item of the pre-set answer text and a second feature value of the answer text under the second reduction item. The second feature value is binned to obtain a second reduction identifier of the second reduction item. The second reduction information of the answer text is obtained according to the second reduction item and the second reduction identifier.

[0172] In the embodiments of the present disclosure, the first obtaining module 72 is further configured to: obtain a question round to which the question text belongs, and generate a first annotation symbol of the question text according to the question round, the first annotation symbol including a question round serial number of the question text. Obtain an answer round to which the answer text belongs, and generate a second annotation symbol of the answer text according to the answer round, the second annotation symbol including an answer round serial number of the answer text. Obtain a third annotation symbol of the first reduction information and the second reduction information. The question text, the answer text, the first reduction information and the second reduction information are spliced based on the first annotation symbol, the second annotation symbol and the third annotation symbol to obtain a first sample annotation vector of the first sample multi-round dialogue.

[0173] In the embodiments of the present disclosure, the first obtaining module 72 is further configured to: obtain a splicing order between each question text, each answer text, the first reduction information and the second reduction information in the first sample multi-round dialogue; annotate the question text through the first annotation symbol to obtain a first annotated segment; annotate the answer text through the second annotation symbol to obtain a second annotated segment; annotate the first reduction information and / or the second reduction information through the third annotation symbol suffix pair to obtain a third annotated segment; obtain a pre-set splicing symbol, and splice the first annotated segment, the second annotated segment and the third annotated segment according to the splicing order through the splicing symbol to obtain the first sample annotation vector.

[0174] In the embodiments of the present disclosure, the first obtaining module 72 is further configured to: in response to two question texts each having a first annotated segment and / or two answer texts each having a second annotated segment, obtain a pre-configured fourth annotation symbol. The fourth annotation symbol is added between the first annotated segments of the two question texts and / or between the second annotated segments of the two answer texts to obtain an added first sample annotation vector.

[0175] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain first region annotation values of the question text and the answer text, second region annotation values of the first resolution information, and third region annotation values of the second resolution information in the first sample annotation vector. The first sample annotation vector is encoded according to the first region annotation values, the second region annotation values, and the third region annotation values to obtain a second sample annotation vector.

[0176] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain a region order between the question text and the answer text, the first resolution information, and the second resolution information in the first sample annotation vector. The first region annotation values, the second region annotation values, and the third region annotation values are sorted according to the region order to obtain the second sample annotation vector.

[0177] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain a token order of each token in the token sequence of the first sample annotation vector. The token positions of each token are sequentially encoded according to the token order to obtain the second sample annotation vector.

[0178] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain a first position annotation value of a token position of a first token in the token sequence according to the token order. A position annotation difference value between any two adjacent token positions of each token is obtained. A second position annotation value of each of the remaining token positions except the first token position is obtained according to the first position annotation value and the position annotation difference value. The first position annotation value and each second position annotation value are sorted according to the token order to obtain the second sample annotation vector.

[0179] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain first role annotation values of the question text, second role annotation values of the answer text, and third role annotation values of the first resolution information and the second resolution information in the first sample annotation vector. The first sample annotation vector is encoded according to the first role annotation values, the second role annotation values, and the third role annotation values to obtain a second sample annotation vector.

[0180] In the embodiment of the present disclosure, the first obtaining module 72 is further configured to: obtain a role order between the question text, the answer text, and the first resolution information and the second resolution information in the first sample annotation vector. The first role annotation values, the second role annotation values, and the third role annotation values are sorted according to the role order to obtain the second sample annotation vector.

[0181] In the embodiment of the present disclosure, the second training module 73 is further configured to input the sample label vector set into the candidate quality evaluation model, obtain a first quality evaluation result output by the candidate quality evaluation model, obtain a first evaluation label of the first sample multi-turn dialogue, and obtain a first training loss of the candidate quality evaluation model based on the first quality evaluation result and the first evaluation label. The candidate quality evaluation model is adjusted according to the first training loss, and the candidate quality evaluation model adjusted based on the next first sample multi-turn dialogue is obtained to continue training until the training is completed, and a trained target quality evaluation model is obtained.

[0182] In the embodiment of the present disclosure, the first training module 71 is further configured to obtain a second sample multi-turn dialogue of the initial quality evaluation model and a second evaluation label of the second sample multi-turn dialogue. The second sample multi-turn dialogue is input into the initial quality evaluation model to obtain a second quality evaluation result output by the initial quality evaluation model. The second training loss of the initial quality evaluation model is obtained based on the second quality evaluation result and the second evaluation label. The initial quality evaluation model is adjusted according to the second training loss, and the initial quality evaluation model adjusted based on the next second sample multi-turn dialogue is obtained to continue training until the training is completed, and a trained candidate quality evaluation model is obtained.

[0183] The training device of the multi-turn dialogue quality evaluation model provided by the present disclosure trains the initial quality evaluation model to be trained to obtain a trained candidate quality evaluation model, obtains a first sample multi-turn dialogue, labels the first sample multi-turn dialogue to obtain a sample label vector set, and performs secondary model training on the candidate quality evaluation model according to the sample label vector set until the training is completed, and a trained target quality evaluation model is obtained. In the present disclosure, the candidate quality evaluation model is obtained by training the initial quality evaluation model, and the candidate quality evaluation model is trained again by the sample label vector obtained by labeling the first sample multi-turn dialogue, which optimizes the semantic learning effect of the candidate quality evaluation model on the first sample multi-turn dialogue, improves the quality evaluation capability of the candidate quality evaluation model on the first sample multi-turn dialogue, and further optimizes the training method and training effect of the quality evaluation model. In the scene of quality evaluation of multi-turn dialogue by the target quality evaluation model, the quality evaluation efficiency and precision of the multi-turn dialogue are improved, the degree of artificial dependence and the cost of artificial are reduced compared with the quality evaluation of multi-turn dialogue relying on artificial, and accurate data support is provided for downstream tasks.

[0184] Corresponding to the multi-turn dialogue quality evaluation method proposed in the above several embodiments, an embodiment of the present disclosure also proposes a multi-turn dialogue quality evaluation device. Since the multi-turn dialogue quality evaluation device proposed in the embodiment of the present disclosure corresponds to the multi-turn dialogue quality evaluation method proposed in the above several embodiments, the implementation manner of the multi-turn dialogue quality evaluation method described above is also applicable to the multi-turn dialogue quality evaluation device proposed in the embodiment of the present disclosure, which will not be described in detail in the following embodiments.

[0185] Figure 7 The structural schematic diagram of the multi-turn dialogue quality evaluation device of an embodiment of the present disclosure is shown in FIG. 8. As shown in FIG. 8, the multi-turn dialogue quality evaluation device 800 includes a second acquisition module 81 and an evaluation module 82. Figure 9

[0186] The second acquisition module 81 is configured to acquire a trained target multi-turn dialogue quality evaluation model, wherein the target multi-turn dialogue quality evaluation model is obtained by the training device of the quality evaluation model proposed in the embodiment. Figure 9

[0187] The evaluation module 82 is configured to acquire a multi-turn dialogue to be evaluated, and input the multi-turn dialogue into the target multi-turn dialogue quality evaluation model to obtain a quality evaluation result output by the target multi-turn dialogue quality evaluation model.

[0188] The multi-turn dialogue quality evaluation device proposed in the present disclosure acquires a trained target multi-turn dialogue quality evaluation model, and inputs a multi-turn dialogue to be evaluated into the target multi-turn dialogue quality evaluation model, and then obtains a quality evaluation result output by the target multi-turn dialogue quality evaluation model. In the present disclosure, the quality evaluation of the multi-turn dialogue is realized by the target multi-turn dialogue quality evaluation model. Compared with the quality evaluation of the multi-turn dialogue realized by relying on artificial, the degree of artificial dependence and the artificial cost are reduced, the quality evaluation precision and the quality evaluation efficiency of the multi-turn dialogue are improved, the quality evaluation method of the multi-turn dialogue is optimized, and then accurate data support is provided for downstream tasks.

[0189] According to the embodiments of the present disclosure, the present disclosure also proposes an electronic device, a readable storage medium and a computer program product.

[0190] ​ ​​A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0191] like ​ As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0192] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 906, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0193] The computing unit 901 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 901 performs various methods and processes described above, such as the training method of the quality assessment model and / or the multi-turn dialogue quality assessment method. For example, in some embodiments, the training method of the quality assessment model and / or the multi-turn dialogue quality assessment method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded onto the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the quality assessment model and / or the multi-turn dialogue quality assessment method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the training method of the quality assessment model and / or the multi-turn dialogue quality assessment method by any other appropriate means, such as by means of firmware.

[0194] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0195] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can execute entirely on a computing device, partially on the computing device, as a stand-alone software package, partially on the computing device and partially on a remote computing device or entirely on the remote computing device or server. These program codes can be offered for sale, transfer, license or distribution to end users.

[0196] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0197] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0198] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user account computer having a graphical user account interface or a web browser through which a user account can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0199] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically happening over a communication network. This relationship between a client and a server is created by executing computer programs on the respective computers with the client and server programs interacting across a data communication which can be any form or medium of digital data communication. The server can be a cloud server, a server of a distributed system, or a server incorporating blockchain.

[0200] It should be understood that various forms of flow shown above can be used with orders of the steps being re-sequenced, added, or deleted. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure are achieved, which are not limited herein.

[0201] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A training method for a quality assessment model, wherein, The method includes: Obtain the initial quality assessment model to be trained, and train the initial quality assessment model to obtain the trained candidate quality assessment model. The process involves acquiring the question text and answer text from a first sample multi-turn dialogue, extracting the first feature value of the question text under a preset first resolution term, binning it to obtain a first resolution identifier, and generating first resolution information based on the first resolution term and the first resolution identifier. It also involves extracting the second feature value of the answer text under a preset second resolution term, binning it to obtain a second resolution identifier, and generating second resolution information based on the second resolution term and the second resolution identifier. The process further involves labeling the question text, answer text, first resolution information, and second resolution information to obtain a set of sample label vectors for the first sample multi-turn dialogue. The first resolution term is obtained by evaluating and summarizing the question text based on three dimensions: question classification, user rating, and question value. The second resolution term is obtained by evaluating and summarizing the answer text based on two dimensions: answerer authentication domain and user feedback. The sample annotation vector set includes a first sample annotation vector and a second sample annotation vector; the first sample annotation vector is obtained by generating a first annotation symbol based on the question turn of the question text, generating a second annotation symbol based on the answer turn of the answer text, and generating a third annotation symbol based on the first resolution information and the second resolution information; based on the first annotation symbol, the second annotation symbol, and the third annotation symbol, the question text, the answer text, the first resolution information, and the second resolution information are concatenated to obtain the first sample annotation vector of the first sample multi-turn dialogue; The candidate quality assessment model is trained based on the sample label vector set. During the training process, the feature weights corresponding to the first and second resolution information are strengthened. The model parameters are adjusted based on the association loss between the resolution identifier and the evaluation label in the sample label vector until the training ends, and the trained target quality assessment model is obtained.

2. The method according to claim 1, wherein, The annotation of the question text, answer text, first resolution information, and second resolution information yields a set of sample annotation vectors for the first sample multi-turn dialogue, including: The question text, the answer text, the first resolution information, and the second resolution information are concatenated to obtain the first sample annotation vector of the first sample multi-turn dialogue; The first sample annotation vector is annotated to obtain the second sample annotation vector of the first sample annotation vector; The sample annotation vector set is obtained based on the first sample annotation vector and the second sample annotation vector.

3. The method according to claim 1, wherein, The first annotation symbol includes the question round number of the question text, and the second annotation symbol includes the answer round number of the answer text.

4. The method according to claim 1, wherein, The first sample annotation vector of the first sample multi-turn dialogue is obtained by concatenating the question text, the answer text, the first resolution information, and the second resolution information based on the first annotation symbol, the second annotation symbol, and the third annotation symbol, including: Obtain the concatenation order between each question text, each answer text, the first resolution information, and the second resolution information in the first sample multi-turn dialogue; The first annotated segment is obtained by annotating the question text using the first annotation symbol; The second annotation segment is obtained by annotating the answer text using the second annotation symbol; The third labeled fragment is obtained by labeling the first and / or second resolution information of the third labeling symbol suffix pair; Obtain a preset splicing symbol, and splice the first annotation fragment, the second annotation fragment, and the third annotation fragment according to the splicing order using the splicing symbol to obtain the first sample annotation vector.

5. The method according to claim 4, wherein, The method further includes: In response to two adjacent spliced ​​segments being the first labeled segments of the two question texts and / or the second labeled segments of the two answer texts, a pre-configured fourth labeled symbol is obtained; The fourth annotation symbol is added between the first annotation segments of each of the two question texts, and / or between the second annotation segments of each of the two answer texts, to obtain the first sample annotation vector after the addition.

6. The method according to claim 2, wherein, The step of annotating the first sample annotation vector to obtain the second sample annotation vector of the first sample annotation vector includes: Obtain the first region annotation value of the question text and the answer text, the second region annotation value of the first resolution information, and the third region annotation value of the second resolution information in the first sample annotation vector; The first sample annotation vector is encoded based on the first region annotation value, the second region annotation value, and the third region annotation value to obtain the second sample annotation vector.

7. The method according to claim 6, wherein, The step of encoding the first sample annotation vector based on the first region annotation value, the second region annotation value, and the third region annotation value to obtain the second sample annotation vector includes: Obtain the region order between the question text and the answer text, and between the first resolution information and the second resolution information in the first sample annotation vector; According to the region order, the label values ​​of the first region, the label values ​​of the second region, and the label values ​​of the third region are sorted to obtain the second sample label vector.

8. The method according to claim 2, wherein, The step of annotating the first sample annotation vector to obtain the second sample annotation vector of the first sample annotation vector includes: Obtain the word order of each word in the word sequence of the first sample labeled vector; The positions of each word are sequentially encoded according to the word order to obtain the second sample annotation vector.

9. The method according to claim 8, wherein, The step of sequentially encoding the position of each word according to the word order to obtain the second sample annotation vector includes: Based on the word order, obtain the first position label value of the word position of the first word in the word sequence; Obtain the positional label difference between any two adjacent word positions in each word position; Based on the first position annotation value and the position annotation difference, obtain the second position annotation value for each of the remaining word positions except the first word position; Based on the word order, the first position annotation value and each second position annotation value are sorted to obtain the second sample annotation vector.

10. The method according to claim 2, wherein, The step of annotating the first sample annotation vector to obtain the second sample annotation vector of the first sample annotation vector includes: Obtain the first role annotation value of the question text, the second role annotation value of the answer text, and the third role annotation value of the first resolution information and the second resolution information in the first sample annotation vector; The first sample annotation vector is encoded based on the first role annotation value, the second role annotation value, and the third role annotation value to obtain the second sample annotation vector.

11. The method according to claim 10, wherein, The step of encoding the first sample annotation vector based on the first role annotation value, the second role annotation value, and the third role annotation value to obtain the second sample annotation vector includes: Obtain the role order among the question text, the answer text, the first resolution information, and the second resolution information in the first sample annotation vector; The first role annotation value, the second role annotation value, and the third role annotation value are sorted according to the role order to obtain the second sample annotation vector.

12. The method according to claim 1, wherein, The step of training the candidate quality assessment model based on the sample labeled vector set until training is complete, to obtain the trained target quality assessment model, includes: Input the set of sample labeled vectors into the candidate quality assessment model to obtain the first quality assessment result output by the candidate quality assessment model; Obtain the first evaluation label of the first sample multi-turn dialogue, and based on the first quality evaluation result and the first evaluation label, obtain the first training loss of the candidate quality evaluation model; The candidate quality assessment model is adjusted based on the first training loss, and the process continues to train the candidate quality assessment model after parameter adjustment by obtaining the next first sample multi-round dialogue until the training ends, resulting in a trained target quality assessment model.

13. The method according to claim 1, wherein, The step of obtaining an initial quality assessment model to be trained and training the initial quality assessment model to obtain a trained candidate quality assessment model includes: Obtain the second sample multi-turn dialogue of the initial quality assessment model, and the second assessment result label of the second sample multi-turn dialogue; The second sample multi-turn dialogue is input into the initial quality assessment model to obtain the second quality assessment result output by the initial quality assessment model; Based on the second quality assessment result and the second assessment result label, the second training loss of the initial quality assessment model is obtained; The parameters of the initial quality assessment model are adjusted according to the second training loss, and the next second sample is obtained. The multi-round dialogue continues to train the adjusted initial quality assessment model until the training ends, and the trained candidate quality assessment model is obtained.

14. A method for assessing the quality of multi-turn dialogues, wherein, The method includes: A trained target multi-turn dialogue quality assessment model is obtained, wherein the target multi-turn dialogue quality assessment model is obtained by the training method of the quality assessment model according to any one of claims 1-13; The multi-turn dialogue to be evaluated is obtained, and the multi-turn dialogue is input into the target multi-turn dialogue quality evaluation model to obtain the quality evaluation result output by the target multi-turn dialogue quality evaluation model.

15. A training device for a quality assessment model, wherein, The device includes: The first training module is used to obtain the initial quality assessment model to be trained and to train the initial quality assessment model to obtain the trained candidate quality assessment model. The first acquisition module is used to acquire the question text and answer text in a first sample multi-turn dialogue; extract the first feature value of the question text under a preset first resolution term and bin it to obtain a first resolution identifier; generate first resolution information based on the first resolution term and the first resolution identifier; extract the second feature value of the answer text under a preset second resolution term and bin it to obtain a second resolution identifier; generate second resolution information based on the second resolution term and the second resolution identifier; and annotate the question text, answer text, first resolution information, and second resolution information to obtain a sample annotation vector set of the first sample multi-turn dialogue; wherein, the first resolution term is obtained by evaluating and summarizing the question text based on three dimensions: question classification, user rating, and question value; and the second resolution term is obtained by evaluating and summarizing the answer text based on two dimensions: answerer authentication domain and user feedback. The sample annotation vector set includes a first sample annotation vector and a second sample annotation vector; the first sample annotation vector is obtained by generating a first annotation symbol based on the question turn of the question text, generating a second annotation symbol based on the answer turn of the answer text, and generating a third annotation symbol based on the first resolution information and the second resolution information; based on the first annotation symbol, the second annotation symbol, and the third annotation symbol, the question text, the answer text, the first resolution information, and the second resolution information are concatenated to obtain the first sample annotation vector of the first sample multi-turn dialogue; The second training module is used to train the candidate quality assessment model based on the sample label vector set. During the training process, the feature weights corresponding to the first and second resolution information are strengthened, and the model parameters are adjusted based on the association loss between the resolution identifier and the evaluation label in the sample label vector until the training ends, and the trained target quality assessment model is obtained.

16. The apparatus according to claim 15, wherein, The first acquisition module is further configured to: The question text, the answer text, the first resolution information, and the second resolution information are concatenated to obtain the first sample annotation vector of the first sample multi-turn dialogue; The first sample annotation vector is annotated to obtain the second sample annotation vector of the first sample annotation vector; The sample annotation vector set is obtained based on the first sample annotation vector and the second sample annotation vector.

17. The apparatus according to claim 15, wherein, The first annotation symbol includes the question round number of the question text, and the second annotation symbol includes the answer round number of the answer text.

18. The apparatus according to claim 15, wherein, The first acquisition module is further configured to: Obtain the concatenation order between each question text, each answer text, the first resolution information, and the second resolution information in the first sample multi-turn dialogue; The first annotated segment is obtained by annotating the question text using the first annotation symbol; The second annotation segment is obtained by annotating the answer text using the second annotation symbol; The third labeled fragment is obtained by labeling the first and / or second resolution information of the third labeling symbol suffix pair; Obtain a preset splicing symbol, and splice the first annotation fragment, the second annotation fragment, and the third annotation fragment according to the splicing order using the splicing symbol to obtain the first sample annotation vector.

19. The apparatus according to claim 18, wherein, The first acquisition module is further configured to: In response to two adjacent spliced ​​segments being the first labeled segments of the two question texts and / or the second labeled segments of the two answer texts, a pre-configured fourth labeled symbol is obtained; The fourth annotation symbol is added between the first annotation segments of each of the two question texts, and / or between the second annotation segments of each of the two answer texts, to obtain the first sample annotation vector after the addition.

20. The apparatus according to claim 16, wherein, The first acquisition module is further configured to: Obtain the first region annotation value of the question text and the answer text, the second region annotation value of the first resolution information, and the third region annotation value of the second resolution information in the first sample annotation vector; The first sample annotation vector is encoded based on the first region annotation value, the second region annotation value, and the third region annotation value to obtain the second sample annotation vector.

21. The apparatus according to claim 20, wherein, The first acquisition module is further configured to: Obtain the region order between the question text and the answer text, and between the first resolution information and the second resolution information in the first sample annotation vector; According to the region order, the label values ​​of the first region, the label values ​​of the second region, and the label values ​​of the third region are sorted to obtain the second sample label vector.

22. The apparatus according to claim 16, wherein, The first acquisition module is further configured to: Obtain the word order of each word in the word sequence of the first sample labeled vector; The positions of each word are sequentially encoded according to the word order to obtain the second sample annotation vector.

23. The apparatus according to claim 22, wherein, The first acquisition module is further configured to: Based on the word order, obtain the first position label value of the word position of the first word in the word sequence; Obtain the positional label difference between any two adjacent word positions in each word position; Based on the first position annotation value and the position annotation difference, obtain the second position annotation value for each of the remaining word positions except the first word position; Based on the word order, the first position annotation value and each second position annotation value are sorted to obtain the second sample annotation vector.

24. The apparatus according to claim 16, wherein, The first acquisition module is further configured to: Obtain the first role annotation value of the question text, the second role annotation value of the answer text, and the third role annotation value of the first resolution information and the second resolution information in the first sample annotation vector; The first sample annotation vector is encoded based on the first role annotation value, the second role annotation value, and the third role annotation value to obtain the second sample annotation vector.

25. The apparatus according to claim 24, wherein, The first acquisition module is further configured to: Obtain the role order among the question text, the answer text, the first resolution information, and the second resolution information in the first sample annotation vector; The first role annotation value, the second role annotation value, and the third role annotation value are sorted according to the role order to obtain the second sample annotation vector.

26. The apparatus according to claim 15, wherein, The second training module is also used for: Input the set of sample labeled vectors into the candidate quality assessment model to obtain the first quality assessment result output by the candidate quality assessment model; Obtain the first evaluation label of the first sample multi-turn dialogue, and based on the first quality evaluation result and the first evaluation label, obtain the first training loss of the candidate quality evaluation model; The candidate quality assessment model is adjusted based on the first training loss, and the process continues to train the candidate quality assessment model after parameter adjustment by obtaining the next first sample multi-round dialogue until the training ends, resulting in a trained target quality assessment model.

27. The apparatus according to claim 15, wherein, The first training module is also used for: Obtain the second sample multi-turn dialogue of the initial quality assessment model, and the second assessment result label of the second sample multi-turn dialogue; The second sample multi-turn dialogue is input into the initial quality assessment model to obtain the second quality assessment result output by the initial quality assessment model; Based on the second quality assessment result and the second assessment result label, the second training loss of the initial quality assessment model is obtained; The parameters of the initial quality assessment model are adjusted according to the second training loss, and the next second sample is obtained. The multi-round dialogue continues to train the adjusted initial quality assessment model until the training ends, and the trained candidate quality assessment model is obtained.

28. A multi-turn dialogue quality assessment device, wherein, The device includes: The second acquisition module is used to acquire a trained target multi-turn dialogue quality assessment model, wherein the target multi-turn dialogue quality assessment model is obtained through the training device of the quality assessment model according to any one of claims 15-27. The evaluation module is used to acquire the multi-turn dialogue to be evaluated, input the multi-turn dialogue into the target multi-turn dialogue quality evaluation model, and obtain the quality evaluation result output by the target multi-turn dialogue quality evaluation model.

29. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13 and / or claim 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13 and / or claim 14.

31. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13 and / or claim 14.

Images