Multi-language man-machine conversation scoring system
By introducing semantic cutting and multi-dimensional scoring modules in the multilingual human-computer dialogue scoring system, the problem that existing systems are difficult to understand the semantics of different natural language texts is solved, and the scoring accuracy and credibility are achieved, and detailed error feedback supports model improvements are provided.
Patent Information
- Application Number
- CN202411690930.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-18
AI Technical Summary
The existing multilingual human-computer dialogue scoring system is difficult to fully and accurately understand the text semantics of different natural languages, resulting in low scoring accuracy and only outputs the overall score, lacking specific error feedback.
A multilingual human-computer dialogue scoring system is proposed, including pre-training database, semantic cutting module, scoring module and output module. The semantic cutting module cuts the language sample problem into multiple semantic words. The scoring module uses semantic words as the minimum comparison unit, compares the dialogue model answers with the reference answers, and scores according to the semantic similarity. The output module outputs the overall score and the score of each minimum comparison unit.
It improves the accuracy and credibility of multilingual human-computer dialogue scores, can conduct detailed analysis of users' answers, provide user scores, errors and reasons, and supports improvement and optimization of human-computer dialogue models.
Smart Images
Figure CN119645790A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of human-computer dialogue, and in particular to a multi-language human-computer dialogue scoring system. Background Art
[0002] Humans can use interactive software to conduct human-computer dialogues, which are referred to as "automated assistants" in this article (also known as "chatbots", "interactive personal assistants", "intelligent personal assistants", "personal voice assistants", "dialog agents", etc.). For example, humans (when they communicate with automated assistants, they can be referred to as "users") using free-form natural language input to provide commands, queries and / or requests (collectively referred to as "queries" in this article) which can be voice utterances converted into text and then processed and / or typed in free-form natural language input. Automated assistants typically use a predetermined voice to invoke utterances (e.g., "OK Assistant"), and typically perform various types of processing only on those voice utterances, such as speech to text processing and / or natural language processing following the invocation phrase. Automated assistants are good at communicating with users in some widely used languages, such as English, but have poor ability to communicate in other languages. However, traditional techniques for configuring automated assistants to communicate in new languages are resource intensive. For each new language, it may be necessary to generate new trigger grammar (i.e., words or phrases designed to invoke specific actions through the automated assistant), recognize query intent in the new language (which requires different natural language processing components for different languages), and generate natural language output that the new language can understand. Adding to the challenge, each language has countless idiosyncrasies (e.g., morphologically rich, assent-decline, gender-neutral, etc.), and linguistic experts who can build these systems in a specific language may be difficult to find.
[0003] Therefore, multilingual human-computer interaction models need to be continuously updated and improved to improve language recognition accuracy. At present, the existing automatic scoring system technology is mainly based on natural language processing technology, such as bag-of-words model, topic model, word vector and neural network, which can convert text information into mathematical form, extract the main information of the text, and then score according to predefined scoring rules. However, due to the complexity of different natural languages, it is usually difficult to fully and accurately understand the semantics of the text, resulting in low scoring accuracy. Moreover, it only outputs an overall score, which is very inconvenient for the improvement of the human-computer dialogue model. Summary of the invention
[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art and proposes a multilingual human-computer dialogue scoring system.
[0005] In a first aspect, the present disclosure provides a multilingual human-computer dialogue scoring system, comprising:
[0006] A pre-training database, including multiple pre-training data, each of which contains a language sample question, a reference answer corresponding to the language sample question, a dialogue model answer, and an answer score;
[0007] Semantic segmentation module, used to semantically segment language sample questions into multiple semantic words;
[0008] The scoring module is used to compare the dialogue model answer with the reference answer using semantic words as the smallest comparison unit and score them according to the semantic similarity;
[0009] The output module is used to output the overall score and the score of each minimum comparison unit.
[0010] Preferably, the pre-training database is provided with a sample update interface, through which new pre-training data is acquired.
[0011] Preferably, the reference answers in the pre-training database are provided by language experts based on their understanding of language sample questions.
[0012] Preferably, the semantic segmentation module is specifically used for:
[0013] The language sample questions are divided into multiple semantic words according to each word as a group;
[0014] Determine whether two adjacent semantic characters can form a word. If so, form a semantic word by combining the two adjacent semantic characters.
[0015] Preferably, the scoring system further comprises:
[0016] A speech extraction module configured to extract a plurality of original speech sounds respectively spoken by a plurality of speakers;
[0017] The speech recognition module is connected to the speech extraction module and is configured to recognize each original speech to form a dialogue model answer and a reference answer through manual input.
[0018] Preferably, the scoring module includes at least two feature matching dimensions and feature matching units corresponding to the at least two feature matching dimensions respectively;
[0019] The feature matching dimensions include:
[0020] Text matching dimension, semantic matching dimension, keyword matching dimension, and string matching dimension;
[0021] The feature matching unit comprises:
[0022] The text matching unit corresponds to the text matching dimension, the semantic matching unit corresponds to the semantic matching dimension, the keyword matching unit corresponds to the keyword matching dimension, and the string matching unit corresponds to the string matching dimension.
[0023] Preferably, the feature matching unit specifically includes:
[0024] Predicting text similarity of the question-answer pair information by the text matching unit to obtain text similarity information;
[0025] Predicting the semantic similarity of the question-answer pair information by the semantic matching unit to obtain semantic similarity information;
[0026] Predicting keyword similarity of question-answer pair information by the keyword matching unit to obtain keyword similarity information;
[0027] Predicting the string similarity of the question-answer pair information by the string matching unit to obtain string similarity information;
[0028] Based on the text similarity information, the semantic similarity information, the keyword similarity information and the character string similarity information, an overall score corresponding to the question-answer pair information is generated.
[0029] Preferably, the scoring system further includes a dialogue model training module:
[0030] The reference answer and the dialogue model answer both include a plurality of answer phrases, and by comparing the similarity between the reference answer and the answer phrases in the dialogue model answer, a score value proportional to the similarity is determined;
[0031] When the score value corresponding to a certain answer phrase and the overall score value are lower than the preset value, a first feedback value is given, and the dialogue model parameters are modified and trained again; when the score values corresponding to each answer phrase are normal and the overall score value is lower than the preset value, a second feedback value is given, and the dialogue model parameters are modified and trained again; until the average of the score values corresponding to each answer phrase and the overall score value are higher than the preset value, the training is stopped. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A principle block diagram of a multilingual human-computer dialogue scoring system provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0034] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure are not any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" are not limited to quantity, but are at least one. "Including" or "including" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used for relative positional relationships. When the absolute position of the described target changes, the relative positional relationship may also change accordingly.
[0035] In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not all drawn to scale. In addition, some well-known parts may not be shown in the drawings.
[0036] Many specific details of the present disclosure are described below, such as the structure, materials, dimensions, processing technology and techniques of components, so as to more clearly understand the present disclosure. However, as those skilled in the art will appreciate, the present disclosure may be implemented without following these specific details.
[0037] like Figure 1 As shown, the embodiment of the present invention provides a multi-language human-computer dialogue scoring system, including the following functional modules:
[0038] A pre-training database, including multiple pre-training data, each of which contains a language sample question, a reference answer corresponding to the language sample question, a dialogue model answer, and an answer score;
[0039] Semantic segmentation module, used to semantically segment language sample questions into multiple semantic words;
[0040] The scoring module is used to compare the dialogue model answer with the reference answer using semantic words as the smallest comparison unit and score them according to the semantic similarity;
[0041] The output module is used to output the overall score and the score of each minimum comparison unit.
[0042] The pre-training data set D includes n pre-training data data, which can be expressed as:
[0043] D=(data0,data1,…,datai …,data n ), where each pre-training data contains a first language sample question question, a reference answer answer corresponding to the first language sample question, and a first dialogue model answer user answer , and the answer score score, which can be expressed as:
[0044] data i =(question i , answer i ,user answeri , score i )
[0045] In the formula, data i represents the i-th pre-training data in the pre-training dataset D, question i represents the first language sample question in the i-th pre-training data, answer i Then it is question i The corresponding reference answer is user answeri For question i The corresponding first user answer, score i Rate 's answers.
[0046] Correspondingly, the sample data in the training phase contains data that enables the model to be better suited for scoring tasks, and can answer questions, score and parse based on the input text. For example, it may include language questions, user answers corresponding to language questions, and scoring specifications for scoring user answers.
[0047] The pre-training database is equipped with a sample update interface, through which new pre-training data can be obtained. This can continuously enrich and improve the training samples and further test and optimize the scoring system.
[0048] In a specific real-time scenario, the semantic segmentation module is specifically used to: segment the language sample question into multiple semantic words according to each word as a group; determine whether two adjacent semantic words can form a word, and if so, combine the two adjacent semantic words into a semantic word. This is conducive to the dialogue model to answer, prevent the dialogue model from having confusing answers due to inaccurate sentence segmentation and word grouping, and facilitate the subsequent word-by-word scoring.
[0049] A further preferred solution takes into account that in traditional solutions, the automatic scoring system lacks an effective error feedback mechanism when performing automatic scoring. It only outputs an overall score and cannot give specific error feedback, nor can it provide a basis and support for scoring. This will not only greatly reduce the reliability and credibility of the scoring, but will also lead to a lack of basis for subsequent improvement and learning of the scoring mechanism. In other words, it is impossible to know the specific situation of its own human-computer dialogue model, the cause of the error, etc., resulting in the inability to make targeted improvements to the scoring mechanism, resulting in the inability to further improve the scoring accuracy and a poor experience.
[0050] Based on this, in order to enable the human-computer dialogue model to analyze the user's answer when scoring the user's answer in the scoring task, and give the user's score, error situation, and cause of error, etc., so as to provide support for the output score, the pre-trained model with strong understanding and answering capabilities obtained after the previous pre-training can be further fine-tuned, so that the dialogue model can not only review the questions and give reference answers in the scoring task of the fine-tuning training stage, but also review the reference answers and user answers for automatic scoring, and output predicted scores and scoring analysis to explain the basis for the model to give this predicted score, improve the credibility of the score, and facilitate targeted learning.
[0051] After obtaining the training data and the pre-trained model, the training data can be used as input and input into the pre-trained model so that the pre-trained model can understand it and output its predicted score and score analysis accordingly. Specifically, the pre-trained model first understands the input second sample question, answers the question, and outputs the corresponding reference answer. Then, based on the reference answer and using the given scoring specifications, the second user answer can be scored to obtain the predicted score and score analysis output by the model.
[0052] That is, based on the reference answer and the scoring specification, the pre-trained model can be applied to analyze the second user's answer to determine the correct answers and the wrong answers, and the scores can be determined according to the scoring specifications. The errors can be analyzed to determine the causes of the errors, the corresponding remedial solutions, etc., and finally the model can output the predicted score of the second user's answer and the score analysis. The score analysis here can include the score of the user's answer, such as which score points are scored, the score difference of each part, etc., the user's mistakes, such as which score points are wrong, the score mistakes of each part, etc., the reasons for the mistakes, the remedial solutions, etc.
[0053] Here, the scoring standard can be a pre-established / obtained / summarized scoring standard for user answers. For example, for comprehensive application questions (total score of 10 points), there are three questions in total, and each question corresponds to a reference answer. For the first question with a lower difficulty, if the answer given by the human-computer dialogue model is correct, 2 points will be given. For the second question with a medium difficulty, if the answer given by the human-computer dialogue model is correct, 3 points will be given. For the third question with a higher difficulty, if the answer given by the human-computer dialogue model is correct, 5 points will be given. If the final score is not 10 points, it means that the answer is wrong and needs to be corrected. When outputting the score, the answer to the specific wrong question and the reference answer will be given at the same time, and the reason for the discrepancy will be pointed out.
[0054] In a preferred solution, the scoring system further includes: a speech extraction module configured to extract multiple original speech sounds spoken by multiple speakers; a speech recognition module connected to the speech extraction module, configured to recognize each original speech sound to form a dialogue model answer, and a reference answer is input manually. When there are multiple input objects at the same time, the speech extraction module is used to extract the questions of different speakers and the language answers to be translated, and then they are recognized by the speech recognition module respectively, so as to obtain the corresponding dialogue model answers. At the same time, the original speech of the corresponding speaker will also be matched to the language sample question in the corresponding pre-training data separately, and then the reference answer will be matched.
[0055] When the input original speech cannot be matched to the corresponding language sample question in the pre-training data, it is necessary to manually input a new language sample question and reference answer and update them to the pre-training database synchronously.
[0056] In a preferred solution, the scoring system further includes a dialogue model training module: the reference answer and the dialogue model answer both include multiple answer phrases, and the similarity between the reference answer and each answer phrase in the dialogue model answer is compared to determine a scoring value proportional to the similarity;
[0057] When the score corresponding to a certain answer phrase and the overall score are lower than the preset value, a first feedback value is given, and the dialogue model parameters are modified and trained again. The model parameters corresponding to the first feedback value are modified for the algorithm trajectory of the specific answer phrase. Through this modification, the answer to the specific phrase can be modified without affecting the corresponding answer phrases of other answer phrases to a minimum extent.
[0058] When the corresponding score of each answer phrase is normal and the overall score is lower than the preset value, a second feedback value is given, and the dialogue model parameters are modified and trained again. In this case, the overall sentence is often not smooth and the translation is stiff. The second feedback value is to modify only the overall answer without affecting the corresponding answer phrases of all single answer phrases to the minimum extent, such as adding more words to polish the whole sentence, or modifying the position relationship of each answer phrase.
[0059] The training stops when the corresponding score of each answer phrase and the average of the overall score are higher than the preset value. The higher the score, the closer the dialogue model answer is to the reference answer, and the higher the accuracy of the trained model.
[0060] In a preferred solution, the scoring module includes at least two feature matching dimensions and feature matching units corresponding to the at least two feature matching dimensions respectively;
[0061] The feature matching dimensions include:
[0062] Text matching dimension, semantic matching dimension, keyword matching dimension, and string matching dimension;
[0063] The feature matching unit comprises:
[0064] The text matching unit corresponds to the text matching dimension, the semantic matching unit corresponds to the semantic matching dimension, the keyword matching unit corresponds to the keyword matching dimension, and the string matching unit corresponds to the string matching dimension.
[0065] Improve the accuracy of the scoring system through multi-dimensional comparative analysis.
[0066] In a more specific solution, the feature matching unit specifically includes:
[0067] The text similarity of the question-answer pair information is predicted by the text matching unit to obtain text similarity information; the semantic similarity of the question-answer pair information is predicted by the semantic matching unit to obtain semantic similarity information; the keyword similarity of the question-answer pair information is predicted by the keyword matching unit to obtain keyword similarity information; the string similarity of the question-answer pair information is predicted by the string matching unit to obtain string similarity information; based on the text similarity information, the semantic similarity information, the keyword similarity information and the string similarity information, an overall score corresponding to the question-answer pair information is generated.
[0068] The similarity information of different dimensions is combined to obtain an overall score by weighted scoring or comprehensive averaging. In addition, the score of each semantic word and the scores of different dimensions can be comprehensively weighted to obtain an overall score.
[0069] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.
Claims
1. A multilingual human-computer dialogue scoring system, characterized in that: include: A pre-training database, including multiple pre-training data, each of which contains a language sample question, a reference answer corresponding to the language sample question, a dialogue model answer, and an answer score; Semantic segmentation module, used to semantically segment language sample questions into multiple semantic words; The scoring module is used to compare the dialogue model answer with the reference answer using semantic words as the smallest comparison unit and score them according to the semantic similarity; The output module is used to output the overall score and the score of each minimum comparison unit.
2. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The pre-training database is provided with a sample update interface, through which new pre-training data is obtained.
3. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The reference answers in the pre-training database are provided by language experts based on their understanding of language sample questions.
4. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The semantic segmentation module is specifically used for: The language sample questions are divided into multiple semantic words according to each word as a group; Determine whether two adjacent semantic characters can form a word. If so, form a semantic word by combining the two adjacent semantic characters.
5. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The scoring system also includes: A speech extraction module configured to extract a plurality of original speech sounds respectively spoken by a plurality of speakers; The speech recognition module is connected to the speech extraction module and is configured to recognize each original speech to form a dialogue model answer and a reference answer through manual input.
6. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The scoring module includes at least two feature matching dimensions and feature matching units corresponding to the at least two feature matching dimensions respectively; The feature matching dimensions include: Text matching dimension, semantic matching dimension, keyword matching dimension, and string matching dimension; The feature matching unit comprises: The text matching unit corresponds to the text matching dimension, the semantic matching unit corresponds to the semantic matching dimension, the keyword matching unit corresponds to the keyword matching dimension, and the string matching unit corresponds to the string matching dimension.
7. The multilingual human-computer dialogue scoring system according to claim 6, characterized in that: The feature matching unit specifically includes: Predicting text similarity of the question-answer pair information by the text matching unit to obtain text similarity information; Predicting the semantic similarity of the question-answer pair information by the semantic matching unit to obtain semantic similarity information; Predicting keyword similarity of question-answer pair information by the keyword matching unit to obtain keyword similarity information; Predicting the string similarity of the question-answer pair information by the string matching unit to obtain string similarity information; Based on the text similarity information, the semantic similarity information, the keyword similarity information and the character string similarity information, an overall score corresponding to the question-answer pair information is generated.
8. The multilingual human-computer dialogue scoring system according to claim 1, characterized in that: The scoring system also includes a dialogue model training module: The reference answer and the dialogue model answer both include a plurality of answer phrases, and by comparing the similarity between the reference answer and the answer phrases in the dialogue model answer, a score value proportional to the similarity is determined; When the score value corresponding to a certain answer phrase and the overall score value are lower than the preset value, a first feedback value is given, and the dialogue model parameters are modified and trained again; when the score values corresponding to each answer phrase are normal and the overall score value is lower than the preset value, a second feedback value is given, and the dialogue model parameters are modified and trained again; until the average of the score values corresponding to each answer phrase and the overall score value are higher than the preset value, the training is stopped.