Methods, devices, electronic equipment and media for assessing question-and-answer consistency

Abstract

This application provides a method, apparatus, electronic device, and medium for evaluating question-and-answer consistency. The method includes: acquiring a first query statement from a first response model and acquiring a first response statement generated by a second response model in response to the first query statement; generating a second query statement based on the first response statement; inputting the second query statement into a second response model and acquiring a second response statement generated by the second response model in response to the second query statement; and evaluating the question-and-answer consistency of the second response model based on the first and second response statements, thereby accurately determining whether the question-and-answer consistency of the second response model meets the requirements.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and in particular to a method, apparatus, electronic device and medium for evaluating question-and-answer consistency. Background Technology

[0002] Artificial intelligence (AI) is a branch of computer science and is considered one of the three cutting-edge technologies of the 21st century. Over the past thirty years, AI has developed rapidly, finding widespread application and yielding fruitful results in many disciplines.

[0003] Natural language processing (NLP) technology in artificial intelligence can be used to achieve intelligent question answering. In existing technologies, for the same question, the question answering robot may give different answers due to different text statements, resulting in inconsistencies between the question and the answer, which prevents users from obtaining a definite answer. Summary of the Invention

[0004] This application aims to provide a method, apparatus, electronic device, and medium for evaluating question-and-answer consistency, which can evaluate the question-and-answer consistency of a response model, thereby accurately determining whether the question-and-answer consistency of a second response model meets the requirements.

[0005] According to one aspect of the embodiments of this application, a method for evaluating question-answer consistency is provided, comprising: obtaining a first query statement of a first response model and obtaining a first response statement generated by a second response model in response to the first query statement; generating a second query statement for the first response statement based on the first response statement; inputting the second query statement into the second response model and obtaining a second response statement generated by the second response model in response to the second query statement; and evaluating the question-answer consistency of the second response model based on the first response statement and the second response statement.

[0006] According to one aspect of the embodiments of this application, a question-and-answer consistency evaluation apparatus is provided, comprising: an acquisition module configured to acquire a first query statement of a first response model and acquire a first response statement generated by a second response model in response to the first query statement; a generation module configured to generate a second query statement for the first response statement based on the first response statement; a sending module configured to input the second query statement into the second response model and acquire a second response statement generated by the second response model in response to the second query statement; and a comparison module configured to evaluate the question-and-answer consistency of the second response model based on the first response statement and the second response statement.

[0007] In one embodiment of this application, based on the foregoing scheme, the generation module is configured to: identify the target word in the first reply statement; and generate a second query statement targeting the target word based on the first reply statement and the target word.

[0008] In one embodiment of this application, based on the foregoing scheme, the generation module is configured to: perform word segmentation on the first reply statement to obtain multiple reply words; and select entity words representing facts and / or opinions from the multiple reply words as the target words.

[0009] In one embodiment of this application, based on the foregoing scheme, the generation module is configured to: combine the first response statement and the target word to obtain a response combination; based on the semantics of the response combination, split the response combination into multiple semantic units; determine the unit representation vector corresponding to each semantic unit to obtain multiple unit representation vectors; and generate the second query statement based on the multiple unit representation vectors.

[0010] In one embodiment of this application, based on the aforementioned scheme, the generation module is configured to: input the plurality of unit representation vectors into the feature extraction unit combination to obtain the word representation vector output by the feature extraction unit combination; select the first query word in the second query statement from the candidate words in the vocabulary based on the word representation vector; re-input the plurality of unit representation vectors and the word representation vector into the feature extraction unit combination to obtain the next word representation vector corresponding to the next query word in the second query statement; select the next query word in the second query statement from the candidate words based on the next word representation vector, until the determination is completed.

[0011] In one embodiment of this application, based on the foregoing scheme, the feature extraction unit combination includes a bidirectional feature extraction unit, a unidirectional feature extraction unit, and a sequence-to-sequence feature extraction unit. The generation module is configured to: input the plurality of unit representation vectors into the bidirectional feature extraction unit to obtain bidirectional features output by the bidirectional feature extraction unit, wherein different unit representation vectors input to the bidirectional feature extraction unit have different parameters; input the bidirectional features and the word representation vectors into the unidirectional feature extraction unit to obtain unidirectional features output by the unidirectional feature extraction unit, wherein different word representation vectors input to the unidirectional feature extraction unit have different parameters; input the bidirectional features and the word representation vectors into the sequence-to-sequence feature extraction unit to obtain sequence features output by the sequence-to-sequence feature extraction unit, wherein different word representation vectors input to the sequence-to-sequence feature extraction unit have different parameters; and concatenate the bidirectional features, the unidirectional features, and the sequence features to obtain the next word representation vector corresponding to the next query word.

[0012] In one embodiment of this application, based on the foregoing scheme, the comparison module is configured to: combine the first response statement and the second response statement to obtain a statement combination; determine the statement parameters corresponding to the statement combination; encode the statement combination based on the statement parameters to obtain a statement code; determine a statement score based on the statement code; and evaluate the question-answer consistency of the second response model based on the statement score.

[0013] According to one aspect of the embodiments of this application, a computer-readable program medium is provided, which stores computer program instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0014] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any of the preceding claims.

[0015] According to one aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative embodiments described above.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0017] In some embodiments of this application, the technical solutions involve obtaining a first query statement from a first response model and a first response statement generated by a second response model in response to the first query statement; generating a second query statement based on the first response statement; inputting the second query statement into a second response model and obtaining a second response statement generated by the second response model in response to the second query statement. Since the second query statement is generated based on the first response statement generated by the evaluated second response model, the generated second query statement can accurately reflect the characteristics of the language processed by the second response model. Based on the first response statement and the second response statement, the question-and-answer consistency of the second response model is evaluated, and it can accurately determine whether the question-and-answer consistency of the second response model meets the requirements.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.

[0020] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0021] Figure 2 A flowchart illustrating a question-and-answer consistency evaluation method according to one embodiment of this application is shown schematically;

[0022] Figure 3 This illustration schematically shows a question-answer consistency evaluation framework applying one embodiment of the present application;

[0023] Figure 4 A block diagram of a question-and-answer consistency evaluation apparatus according to one embodiment of this application is illustrated schematically;

[0024] Figure 5 This is a hardware diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0026] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0029] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0030] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0031] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0032] The solutions provided in this application involve technologies such as natural language processing in artificial intelligence, and are specifically illustrated through the following embodiments:

[0033] Figure 1 A schematic diagram of an exemplary system architecture 100 to which the technical solutions of the embodiments of this application can be applied is shown.

[0034] like Figure 1 As shown, the system architecture 100 may include a first client 101, a second client 102, a server 103, and a network 104. The network 104 is used as a medium to provide a communication link between the first client 101, the second client 102, and the server 103. The network 104 may include various connection types, such as wired communication links, wireless communication links, etc., which are not limited herein.

[0035] It should be understood that Figure 1 The number of first client 101, second client 102, server 103, and network 104 in the diagram is merely illustrative. Depending on implementation needs, there can be any number of first client 101, second client 102, server 103, and network 104. For example, server 103 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. First client 101 and second client 102 can be electronic devices capable of question-and-answer or dialogue, such as smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, and chatbots, but are not limited to these.

[0036] In one embodiment of this application, server 103 obtains a first query statement of a first response model stored in first client 101 and a first reply statement generated by a second response model in response to the first query statement stored in second client 102; based on the first reply statement, a second query statement is generated for the first reply statement; the second query statement is input into the second response model, and a second reply statement generated by the second response model in response to the second query statement is obtained. Since the second query statement is generated based on the first reply statement generated by the evaluated second response model, the generated second query statement can accurately reflect the characteristics of the language processed by the second response model. Based on the first reply statement and the second reply statement, the question-and-answer consistency of the second response model is evaluated, and it can accurately determine whether the question-and-answer consistency of the second response model meets the requirements.

[0037] It should be noted that the question-and-answer consistency evaluation method provided in this application embodiment is generally executed by server 103, and correspondingly, the question-and-answer consistency evaluation device is generally set in server 103. However, in other embodiments of this application, the first client 101 and the second client 102 may also have similar functions to server 103, thereby executing the question-and-answer consistency evaluation method provided in this application embodiment.

[0038] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0039] Figure 2 The flowchart illustrating a question-answer consistency evaluation method according to an embodiment of this application is shown in the schematic diagram. The execution entity of the question-answer consistency evaluation method can be a server, for example, it can be... Figure 1 Server 103 is shown in the image.

[0040] Reference Figure 2 As shown, the method for evaluating question-and-answer consistency includes at least steps S210 to S240, which are detailed below:

[0041] In step S210, the first query statement of the first response model is obtained, and the first reply statement generated by the second response model in response to the first query statement is obtained.

[0042] In one embodiment of this application, the first response model can be in the first client 101 to implement the dialogue function of the first client 101, and the second response model can be in the second client 102 to implement the dialogue function of the second client 102. In other embodiments of this application, the first response model and the second response model can both be stored in the first client 101, or both can be stored in the second client 102, mutually assisting each other in implementing the dialogue function of the first client 101 or the second client 102.

[0043] In one embodiment of this application, the first query statement may be sent by a user or other client through a first response model, or it may be generated by the first response model.

[0044] In one embodiment of this application, after receiving the first query statement, the second response model can identify the source identifier carried in the first query statement, determine the session in which the first query statement belongs based on the source identifier, generate a first response statement based on the context information of the session containing the first query statement and the statements contained in the session.

[0045] In one embodiment of this application, the context after the first reply statement appears in the session can be predicted based on the context before the first query statement is received and the context after the first query statement is received. Based on the difference between the context before the first reply statement appears and the context after the first reply statement appears, the influence of the first reply statement on the context is determined as the semantics of the first reply statement, and thus the first reply statement is generated based on the semantics of the first reply statement.

[0046] In one embodiment of this application, the second response model can generate the first response statement through a neural network model. The neural network model can be pre-trained using a question-and-answer dataset. The training process of the neural network model can include: acquiring the question-and-answer dataset, where the standard response statement corresponding to each question statement sample in the question-and-answer dataset is known; inputting the question statement sample into the neural network model to obtain the output response statement output by the neural network model; if the output response statement is inconsistent with the standard response statement, adjusting the neural network model to make the output response statement consistent with the standard response statement.

[0047] Continue to refer to Figure 2 In step S220, a second query statement is generated based on the first response statement.

[0048] In one embodiment of this application, a target word in a first response statement can be identified; based on the first response statement and the target word, a second query statement is generated to query the target word, which can more accurately query the target word. Compared with generating a second statement similar to the first query statement based on the first query statement, this application uses the first response statement output by the second response model of the evaluation object as a basis, so that the obtained second query statement can contain the features of the second response model, thereby making the evaluation of the second response model more accurate.

[0049] In one embodiment of this application, the first reply statement can be segmented to obtain multiple reply words; entity words representing facts and / or opinions are selected from the multiple reply words as target words, wherein the semantics of the first reply statement can be obtained, the boundaries of the first reply statement can be identified based on the semantics, and then the first reply statement can be segmented based on the boundaries.

[0050] In one embodiment of this application, multiple response words can be tagged with their parts of speech, and entity words used to express facts and / or opinions can be selected according to their parts of speech. Entity words can include words used to express names of people, places, organizations, times, dates, etc. For example, if the first response statement is "I work at a daycare center," the target word can be "daycare center." If the first response statement is "I have three kids," the target words can be "three" and "kids." Words with the part of speech of adjectives can be determined as words expressing opinions. For example, if the first response statement is "I love kids," the target words can be "love" and "kids."

[0051] In other embodiments of this application, entity words used to represent facts and / or opinions can be selected based on the part of speech and the semantics of the response words.

[0052] In one embodiment of this application, a first response statement and a target word can be combined to obtain a response combination; based on the semantics of the response combination, the response combination is split into multiple semantic units; the unit representation vector corresponding to each semantic unit is determined to obtain multiple unit representation vectors; based on the multiple unit representation vectors, a second query statement is generated, wherein each word in the first response statement can be used as a semantic unit split from the response combination, and the target word can also be used as a semantic unit split from the response combination.

[0053] In one embodiment of this application, each semantic unit can be encoded to obtain the unit representation vector corresponding to each semantic unit.

[0054] In one embodiment of this application, each semantic unit can be input into an encoding model, which includes an encoding layer, to obtain the unit representation vector corresponding to each semantic unit output by the encoding model.

[0055] In one embodiment of this application, multiple unit representation vectors can be input into a feature extraction unit combination to obtain a word representation vector output by the feature extraction unit combination; based on the word representation vector, the first query word in the second query statement is selected from the candidate words in the vocabulary; the multiple unit representation vectors and the word representation vector are re-input into the feature extraction unit combination to obtain the next word representation vector corresponding to the next query word in the second query statement; based on the next word representation vector, the next query word in the second query statement is selected from the candidate words until the determination is completed, thereby obtaining the second query statement.

[0056] In one embodiment of this application, the determination of whether to complete the process can be based on the semantics and part-of-speech of the predicted query words. Specifically, the determination is complete if the predicted meaning of the query word matches the semantics of the second query statement, and if the predicted part-of-speech of the query word forms a complete sentence structure.

[0057] In one embodiment of this application, the context after the second query statement appears can be predicted based on the context before and after the first reply statement appears, and the impact of the second query statement on the context can be determined based on the context after the first reply statement appears and the context after the second query statement appears, which is used as the semantics of the second query statement.

[0058] In one embodiment of this application, the second query statement can be predicted based on the semantics of the second query statement.

[0059] In one embodiment of this application, the feature extraction unit combination may include a bidirectional feature extraction unit, a unidirectional feature extraction unit, and a sequence-to-sequence feature extraction unit. Multiple unit representation vectors can be input into the bidirectional feature extraction unit to obtain bidirectional features output by the bidirectional feature extraction unit. Different unit representation vectors input to the bidirectional feature extraction unit have different parameters, thereby extracting features of the word representation vector in different aspects. The bidirectional features and word representation vectors are input into the unidirectional feature extraction unit to obtain unidirectional features output by the unidirectional feature extraction unit. Different word representation vectors input to the unidirectional feature extraction unit have different parameters, thereby extracting features of the word representation vector in different aspects. The bidirectional features and word representation vectors are input into the sequence-to-sequence feature extraction unit to obtain sequence features output by the sequence-to-sequence feature extraction unit. Different word representation vectors input to the sequence-to-sequence feature extraction unit have different parameters, thereby extracting features of the word representation vector in different aspects. The bidirectional features, unidirectional features, and sequence features are concatenated to obtain the next word representation vector corresponding to the next query word.

[0060] In this embodiment, when obtaining the next word representation vector, the bidirectional features obtained from multiple unit representation vectors are taken into account, that is, the overall features of the first reply statement and the features of each unit representation vector in the first reply statement are taken into account; the features of the word representation vectors before the next word are also taken into account, so that the obtained next word representation vector is more accurate.

[0061] Continue to refer to Figure 2 In step S230, the second query statement is input into the second response model, and the second response statement generated by the second response model in response to the second query statement is obtained.

[0062] In one embodiment of this application, the process of the second response model generating the second response statement can refer to the process of the second response model generating the first response statement in step S210. The process of the second response model generating the second response statement can also refer to the process of the server 103 generating the second query statement based on the first response statement in step S220, which will not be described in detail here.

[0063] In step S240, the question-and-answer consistency of the second response model is evaluated based on the first response statement and the second response statement.

[0064] In one embodiment of this application, the first response statement and the second response statement can be combined to obtain a statement combination; the statement parameters corresponding to the statement combination are determined; the statement combination is encoded based on the statement parameters to obtain a statement code; the statement score is determined based on the statement code; and the question-answer consistency of the second response model is evaluated based on the statement score.

[0065] exist Figure 2 In this embodiment, a first query statement from a first response model is obtained, and a first response statement generated by a second response model in response to the first query statement is obtained; a second query statement is generated based on the first response statement; the second query statement is input into the second response model, and a second response statement generated by the second response model in response to the second query statement is obtained. Since the second query statement is generated based on the first response statement generated by the evaluated second response model, the generated second query statement can accurately reflect the characteristics of the language processed by the second response model. Based on the first response statement and the second response statement, the question-and-answer consistency of the second response model is evaluated, and it can accurately determine whether the question-and-answer consistency of the second response model meets the requirements.

[0066] In this embodiment, not only can the consistency quality of the second response model be evaluated, but a consistency index can also be provided to guide designers to improve the second response model so that it generates dialogue text with higher consistency quality, thus promoting the better and faster development of the chatbot field.

[0067] In one embodiment of this application, human evaluators can be hired to score the consistency of multi-turn dialogue content generated by the chatbot. Human evaluation results are more realistic and reliable than those from automated evaluators. By accumulating a large number of human evaluation samples to build a training dataset, the model's consistency capability can be further improved after training. Furthermore, this human evaluation mechanism is more time-saving and efficient than hiring humans to read all the content, as it only requires reading two responses to the same question.

[0068] exist Figure 2 In this embodiment, in step S210, the first response model can use a chatbot model to generate a first reply statement in response to the first query statement from the second response model. In step S220, a second query statement can be generated in response to the first reply statement using the chatbot model. In step S230, the second response model can use the chatbot model to generate a second reply statement in response to the second query statement. The chatbot model can be a generative multi-turn dialogue model (DialoGPT), a Blender model, an open-domain chatbot model (Plato), or a deep learning-based chatbot model (DialoFlow), etc.

[0069] Figure 3 This diagram illustrates a question-and-answer consistency evaluation framework applying one embodiment of this application. Figure 3 As shown, the evaluation framework may include an inquiry module, a multi-turn dialogue module, and a consistency detection module. The inquiry module includes an Inquirer, which is used to identify entity words in the first response statement and generate a second inquiry statement. In the multi-turn dialogue module, the participants in the dialogue are Chatbot1 and Chatbot2. The consistency detection module may include an automatic evaluator or a human evaluator.

[0070] Figure 3 The consistency quality assessment of the evaluation framework can be carried out in the following steps:

[0071] Step 1: In the multi-turn dialogue module, chatbot 1 and chatbot 2 complete a round of dialogue. Specifically, chatbot 1 sends the first question: "Hello, what do you do for a living?" Chatbot 2 replies with the first response: "I work at a daycare center, do you have kids?"

[0072] Step 2: The query module captures the entity "daycarecenter" appearing in the chatbot 2's reply text and generates a second query statement corresponding to the entity "Where do you work?".

[0073] Step 3: Insert the generated second query "Where do you work?" into the multi-turn dialogue module as the output of chatbot 1;

[0074] Step 4: In the multi-turn dialogue module, chatbot 2 responds to the second question "Where do you work?" with the second reply "A daycare center for young children."

[0075] Step 5: In the consistency detection module, take the first response text "A daycare center for young children." and the second response text "I work at a daycare center, do you have kids?" as input. The automatic evaluator outputs an evaluation result on whether the two responses of chatbot 2 are consistent, or a human evaluator evaluates whether the two responses of chatbot 2 are consistent with the question-and-answer consistency, and obtains the consistency result displayed in the consistency detection module.

[0076] exist Figure 3In this embodiment, chatbot 2 underwent a second evaluation. The one-round dialogue used was as follows: Chatbot 1 issued the first query: "Yes, I have two children. How about you?" Chatbot 2 replied with the first response: "I have three kids, one son and two daughters." The query robot recognized the entity "three" in the first response and generated a second query with the corresponding entity "How many kids do you have?" as the output of chatbot 1. Chatbot 2 responded to the second query "How many kids do you have?" with the second response "I have no kids. I wish to have one." The first response text "I have three kids, one son and two daughters." and the second response text "I have no kids. I wish to have one" were input into the consistency detection module to evaluate whether the two responses of chatbot 2 met the evaluation result of consistency between the question and answer. The consistency detection module showed a non-compliance result.

[0077] exist Figure 3In this embodiment, chatbots 1 and 2 can be implemented using open-source chatbot models, such as the Generative Multi-Turn Dialogue Model (DialoGPT), the Blender model, the Open Domain Chatbot Model (Plato), and the Deep Learning-Based Chatbot Model (DialoFlow). Chatbot 1 can randomly output a sentence of natural language text. Chatbot 2 takes this natural language text as input as a first query statement and outputs a reply text as a first reply statement. When chatbot 1 receives the output from the query module as input, it takes the output of the query module as a second query statement and uses the second query statement as input to chatbot 2, causing chatbot 2 to generate a second reply statement text corresponding to the output of the query module. In the query module, the entity queryer can be implemented using the named entity recognition tool Stanza, and the question generator can be implemented using the question generation model UniLM. The queryer takes the reply text from chatbot 2 as input, extracts entity words related to opinions and factual knowledge, and the question generator takes the entity words as input, generates questions challenging the entity words, and inserts them into the multi-turn dialogue module. In the consistency detection module, the automatic evaluator can be implemented using the Roberta-large contradiction detection model. It takes two responses generated by chatbot 2 to the same question as input and outputs a consistency score. Human evaluators use the same input as the automatic evaluator and can provide more accurate evaluation scores.

[0078] Figure 3 The embodiments provide a public evaluation metric framework for calculating the consistency quality assessment score of any chatbot. Table 1 shows the consistency evaluation metric results of the four best-performing open-domain chatbots using this framework. Table 1 is denoted as Contradiction Rate (Auto). The rows and columns of Table 1 are the specific names of chatbot 1 and chatbot 2, respectively. BL, PL, DG, and DF represent Blender, Plato, DialoGPT, and DialoFlow, respectively. The table element values ​​are the evaluation metric results. Table 1 shows the evaluation metric results of the proposed AIH, with the consistency ranking of the four chatbots as: PL > DG > DF > BL. Table 2 shows the results of the human evaluation metric, denoted as Contradiction Rate (Human), with the chatbot ranking as: PL > DG > DF > BL, consistent with the ranking of the evaluation metric, demonstrating the effectiveness of the evaluation metric.

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] Figure 3 The proposed framework can encourage open-domain chatbot models to generate more consistent responses, reduce the occurrence of contradictory responses, provide convenience and guidance for future research on open-domain chatbots, and make chatbots more "human-like".

[0084] The following describes an embodiment of the apparatus described in this application, which can be used to perform the question-and-answer consistency evaluation method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the question-and-answer consistency evaluation method described above in this application.

[0085] Figure 4 A block diagram of a question-and-answer consistency evaluation apparatus according to one embodiment of this application is illustrated schematically.

[0086] Reference Figure 4 As shown, a question-and-answer consistency evaluation apparatus 400 according to an embodiment of this application includes an acquisition module 401, a generation module 402, a sending module 403, and a comparison module 404.

[0087] According to one aspect of the embodiments of this application, based on the foregoing scheme, the acquisition module 401 is configured to acquire a first query statement of the first response model and acquire a first reply statement generated by the second response model in response to the first query statement; the generation module 402 is configured to generate a second query statement for the first reply statement based on the first reply statement; the sending module 403 is configured to input the second query statement into the second response model and acquire a second reply statement generated by the second response model in response to the second query statement; and the comparison module 404 is configured to evaluate the question-answer consistency of the second response model based on the first reply statement and the second reply statement.

[0088] In one embodiment of this application, based on the aforementioned scheme, the generation module 402 is configured to: identify the target word in the first reply statement; and generate a second query statement that queries the target word based on the first reply statement and the target word.

[0089] In one embodiment of this application, based on the aforementioned scheme, the generation module 402 is configured to: perform word segmentation on the first reply statement to obtain multiple reply words; and select entity words representing facts and / or opinions from the multiple reply words as target words.

[0090] In one embodiment of this application, based on the aforementioned scheme, the generation module 402 is configured to: combine the first response statement and the target word to obtain a response combination; based on the semantics of the response combination, split the response combination into multiple semantic units; determine the unit representation vector corresponding to each semantic unit to obtain multiple unit representation vectors; and generate a second query statement based on the multiple unit representation vectors.

[0091] In one embodiment of this application, based on the aforementioned scheme, the generation module 402 is configured to: input multiple unit representation vectors into the feature extraction unit combination to obtain the word representation vector output by the feature extraction unit combination; select the first query word in the second query statement from the candidate words in the vocabulary based on the word representation vector; re-input the multiple unit representation vectors and the word representation vector into the feature extraction unit combination to obtain the next word representation vector corresponding to the next query word in the second query statement; select the next query word in the second query statement from the candidate words based on the next word representation vector, until the determination is completed.

[0092] In one embodiment of this application, based on the aforementioned scheme, the feature extraction unit combination includes a bidirectional feature extraction unit, a unidirectional feature extraction unit, and a sequence-to-sequence feature extraction unit. The generation module 402 is configured to: input multiple unit representation vectors into the bidirectional feature extraction unit to obtain bidirectional features output by the bidirectional feature extraction unit, wherein different unit representation vectors input to the bidirectional feature extraction unit have different parameters; input the bidirectional features and word representation vectors into the unidirectional feature extraction unit to obtain unidirectional features output by the unidirectional feature extraction unit, wherein different word representation vectors input to the unidirectional feature extraction unit have different parameters; input the bidirectional features and word representation vectors into the sequence-to-sequence feature extraction unit to obtain sequence features output by the sequence-to-sequence feature extraction unit, wherein different word representation vectors input to the sequence-to-sequence feature extraction unit have different parameters; and concatenate the bidirectional features, unidirectional features, and sequence features to obtain the next word representation vector corresponding to the next query word.

[0093] In one embodiment of this application, based on the foregoing scheme, the comparison module 404 is configured to: combine the first response statement and the second response statement to obtain a statement combination; determine the statement parameters corresponding to the statement combination; encode the statement combination based on the statement parameters to obtain a statement code; determine the statement score based on the statement code; and evaluate the question-answer consistency of the second response model based on the statement score.

[0094] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0095] The following reference Figure 5 To describe an electronic device 50 according to this embodiment of the present application. Figure 5 The electronic device 50 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0096] like Figure 5 As shown, the electronic device 50 is presented in the form of a general-purpose computing device. The components of the electronic device 50 may include, but are not limited to: at least one processing unit 51, at least one storage unit 52, a bus 53 connecting different system components (including storage unit 52 and processing unit 51), and a display unit 54.

[0097] The storage unit stores program code that can be executed by the processing unit 51, causing the processing unit 51 to perform the steps described in the "Embodiment Method" section above according to various exemplary embodiments of this application.

[0098] Storage unit 52 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 521 and / or a cache memory unit 522, and may further include a read-only memory unit (ROM) 523.

[0099] Storage unit 52 may also include a program / utility 524 having a set (at least one) program module 525, such program module 525 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0100] Bus 53 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0101] Electronic device 50 can also communicate with one or more external devices (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 50, and / or any device that enables electronic device 50 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 55. Furthermore, electronic device 50 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 56. As shown, network adapter 56 communicates with other modules of electronic device 50 via bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 50, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this application.

[0103] According to one embodiment of this application, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above in this specification is stored. In some possible implementations, various aspects of this application may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section of this specification.

[0104] According to one embodiment of this application, the program product for implementing the above method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0108] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0109] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0110] It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for evaluating question-answer consistency, characterized in that, include: Obtain the first query statement of the first response model, and obtain the first reply statement generated by the second response model in response to the first query statement; Identify the target words in the first response statement; The first response statement and the target word are combined to obtain a response combination. Based on the semantics of the response combination, the response combination is split into multiple semantic units, and the unit representation vector corresponding to each semantic unit is determined to obtain multiple unit representation vectors. The word representation vector is obtained based on the multiple unit representation vectors; Based on the word representation vector, select the first query word from the candidate words in the vocabulary, which is the first word in the second query statement to be determined for the first response statement; Based on the multiple unit representation vectors and the word representation vectors, the next word representation vector corresponding to the next query word in the second query statement to be determined is obtained. Based on the next word representation vector, the next query word in the second query statement to be determined is selected from the candidate words until the determination is completed, and the second query statement is obtained. Input the second query statement into the second response model, and obtain the second response statement generated by the second response model in response to the second query statement; The consistency of the question-and-answer model is evaluated based on the first and second response statements.

2. The method for evaluating question-and-answer consistency according to claim 1, characterized in that, The identification of target words in the first response statement includes: The first reply statement is segmented into multiple reply words; Entity words that express facts and / or opinions are selected from the plurality of response words as the target words.

3. The method for evaluating question-and-answer consistency according to claim 1, characterized in that, The process of obtaining word representation vectors based on the multiple unit representation vectors includes: The multiple unit representation vectors are input into the feature extraction unit for combination to obtain the word representation vector output by the feature extraction unit combination. The step of obtaining the next word representation vector corresponding to the next query word in the second query statement to be determined, based on the plurality of unit representation vectors and the word representation vector, includes: The multiple unit representation vectors and the word representation vectors are re-inputted into the feature extraction unit to obtain the next word representation vector corresponding to the next query word in the second query statement to be determined.

4. The method for evaluating question-and-answer consistency according to claim 3, characterized in that, The feature extraction unit combination includes a bidirectional feature extraction unit, a unidirectional feature extraction unit, and a sequence-to-sequence feature extraction unit. The step of re-inputting the multiple unit representation vectors and the word representation vector into the feature extraction unit combination to obtain the next word representation vector corresponding to the next query word in the second query statement to be determined includes: The multiple unit representation vectors are input into the bidirectional feature extraction unit to obtain the bidirectional features output by the bidirectional feature extraction unit, wherein different unit representation vectors are input into the bidirectional feature extraction unit with different parameters; The bidirectional features and the word representation vectors are input into the unidirectional feature extraction unit to obtain the unidirectional features output by the unidirectional feature extraction unit. The parameters of the unidirectional feature extraction unit are different for different word representation vectors. The bidirectional features and the word representation vectors are input into the sequence-to-sequence feature extraction unit to obtain the sequence features output by the sequence-to-sequence feature extraction unit. Different word representation vectors are input into the sequence-to-sequence feature extraction unit with different parameters. The bidirectional features, the unidirectional features, and the sequence features are concatenated to obtain the next word representation vector corresponding to the next query word.

5. The method for evaluating question-and-answer consistency according to claim 1, characterized in that, The process of determining the unit representation vector corresponding to each semantic unit to obtain multiple unit representation vectors includes: Each semantic unit is input into the encoding model, and the encoding layer contained in the encoding model encodes each semantic unit to determine the unit representation vector corresponding to each semantic unit, thereby obtaining the multiple unit representation vectors.

6. The method for evaluating question-and-answer consistency according to claim 1, characterized in that, The method further includes: Based on the predicted semantics and part of speech of the query words, the determination is completed.

7. The method for evaluating question-and-answer consistency according to claim 1, characterized in that, The step of evaluating the question-answer consistency of the second response model based on the first response statement and the second response statement includes: The first response statement and the second response statement are combined to obtain a statement combination; Determine the statement parameters corresponding to the statement combination; The statement combination is encoded based on the statement parameters to obtain the statement encoding; Based on the statement encoding, a statement score is determined, and the question-answer consistency of the second response model is evaluated based on the statement score.

8. A device for evaluating question-and-answer consistency, characterized in that, include: The acquisition module is configured to acquire the first query statement of the first response model and the first response statement generated by the second response model in response to the first query statement; The generation module is configured to identify target words in the first response statement; The first response statement and the target word are combined to obtain a response combination. Based on the semantics of the response combination, the response combination is split into multiple semantic units, and the unit representation vector corresponding to each semantic unit is determined to obtain multiple unit representation vectors. Based on the multiple unit representation vectors, a word representation vector is obtained. Based on the word representation vector, the starting query word at the beginning of the second query statement to be determined for the first response statement is selected from the candidate words in the vocabulary. Based on the multiple unit representation vectors and the word representation vectors, the next word representation vector corresponding to the next query word in the second query statement to be determined is obtained. Based on the next word representation vector, the next query word in the second query statement to be determined is selected from the candidate words until the determination is completed, and the second query statement is obtained. The sending module is configured to input the second query statement into the second response model and obtain the second response statement generated by the second response model in response to the second query statement; The comparison module is configured to evaluate the question-answer consistency of the second response model based on the first response statement and the second response statement.

9. An electronic device, characterized in that, include: Memory, which stores computer-readable instructions; A processor reads computer-readable instructions stored in memory to perform the method described in any one of claims 1-7.

10. A computer program medium having stored computer-readable instructions thereon, which, when executed by a processor of a computer, cause the computer to perform the method of any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the method of any one of claims 1-7.

Images