Machine Reading Comprehension Data Processing Method and Apparatus, Electronic Device, and Storage Medium

Abstract

The present disclosure provides a method and apparatus for processing machine reading comprehension data, an electronic device, and a storage medium. Among them, the method for processing machine reading comprehension data includes: inputting a plurality of candidate answers into a self-evaluation model; in the self-evaluation model, calculating the evaluation results of each candidate answer; determining a final answer from the plurality of candidate answers according to the evaluation results of each candidate answer; outputting the final answer through the self-evaluation model; wherein, calculating the evaluation results of each candidate answer includes performing the following operations on each candidate answer respectively: calculating the number of intersections between the target candidate answer and other candidate answers among the plurality of candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each other candidate answer; calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answers.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more particularly, to a method and apparatus for processing machine reading comprehension data, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence technology, the application of machine reading comprehension has become increasingly widespread. Machine reading comprehension refers to the task of enabling a computer to understand the semantics of an article and correctly answer relevant questions by using natural language processing technology. Among them, the answers to the questions raised generally can be found in the article.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art. When a pre-trained prediction model based on a machine reading comprehension algorithm answers questions, generally, the pre-trained prediction model cannot well predict the correct starting position and correct ending position of the correct answer in the article, and the inaccurate positions predicted will cause the predicted answer to lack some information or have information redundancy compared with the correct answer, resulting in an unsatisfactory prediction result. Summary of the Invention

[0004] In view of this, the present disclosure provides a method and apparatus for processing machine reading comprehension data, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] One aspect of the present disclosure provides a method for processing machine reading comprehension data, including:

[0006] Inputting a plurality of candidate answers into a self-evaluation model;

[0007] Calculating the evaluation result of each candidate answer in the self-evaluation model;

[0008] Determining a final answer from the plurality of candidate answers according to the evaluation result of each candidate answer;

[0009] Outputting the final answer through the self-evaluation model;

[0010] Wherein, calculating the evaluation result of each candidate answer includes performing the following operations on each candidate answer respectively:

[0011] Calculating the intersection number between the target candidate answer and other candidate answers among the plurality of candidate answers, where the intersection number is used to represent the number of overlapping words between the target candidate answer and each other candidate answer;

[0012] Calculating the evaluation result of the target candidate answer according to the intersection number, the number of words in the target candidate answer, and the number of words in other candidate answers.

[0013] According to an embodiment of the present disclosure, wherein, according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers, calculating the evaluation result of the target candidate answer includes:

[0014] Calculating a first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers, wherein the first overlap rate is used to characterize: the degree of similarity between the target candidate answer and other candidate answers with other candidate answers as a reference;

[0015] Calculating a second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer, wherein the second overlap rate is used to characterize: the degree of similarity between other candidate answers and the target candidate answer with the target candidate answer as a reference;

[0016] Calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate.

[0017] According to an embodiment of the present disclosure, wherein, calculating the first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers includes:

[0018] Calculating the ratio of the number of intersections to the number of words in other candidate answers to obtain the first overlap rate.

[0019] According to an embodiment of the present disclosure, wherein, calculating the second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer includes:

[0020] Calculating the ratio of the number of intersections to the number of words in the target candidate answer to obtain the second overlap rate.

[0021] According to an embodiment of the present disclosure, wherein, calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate includes:

[0022] Calculating an accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate;

[0023] Calculating the evaluation result of the target candidate answer according to the accuracy score.

[0024] According to an embodiment of the present disclosure, wherein, calculating the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate includes:

[0025] Calculating an accuracy score of the target candidate answer relative to other candidate answers according to a preset coefficient, the product of the first overlap rate and the second overlap rate, and the sum of the first overlap rate and the second overlap rate.

[0026] According to an embodiment of the present disclosure, one accuracy score corresponds to one other candidate answer, and calculating the evaluation result of the target candidate answer according to the accuracy score includes:

[0027] Calculating the average value of multiple accuracy scores to obtain the evaluation result of the target candidate answer.

[0028] According to an embodiment of the present disclosure, it further includes:

[0029] Inputting the text to be measured and a preset question proposed for the text to be measured into a pre-trained language model;

[0030] Outputting by the pre-trained language model: the start probability and the end probability of each word in the text to be measured, where the start probability of each word is the probability that each word is at the start position of the correct answer to the preset question, and the end probability of each word is the probability that each word is at the end position of the correct answer to the preset question; and

[0031] Determining multiple candidate answers for the preset question according to the start probability and the end probability of each word.

[0032] A machine reading comprehension data processing device includes a first input module, a calculation module, a first determination module, and a first output module.

[0033] The first input module is used to input multiple candidate answers into the self-evaluation model;

[0034] The calculation module is used to calculate the evaluation result of each candidate answer in the self-evaluation model;

[0035] The first determination module is used to determine the final answer from multiple candidate answers according to the evaluation result of each candidate answer;

[0036] The first output module is used to output the final answer through the self-evaluation model;

[0037] Among them, calculating the evaluation result of each candidate answer includes performing the following operations on each candidate answer respectively:

[0038] Calculating the number of intersections between the target candidate answer and other candidate answers among multiple candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each other candidate answer;

[0039] Calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers.

[0040] According to an embodiment of the present disclosure, the calculation module includes a first calculation unit, a second calculation unit, and a third calculation unit.

[0041] Among them, the first calculation unit is used to calculate the first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers, where the first overlap rate is used to characterize: the similarity degree between the target candidate answer and other candidate answers based on other candidate answers;

[0042] The second calculation unit calculates the second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer, where the second overlap rate is used to characterize: the similarity degree between other candidate answers and the target candidate answer based on the target candidate answer;

[0043] The third calculation unit calculates the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate.

[0044] According to an embodiment of the present disclosure, among them, in the first calculation unit, calculating the first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers includes: calculating the ratio of the number of intersections to the number of words in other candidate answers to obtain the first overlap rate.

[0045] According to an embodiment of the present disclosure, among them, in the second calculation unit, calculating the second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer includes: calculating the ratio of the number of intersections to the number of words in the target candidate answer to obtain the second overlap rate.

[0046] According to an embodiment of the present disclosure, among them, the third calculation unit includes a first calculation subunit and a second calculation subunit.

[0047] Among them, the first calculation subunit calculates the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate. The second calculation subunit calculates the evaluation result of the target candidate answer according to the accuracy score.

[0048] According to an embodiment of the present disclosure, among them, in the first calculation subunit, calculating the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate includes:

[0049] Calculating the accuracy score of the target candidate answer relative to other candidate answers according to a preset coefficient, the product of the first overlap rate and the second overlap rate, and the sum of the first overlap rate and the second overlap rate.

[0050] According to an embodiment of the present disclosure, where one accuracy score corresponds to one other candidate answer, and among them, in the second calculation subunit, calculating the evaluation result of the candidate answer according to the accuracy score includes: calculating the average value of multiple accuracy scores to obtain the evaluation result of the target candidate answer.

[0051] According to an embodiment of the present disclosure, it further includes a second input module, a second output module, and a second determination module.

[0052] The second input module is configured to input the text to be tested and a preset question proposed for the text to be tested into a pre-trained language model.

[0053] The second output module is configured to output, through the pre-trained language model: the start probability and the end probability of each word in the text to be tested, where the start probability of each word is: the probability that each word is the start position of the correct answer to the preset question, and the end probability of each word is: the probability that each word is the end position of the correct answer to the preset question.

[0054] The second determination module is configured to determine multiple candidate answers to the preset question according to the start probability and the end probability of each word.

[0055] Another aspect of the present disclosure provides an electronic device, including: one or more processors, and a memory; wherein the memory is used to store one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the machine reading comprehension data processing method as described above.

[0056] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, and the instructions are used to implement the machine reading comprehension data processing method as described above when executed.

[0057] Another aspect of the present disclosure provides a computer program product, the computer program product includes computer-executable instructions, and the instructions are used to implement the machine reading comprehension data processing method as described above when executed.

[0058] According to an embodiment of the present disclosure, by calculating the number of intersections between a target candidate answer and other candidate answers among multiple candidate answers, and then calculating the evaluation result of the target candidate answer based on the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers. It can be seen that this method performs calculation and analysis based on the number of intersections in combination with the number of words. Since the number of intersections takes into account the common parts between two answers, therefore, the evaluation result of the candidate answer calculated based on the number of intersections in combination with the number of words takes into account the similarity factor between the candidate answer and each other candidate answer. Because the part with higher similarity has a higher possibility of being the correct answer, it can improve the accuracy of the prediction result to a certain extent. Further, since the evaluation result of each candidate answer is obtained by fully considering the similarity between every two answers, that is, fully considering the possibility of each candidate answer being the correct answer. Compared with the related art, in the related art, when considering the accuracy of each candidate answer, only the accuracy of the current candidate answer itself is considered, while the method described in the embodiment of the present disclosure fully comprehensively considers the possibility of each candidate answer being the correct answer, so the accuracy of the prediction can be improved. It can be seen that the machine reading comprehension data processing method described in the embodiment of the present disclosure can evaluate the prediction result of the pre-trained model through a self-evaluation method without additional input, and can achieve the technical effect and purpose of calibrating the model output result. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0060] Figure 1 Schematically shows an exemplary system architecture to which the machine reading comprehension data processing method and device of the present disclosure can be applied;

[0061] Figure 2 Schematically shows a flowchart of the machine reading comprehension data processing method according to an embodiment of the present disclosure;

[0062] Figure 3 Schematically shows a flowchart of calculating the evaluation result of a candidate answer according to the number of intersections, the number of words in the candidate answer, and the number of words in other candidate answers according to an embodiment of the present disclosure;

[0063] Figure 4 Schematically shows a flowchart of the machine reading comprehension data processing method according to another embodiment of the present disclosure;

[0064] Figure 5 Schematically shows a block diagram of the answer evaluation device according to an embodiment of the present disclosure; and

[0065] Figure 6A block diagram of an electronic device for implementing a machine reading comprehension data processing method according to an embodiment of the present disclosure is schematically shown. Detailed implementation manners

[0066] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, for the sake of explanation, numerous specific details are set forth in order to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is obvious that one or more embodiments may be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts of the present disclosure.

[0067] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0068] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0069] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include, but is not limited to, a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0070] Before elaborating on the embodiments of the present disclosure in detail, the system structure and application scenarios involved in the method provided by the embodiments of the present disclosure are introduced as follows.

[0071] Figure 1 An exemplary system architecture 100 to which the machine reading comprehension data processing method and apparatus of the present disclosure can be applied is schematically shown. It should be noted that Figure 1The figure shown is only an example of the system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0072] As Figure 1 shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0073] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications, such as web browser applications, search applications, various algorithmic applications, etc. (only as examples), may be installed on the terminal devices 101, 102, 103.

[0074] The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc.

[0075] The server 105 may be a server that provides various services, such as a background management server that supports websites browsed by users using the terminal devices 101, 102, 103, a server that executes computational tasks corresponding to user requests, etc. The background management server may analyze and process data such as received user requests, and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0076] According to the embodiments of the present disclosure, the above system architecture can be used in the scenario of machine reading comprehension of the embodiments of the present disclosure. Users can input the text to be tested and a preset question for the text to be tested through the terminal devices 101, 102, 103, and send a request to the server 105 to perform reading comprehension on the text to be tested to obtain the correct answer for the preset question. After receiving the request sent by the terminal devices 101, 102, 103, the server 105 executes the answer evaluation algorithm of the embodiments of the present disclosure in the server 105, and after outputting the correct answer, feeds back the correct answer to the terminal devices 101, 102, 103.

[0077] Note that the machine reading comprehension data processing method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the machine reading comprehension data processing device provided by the embodiments of the present disclosure can generally be set in the server 105. The machine reading comprehension data processing method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the machine reading comprehension data processing device provided by the embodiments of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the machine reading comprehension data processing method provided by the embodiments of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or can also be executed by other terminal devices different from the terminal devices 101, 102, or 103. Correspondingly, the machine reading comprehension data processing device provided by the embodiments of the present disclosure can also be set in the terminal devices 101, 102, or 103, or can be set in other terminal devices different from the terminal devices 101, 102, or 103.

[0078] It should be understood that Figure 1 the numbers of the terminal devices, networks and servers in

[0079] Machine reading comprehension refers to the task of enabling a computer to understand the semantics of an article and correctly answer relevant questions by using natural language processing technology. Among them, the answers involved in the questions raised can generally be found in the article. When the pre-trained prediction model based on the machine reading comprehension algorithm answers questions, through model prediction, it outputs the probabilities of the start position and the end position of the answer in the article, and after scoring according to the probability size, it obtains multiple candidate answers, and selects the answer with the highest score as the final answer. However, the answer with the highest score is not necessarily the correct answer. Because the start position and the end position of the answer corresponding to the answer with the highest score in the article are not necessarily the correct start position and end position, and the inaccurate positions predicted will cause the predicted answer to lack some information or have information redundancy compared with the correct answer, resulting in an unsatisfactory prediction result.

[0080] As shown in the following example:

[0081] The text to be tested is:

[0082] Zhang San became an apprentice following his father and recorded the conditions of some patients. His elder brother, Li Si, had entered University A three years ago. Zhang San felt unable to adapt. Zhang San's friend arranged for Zhang San to go to City B to study at university. At that time, the medical school in City B had better facilities and faculty than University A. After Li Si returned from University A, he was going to study at an off-campus hospital in City B. The two brothers then arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room.

[0083] The preset question is: What is Zhang San's house like in City B?

[0084] The correct answer is: A house with two bedrooms and a living room.

[0085] The multiple candidate answers output by the pre-trained prediction model are:

[0086] TOP1: A house with a living room (lack of information);

[0087] TOP2: The two brothers arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room (redundant information);

[0088] TOP3: A house with two bedrooms and a living room (correct answer);

[0089] TOP4: Arrived in City B together and rented a house with two bedrooms and a living room (redundant information);

[0090] TOP5: Rented a house with two bedrooms (lack of information).

[0091] It can be seen that among the output candidate answers, the answer with the highest score is not the correct answer. Although the correct answer can be predicted by the model, its ranking is relatively low and thus it is not selected as the final answer.

[0092] In the process of implementing the present disclosure, it is found that in the related art, the multiple answers obtained by using the output result of the pre-trained prediction model are simply sorted according to the probabilities of the starting position and the ending position predicted by the model, and the accuracy of the answers is relatively low. Moreover, the general improvement idea in the related art lies in how to more accurately determine the starting position and the ending position of the candidate answers, but the effect is not ideal. In view of this, the embodiments of the present disclosure provide a method for processing machine reading comprehension data, which performs an accuracy evaluation on multiple candidate answers based on the output result of the pre-trained prediction model to improve the accuracy of the prediction result.

[0093] Figure 2 The flowchart of the method for processing machine reading comprehension data according to the embodiments of the present disclosure is schematically shown.

[0094] As Figure 2As shown, the method includes operations S201 to S204.

[0095] In operation S201, multiple candidate answers are input into the self-evaluation model.

[0096] In operation S202, in the self-evaluation model, the evaluation results of each candidate answer are calculated; specifically, calculating the evaluation results of each candidate answer includes performing the following operations on each candidate answer respectively: calculating the number of intersections between the target candidate answer and other candidate answers among the multiple candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each other candidate answer; calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answers.

[0097] In operation S203, the final answer is determined from the multiple candidate answers according to the evaluation results of each candidate answer.

[0098] In operation S204, the final answer is output through the self-evaluation model.

[0099] According to an embodiment of the present disclosure, the multiple candidate answers may be obtained by outputting the results of a pre-trained prediction model. For example, through model prediction, the probability that each word in the text to be tested is the starting position of the correct answer to a preset question and the probability that each word is the ending position of the correct answer to the preset question are output, and after determining the score of each word according to the probability magnitude, multiple candidate answers with a preset quantity are obtained according to the score from high to low.

[0100] According to an embodiment of the present disclosure, in the above operation S202, calculating the number of intersections between the target candidate answer and other candidate answers among the multiple candidate answers may be calculating the number of overlapping words between the target candidate answer and each other candidate answer. For example, candidate answer 1 is: ate 5 apples, candidate answer 2 is: ate a total of 5 apples today, and the overlapping words between candidate answer 1 and candidate answer 2 are: ate 5 apples, then the number of intersections is: 6. According to an embodiment of the present disclosure, the text to be tested may be a text in multiple languages, and the "word" therein represents the smallest unit that can express a clear meaning in the text. For example, when the text to be tested is in Chinese, each Chinese character represents a word, and when the text to be tested is in English, each English word represents a word.

[0101] According to an embodiment of the present disclosure, in the above operation S202, the evaluation result of the target candidate answer is calculated according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers. For example, the similarity between the target candidate answer and each other candidate answer can be calculated according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers, and then the accuracy rate of the target candidate answer can be calculated according to the similarity between the target candidate answer and each other candidate answer.

[0102] According to an embodiment of the present disclosure, in the above operation S203, the evaluation result of each candidate answer can be an accuracy score result. According to the evaluation result of each candidate answer, the final answer is determined from multiple candidate answers, and the candidate answer with the highest accuracy score can be selected as the final answer.

[0103] According to an embodiment of the present disclosure, by calculating the number of intersections between the target candidate answer and other candidate answers among multiple candidate answers, and then calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers, it can be seen that this method is based on the number of intersections and combines the number of words for calculation and analysis. Since the number of intersections takes into account the same part between two answers, therefore, the evaluation result of the candidate answer calculated based on the number of intersections and combined with the number of words takes into account the similarity factor between the candidate answer and each other candidate answer. Also, because the part with higher similarity has a higher possibility of being the correct answer, the accuracy of the prediction result can be improved to a certain extent. Further, since the evaluation result of each candidate answer fully considers the similarity between every two answers, that is, fully considers the possibility that each candidate answer is the correct answer. Compared with the related art, in the related art, when considering the accuracy of each candidate answer, only the accuracy of the current candidate answer itself is considered, while the method described in the embodiment of the present disclosure fully comprehensively considers the possibility that each candidate answer is the correct answer, so the accuracy of the prediction is improved. It can be seen that the machine reading comprehension data processing method described in the embodiment of the present disclosure evaluates the prediction result of the pre-trained model through self-evaluation without additional input, and can achieve the technical effect and purpose of calibrating the model output result.

[0104] Figure 3 Schematically shows a flowchart of calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in other candidate answers according to an embodiment of the present disclosure.

[0105] As Figure 3 shown, the method includes operations S301 to S303.

[0106] In operation S301, according to the number of intersections and the number of words in other candidate answers, calculate the first overlap rate of the target candidate answer relative to other candidate answers, where the first overlap rate is used to characterize: the degree of similarity between the target candidate answer and other candidate answers based on other candidate answers. According to an embodiment of the present disclosure, calculating the first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers includes: calculating the ratio of the number of intersections to the number of words in other candidate answers to obtain the first overlap rate.

[0107] In operation S302, according to the number of intersections and the number of words in the target candidate answer, calculate the second overlap rate of the target candidate answer relative to other candidate answers, where the second overlap rate is used to characterize: the degree of similarity between other candidate answers and the target candidate answer based on the target candidate answer. According to an embodiment of the present disclosure, calculating the second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer includes: calculating the ratio of the number of intersections to the number of words in the target candidate answer to obtain the second overlap rate.

[0108] In operation S303, calculate the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate. According to an embodiment of the present disclosure, calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate may be obtained by calculating the product of the first overlap rate and the second overlap rate, or may be obtained by calculating the sum of the first overlap rate and the second overlap rate, or may be obtained according to the product of the first overlap rate and the second overlap rate and the sum of the first overlap rate and the second overlap rate.

[0109] For ease of understanding, the following is an exemplary illustration of the technical solutions described in the above embodiments. Still taking the following text to be measured and preset questions as examples, the description is as follows:

[0110] The text to be measured is:

[0111] Zhang San became an apprentice following his father and recorded the conditions of some patients. His elder brother Li Si entered University A three years ago. Zhang San felt unable to adapt, and Zhang San's friend arranged for Zhang San to go to University in City B. At that time, the medical school in City B exceeded University A in terms of both facilities and teaching staff. After his elder brother Li Si returned from University A, he was going to study at an off-campus hospital in City B. The two brothers then arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room.

[0112] The preset question is: What is Zhang San's house like in City B?

[0113] The correct answer is: A house with two bedrooms and a living room.

[0114] The word sets of the 5 candidate answers output by the pre-trained prediction model are as follows:

[0115] word1: A house with a living room;

[0116] word 2: The two brothers arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room;

[0117] word 3: A house with two bedrooms and a living room;

[0118] word 4: Arrived in City B together and rented a house with two bedrooms and a living room;

[0119] word 5: Rented a house with two bedrooms.

[0120] Next, only the method for evaluating the target candidate answer word1 is introduced. The evaluation methods for the remaining candidate answers are the same as those for word1 and will not be elaborated here.

[0121] (1) First, calculate the size (number of words) of the word set of each of the five answers. num1 = 8, num2 = 36, num3 = 10, num4 = 22, num5 = 10.

[0122] And calculate the intersection number between the target candidate answer and the other candidate answers among the multiple candidate answers, that is, calculate how many words in the set word1 are included in the other four answers, and obtain the corresponding intersection number: num (1-2) = 8, num (1-3) = 5, num (1-4) = 5, num (1-5) = 1.

[0123] (2) Calculate the ratio of the intersection number to the number of words in the other candidate answers to obtain the first overlap rates respectively:

[0124]

[0125] Specifically for the above example:

[0126] R (1-2) = 8 / 36; R (1-3) = 5 / 10; R (1-4) = 5 / 22; R (1-5) = 1 / 10.

[0127] (3) Calculate the ratio of the intersection number to the number of words in the target candidate answer to obtain the second overlap rates respectively:

[0128]

[0129] Specifically for the above example:

[0130] P (1-2) = 8 / 8; P (1-3) = 5 / 8; P (1-4) = 5 / 8; P (1-5) = 1 / 8.

[0131] (4) According to the first overlap rate and the second overlap rate, calculate the evaluation result Self-F1 of the target candidate answer, which can be obtained by calculating the product of the first overlap rate and the second overlap rate, or by calculating the sum of the first overlap rate and the second overlap rate, or by calculating the product of the first overlap rate and the second overlap rate and the sum of the first overlap rate and the second overlap rate.

[0132] According to the embodiments of the present disclosure, the first overlap rate and the second overlap rate are respectively used to characterize the similarity degree between the target candidate answer and other candidate answers in different dimensions. By calculating the first overlap rate and the second overlap rate and then calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate, the similarity factors of double dimensions are considered. Since the part with a higher similarity is more likely to be the correct answer, based on considering the similarity factors of more than one dimension, the accuracy of the prediction result can be improved.

[0133] According to the embodiments of the present disclosure, wherein calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate includes:

[0134] Calculate the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate;

[0135] Calculate the evaluation result of the target candidate answer according to the accuracy score.

[0136] According to the embodiments of the present disclosure, according to the first overlap rate R (1-i) and the second overlap rate P (1-i) , calculate the accuracy score of the target candidate answer relative to other candidate answers, which can be obtained by calculating the product of the first overlap rate and the second overlap rate, or by calculating the sum of the first overlap rate and the second overlap rate, or by calculating the product of the first overlap rate and the second overlap rate and the sum of the first overlap rate and the second overlap rate.

[0137] For example: Calculate the accuracy scores of the target candidate answer relative to other candidate answers respectively:

[0138] F1 (1-i) = preset coefficient * R (1-i) * P (1-i) Formula (III)

[0139] Or calculate separately:

[0140] F1 (1-i) = preset coefficient * (R (1-i) + P (1-i) ) Formula (4)

[0141] Or calculate separately:

[0142]

[0143] Among them, in the above Formulas (3), (4), and (5), i = 2, 3, 4, 5.

[0144] Then, based on each F1 (1-i) obtain the evaluation result Self-F1 of the target candidate answer word1.

[0145] According to the embodiments of the present disclosure, wherein, calculating the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate includes:

[0146] Calculate the accuracy score of the target candidate answer relative to other candidate answers according to the product of the preset coefficient, the first overlap rate and the second overlap rate, and the sum of the first overlap rate and the second overlap rate.

[0147] For example, referring to the above Formula (5), calculate each accuracy score F1 (1-i) :

[0148]

[0149] According to the embodiments of the present disclosure, wherein, one accuracy score corresponds to one other candidate answer, and wherein, calculating the evaluation result of the target candidate answer according to the accuracy score includes: calculating the average value of multiple accuracy scores to obtain the evaluation result of the target candidate answer.

[0150] For example, still taking the above example as an example, calculate the evaluation result of the target candidate answer word1:

[0151]

[0152] Among them, in the above Formula (6), n is the number of candidate answers.

[0153] According to the embodiments of the present disclosure, since the accuracy score of each candidate answer is obtained according to the similarity between every two answers, the possibility of whether each candidate answer is the correct answer is fully considered. On this basis, by calculating the average value of multiple accuracy scores to obtain the evaluation result of the candidate answer, the influence of the accuracy score of each candidate answer on the evaluation result is further fully considered, and the accuracy of the test result is further improved.

[0154] Figure 4 Schematically shows a flowchart of a machine reading comprehension data processing method according to another embodiment of the present disclosure.

[0155] As Figure 4 shown, the machine reading comprehension data processing method includes:

[0156] (1) Input the text to be tested and a preset question proposed for the text to be tested into a pre-trained language model. It can be to splice the text to be tested and the preset question, and distinguish between the text to be tested and the preset question through a special separator symbol "[SEP]". Among them, the pre-trained model can adopt the ELECTRA model, and the ELECTRA model consists of 24 layers of Transformer-based encoders.

[0157] (2) In the ELECTRA model, regard the encoded vector obtained from the last layer encoder of ELECTRA as the feature representation of the input text. The text feature representation will be used to predict the start position and end position of the answer in the article. Taking the prediction of the start position as an example, the text features pass through the parameter matrix W start and are projected into the start position prediction space to obtain a vector with a total length equal to the length of the input text. The number at each position of the vector represents the probability that this position is predicted as the start position. Similarly, the same method is used to predict the end position, except that a different parameter matrix W end is required.

[0158] Output through the pre-trained language model: the start probability and end probability of each word in the text to be tested, where the start probability of each word is: the probability that each word is the start position of the correct answer to the preset question, and the end probability of each word is: the probability that each word is the end position of the correct answer to the preset question.

[0159] (3) Determine multiple candidate answers to the preset question according to the start probability and end probability of each word. For example, they can be sorted according to the probability level, and the top several ranked answers can be used as candidate answers.

[0160] (4) Input the multiple candidate answers into the self-evaluation model. In the self-evaluation model, perform the evaluation operations as Figure 2 shown to obtain the evaluation results of each candidate answer, and determine the final answer from the multiple candidate answers according to the evaluation results of each candidate answer.

[0161] Still taking the text to be tested as an example: Zhang San became an apprentice following his father and recorded the conditions of some patients. His elder brother Li Si entered University A three years ago. Zhang San felt unable to adapt, and Zhang San's friend arranged for Zhang San to go to City B to study at university. At that time, the medical school in City B had better facilities and teaching staff than University A. After his elder brother Li Si returned from University A, he was going to study at an off-campus hospital in City B. The two brothers then arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room.

[0162] The preset question is: What is Zhang San's house like in City B?

[0163] The correct answer is: A house with two bedrooms and a living room.

[0164] After the self-evaluation, the answers output after sorting according to the evaluation results of each candidate answer are:

[0165] TOP1: A house with two bedrooms and a living room (correct answer);

[0166] TOP2: Rented a house with two bedrooms (lack of information).

[0167] TOP3: Arrived in City B together and rented a house with two bedrooms and a living room (redundant information);

[0168] TOP4: A house with a living room (lack of information);

[0169] TOP5: The two brothers arrived in City B together at the end of October of that year and rented a house with two bedrooms and a living room (redundant information);

[0170] Among them, the answer ranked first is the correct answer. It can be seen that through this evaluation method, the correct answer can be predicted relatively accurately, achieving the purpose of calibrating the answer.

[0171] Another aspect of the present disclosure provides a machine reading comprehension data processing device, Figure 5 Schematically shows a block diagram of a machine reading comprehension data processing device 500 according to an embodiment of the present disclosure.

[0172] The machine reading comprehension data processing device 500 can be used to implement the method Figure 2 shown in the reference.

[0173] As Figure 5 shown, the machine reading comprehension data processing device 500 includes: a first input module 501, a calculation module 502, a first determination module 503, and a first output module 504.

[0174] Among them, the first input module 501 is configured to input multiple candidate answers into the self-evaluation model;

[0175] The calculation module 502 is configured to calculate the evaluation results of each candidate answer in the self-evaluation model. Calculating the evaluation results of each candidate answer includes performing the following operations on each candidate answer respectively: calculating the number of intersections between the target candidate answer and other candidate answers among the multiple candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each other candidate answer; calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answer.

[0176] The first determination module 503 is configured to determine the final answer from the multiple candidate answers according to the evaluation results of each candidate answer;

[0177] The first output module 504 is configured to output the final answer through the self-evaluation model.

[0178] According to an embodiment of the present disclosure, in the calculation module 502, by calculating the number of intersections between a candidate answer and other candidate answers among the multiple candidate answers, and then calculating the evaluation result of the candidate answer according to the number of intersections, the number of words in the candidate answer, and the number of words in the other candidate answer, it can be seen that the device performs calculation and analysis based on the number of intersections in combination with the number of words. Since the number of intersections takes into account the common part between two answers, therefore, the evaluation result of the candidate answer calculated based on the number of intersections in combination with the number of words takes into account the similarity factor between the candidate answer and each other candidate answer. Since the part with higher similarity has a higher possibility of being the correct answer, the accuracy of the prediction result can be improved to a certain extent. Further, since the evaluation result of each candidate answer fully considers the similarity between every two answers, that is, fully considers the possibility that each candidate answer is the correct answer. Compared with the related art, in the related art, when considering the accuracy of each candidate answer, only the accuracy of the current candidate answer itself is considered, while the device described in the embodiment of the present disclosure fully comprehensively considers the possibility that each candidate answer is the correct answer, so the accuracy of the prediction is improved. It can be seen that the machine reading comprehension data processing device described in the embodiment of the present disclosure evaluates the prediction result of the pre-trained model through self-evaluation without additional input, and can achieve the technical effect and purpose of calibrating the model output result.

[0179] According to an embodiment of the present disclosure, among them, the calculation module 502 includes a first calculation unit, a second calculation unit, and a third calculation unit.

[0180] Among them, the first calculation unit is configured to calculate a first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers, where the first overlap rate is used to characterize: the similarity degree between the target candidate answer and other candidate answers based on other candidate answers.

[0181] The second calculation unit calculates a second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer, where the second overlap rate is used to characterize: the similarity degree between other candidate answers and the target candidate answer based on the target candidate answer.

[0182] The third calculation unit calculates an evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate.

[0183] According to an embodiment of the present disclosure, among them, in the first calculation unit, calculating the first overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in other candidate answers includes: calculating the ratio of the number of intersections to the number of words in other candidate answers to obtain the first overlap rate.

[0184] According to an embodiment of the present disclosure, among them, in the second calculation unit, calculating the second overlap rate of the target candidate answer relative to other candidate answers according to the number of intersections and the number of words in the target candidate answer includes: calculating the ratio of the number of intersections to the number of words in the target candidate answer to obtain the second overlap rate.

[0185] According to an embodiment of the present disclosure, among them, the third calculation unit includes a first calculation subunit and a second calculation subunit.

[0186] Among them, the first calculation subunit calculates an accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate. The second calculation subunit calculates an evaluation result of the target candidate answer according to the accuracy score.

[0187] According to an embodiment of the present disclosure, among them, in the first calculation subunit, calculating the accuracy score of the target candidate answer relative to other candidate answers according to the first overlap rate and the second overlap rate includes:

[0188] Calculating an accuracy score of the target candidate answer relative to other candidate answers according to a preset coefficient, the product of the first overlap rate and the second overlap rate, and the sum of the first overlap rate and the second overlap rate.

[0189] According to an embodiment of the present disclosure, where one accuracy score corresponds to one other candidate answer, and among them, in the second calculation subunit, calculating the evaluation result of the candidate answer according to the accuracy score includes: calculating the average value of multiple accuracy scores to obtain the evaluation result of the target candidate answer.

[0190] According to an embodiment of the present disclosure, the machine reading comprehension data processing device 500 further includes a second input module, a second output module, and a second determination module.

[0191] According to an embodiment of the present disclosure, it further includes a second input module, a second output module, and a second determination module.

[0192] The second input module is configured to input the text to be tested and a preset question proposed for the text to be tested into a pre-trained language model.

[0193] The second output module is configured to output, through the pre-trained language model: the start probability and the end probability of each word in the text to be tested, where the start probability of each word is the probability that each word is the start position of the correct answer to the preset question, and the end probability of each word is the probability that each word is the end position of the correct answer to the preset question.

[0194] The second determination module is configured to determine multiple candidate answers to the preset question according to the start probability and the end probability of each word.

[0195] According to an embodiment of the present disclosure, any plurality of modules, sub-modules, units, and sub-units, or at least part of the functions of any of them can be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system in a package, an application specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in a suitable combination of any several of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions can be executed.

[0196] For example, any combination of the first input module 501, the calculation module 502, the first determination module 503, and the first output module 504 can be implemented in one module / unit / sub-unit, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first input module 501, the calculation module 502, the first determination module 503, and the first output module 504 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner that can integrate or package circuits, etc., implemented by hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the first input module 501, the calculation module 502, the first determination module 503, and the first output module 504 can be at least partially implemented as a computer program module, which can execute corresponding functions when the computer program module is run.

[0197] Figure 6 FIG. schematically shows a block diagram of an electronic device for implementing a machine reading comprehension data processing method according to an embodiment of the present disclosure. Figure 6 The illustrated electronic device is only an example and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.

[0198] As Figure 6 shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The processor 601 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application specific integrated circuit (ASIC)), and so on. The processor 601 can also include on-board memory for caching purposes. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0199] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 602 and / or the RAM 603. It should be noted that the programs may also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also perform various operations of the method flow according to the embodiments of the present disclosure by executing programs stored in the one or more memories.

[0200] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, and the input / output (I / O) interface 605 is also connected to the bus 604. The system 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom can be installed into the storage portion 608 as needed.

[0201] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable storage medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system according to the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules.

[0202] The present disclosure also provides a computer-readable storage medium, which may be included in the device / device / system described in the above embodiments; or may exist separately without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0203] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0204] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the above-described ROM 602 and / or RAM 603 and / or ROM 602 and RAM 603.

[0205] An embodiment of the present disclosure further includes a computer program product, which includes a computer program that contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to cause the electronic device to implement the machine reading comprehension data processing method provided by the embodiment of the present disclosure.

[0206] When the computer program is executed by the processor 601, the above functions defined in the system / apparatus of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0207] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 609, and / or installed from the removable medium 611. The program code contained in the computer program may be transmitted by any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0208] In accordance with embodiments of the present disclosure, program code for executing the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly recited in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features recited in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0210] The above describes the embodiments of the present disclosure. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. A method for processing machine reading comprehension data, comprising: Inputting multiple candidate answers into a self-evaluation model; Calculating the evaluation results of each of the candidate answers in the self-evaluation model; Determining a final answer from the multiple candidate answers according to the evaluation results of each of the candidate answers; Outputting the final answer through the self-evaluation model; Wherein, calculating the evaluation results of each of the candidate answers includes performing the following operations on each of the candidate answers respectively: Calculating the number of intersections between a target candidate answer and other candidate answers among the multiple candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each of the other candidate answers; Calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answers.

2. The method according to claim 1, wherein, Calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answers includes: Calculating a first overlap rate of the target candidate answer relative to the other candidate answers according to the number of intersections and the number of words in the other candidate answers, where the first overlap rate is used to represent: the degree of similarity between the target candidate answer and the other candidate answers based on the other candidate answers; Calculating a second overlap rate of the target candidate answer relative to the other candidate answers according to the number of intersections and the number of words in the target candidate answer, where the second overlap rate is used to represent: the degree of similarity between the other candidate answers and the target candidate answer based on the target candidate answer; Calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate.

3. The method according to claim 2, wherein, The calculating the first overlap rate of the target candidate answer relative to the other candidate answers according to the number of intersections and the number of words in the other candidate answers includes: Calculating the ratio of the number of intersections to the number of words in the other candidate answers to obtain the first overlap rate.

4. The method according to claim 2, wherein The calculating the second overlap rate of the target candidate answer relative to the other candidate answers according to the number of intersections and the number of words in the target candidate answer includes: Calculating the ratio of the number of intersections to the number of words in the target candidate answer to obtain the second overlap rate.

5. The method according to claim 2, wherein, The calculating the evaluation result of the target candidate answer according to the first overlap rate and the second overlap rate includes: Calculating an accuracy score of the target candidate answer relative to the other candidate answers according to the first overlap rate and the second overlap rate; Calculating the evaluation result of the target candidate answer according to the accuracy score.

6. The method according to claim 5, wherein The calculating the accuracy score of the target candidate answer relative to the other candidate answers according to the first overlap rate and the second overlap rate includes: Calculating the accuracy score of the target candidate answer relative to the other candidate answers according to a preset coefficient, the product of the first overlap rate and the second overlap rate, and the sum of the first overlap rate and the second overlap rate.

7. The method according to claim 5, wherein, One of the accuracy scores corresponds to one of the other candidate answers, and calculating the evaluation result of the target candidate answer according to the accuracy score includes: Calculating the average of multiple accuracy scores to obtain the evaluation result of the target candidate answer.

8. The method according to claim 1, further comprising: Inputting the text to be tested and a preset question proposed for the text to be tested into a pre-trained language model; Outputting by the pre-trained language model: the start probability and the end probability of each word in the text to be tested, where the start probability of each word is the probability that the word is at the start position of the correct answer to the preset question, and the end probability of each word is the probability that the word is at the end position of the correct answer to the preset question; and Determining multiple candidate answers to the preset question according to the start probability and the end probability of each word.

9. A machine reading comprehension data processing device, comprising: A first input module for inputting multiple candidate answers into a self-evaluation model; A calculation module for calculating the evaluation result of each candidate answer in the self-evaluation model; A determination module for determining a final answer from the multiple candidate answers according to the evaluation result of each candidate answer; A first output module for outputting the final answer through the self-evaluation model; where calculating the evaluation result of each candidate answer includes performing the following operations on each candidate answer respectively: Calculating the number of intersections between the target candidate answer and other candidate answers among the multiple candidate answers, where the number of intersections is used to represent the number of overlapping words between the target candidate answer and each of the other candidate answers; Calculating the evaluation result of the target candidate answer according to the number of intersections, the number of words in the target candidate answer, and the number of words in the other candidate answer.

10. An electronic device, comprising: One or more processors; A memory for storing one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor implements the method according to any one of claims 1 to 8.

12. A computer program product, the computer program product includes computer-executable instructions, and the instructions are used to implement the method according to any one of claims 1 to 8 when executed.

Images