Processing system, processing method, and program

Abstract

To provide a processing system capable of evaluating an answer sentence generated in response to a question sentence of an inquiry from a user. [Solution] The processing system comprises a first generation means that has performed additional learning on past performance data, a second generation means that has not performed the additional learning, and a first calculation means that calculates a first document similarity which is the similarity between a first answer sentence that is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence that is generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, and which is the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence.

Description

[Technical Field]

[0001] The present disclosure relates to a processing system, a processing method, and a program. [Background technology]

[0002] Large-scale language models are used in various technical fields. Patent Document 1 discloses, as a related technique, a technique relating to a sentence generation device that generates sentences to be input as prompts to a large-scale language model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7313757 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technical field related to Patent Document 1, there is a demand for a technology that can evaluate answers generated in response to questions from users.

[0005] One of the objectives of each aspect of the present disclosure is to provide a processing system, a processing method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a processing system includes: The system includes a first generation means that has undergone additional learning and has performed additional learning on past performance data, a second generation means that has not performed the additional learning, a first calculation means that calculates a first document similarity, which is a similarity relating to a first answer sentence that is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence that is generated by the second generation means when the same question sentence is input to the second generation means and includes the conclusion sentence and the reason sentence, and is the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence, a second calculation means, and a determination means, wherein the second generation means generates an answer sentence that includes the conclusion sentence and the reason sentence as a third answer sentence when the reason sentence included in the first answer sentence is input as additional information, and the second calculation means calculates a second document similarity, which is the similarity between the first answer sentence and the conclusion sentence included in the third answer sentence, and the determination means determines whether or not the first answer sentence is correct based on a result of comparing the first document similarity with the second document similarity. .

[0007] According to another aspect of the present disclosure, a processing method includes: A processing method executed by a processing system including a first generation means that has undergone additional learning by performing additional learning on past performance data, a second generation means that has not performed the additional learning, a first calculation means, a second calculation means, and a determination means, wherein the first calculation means calculates a similarity between a first answer sentence that is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence that is generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, calculating a first document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; when the reason sentence included in the first answer sentence is input as additional information by the second generation means, generating an answer sentence including a conclusion sentence and a reason sentence as a third answer sentence; calculating a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence by the second calculation means; and determining whether or not the first answer sentence is correct based on a comparison result between the first document similarity and the second document similarity. .

[0008] According to another aspect of the present disclosure, a program includes: A computer of a processing system including a first generation means that has undergone additional learning by performing additional learning on past performance data, a second generation means that has not performed the additional learning, a first calculation means, a second calculation means, and a determination means, wherein the first calculation means calculates a similarity between a first answer sentence that is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence that is generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, The computer program executes the following: calculating a first document similarity which is a similarity between a first answer sentence and the conclusion sentence included in the second answer sentence; generating an answer sentence including a conclusion sentence and a reason sentence as a third answer sentence when the reason sentence included in the first answer sentence is input as additional information by the second generating means; calculating a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence by the second calculating means; and determining whether or not the first answer sentence is correct based on a comparison result between the first document similarity and the second document similarity. . [Effects of the Invention]

[0009] According to each aspect of the present disclosure, it is possible to evaluate an answer sentence generated in response to a question sentence of an inquiry from a user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a configuration of a processing system according to some embodiments of the present disclosure. [Figure 2] FIG. 10 illustrates an example of a prompt according to some embodiments of the present disclosure. [Figure 3] FIG. 2 is a diagram illustrating the difference between a first generated AI and a second generated AI according to some embodiments of the present disclosure. [Figure 4] FIG. 10 illustrates an example of a prompt according to some embodiments of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating an example of a decision table according to some embodiments of the present disclosure. [Figure 6] FIG. 10 illustrates an example of a user query according to some embodiments of the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating a first example of a processing flow of a processing system according to some embodiments of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating a second example of a processing flow of a processing system according to some embodiments of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating a third example of a processing flow of a processing system according to some embodiments of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating a first example of verification data according to some embodiments of the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating a second example of verification data according to some embodiments of the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating a third example of verification data according to some embodiments of the present disclosure. [Figure 13] FIG. 1 illustrates an example of a configuration of a processing system according to some embodiments of the present disclosure. [Figure 14] FIG. 1 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure. [Figure 15] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. <Embodiment> A processing system 1 according to an embodiment of the present disclosure will be described with reference to the drawings. The processing system 1 is a system capable of determining whether an answer generated by a generation AI (Artificial Intelligence) in response to a question message for an inquiry from a user U is correct. The user in the embodiment of the present disclosure is a customer who has made an inquiry. The processing system 1 is also a system that provides an operator with hints for using the answer generated by the generation AI to assist in creating a correct answer. The operator is a worker who rewrites an answer when the answer generated by the generation AI is evaluated as inappropriate. The operator is also a worker who responds to inquiries from users. The processing system 1 is used in highly specialized technical fields, such as responding to failures in ICT (Information and Communication Technology) systems, for example.

[0012] (Processing system configuration) 1 is a diagram illustrating an example of a configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in FIG. 1, the processing system 1 includes an information input unit 10, a first generation unit 20, a second generation unit 30, an evaluation unit 40, a notification unit 50, and an answer providing unit 60.

[0013] The information input unit 10 receives an inquiry from a user U. The information input unit 10 processes a question Q10 of the received inquiry into a prompt P201. The information input unit 10 then outputs the prompt P201 to the first generation unit 20 and the second generation unit 30.

[0014] The inquiry includes the email address of user U. The information input unit 10 associates the inquiry with the email address by assigning the email address of user U as a unique inquiry ID (IDentification) for each inquiry. In the processing system 1, the question text, answer text, and related content are all associated with the unique inquiry ID. Therefore, in the following explanation, the explanation of the process of assigning and associating the unique inquiry ID will be omitted.

[0015] The prompt P201 includes a command statement that instructs the first generation AI201 (i.e., a large-scale language model) described below to generate an answer in the format of a "conclusion statement" and a "reason statement" in response to the question statement. FIG. 2 is a diagram illustrating an example of the prompt P201 according to some embodiments of the present disclosure. For example, as shown in FIG. 2, the prompt P201 has the format "You are a specialized operator who can output an answer to a question statement from a user U in an ICT system. Please create an answer to the following question statement. Please output the answer in the format of [conclusion statement] [reason statement]. [Question statement]."

[0016] The first generation unit 20 includes a first generation AI201 that has undergone additional training by additionally learning historical data of past inquiries. Each inquiry in the historical data of past inquiries used for the additional training has a format of a "question sentence," a "conclusion sentence," and a "reason sentence." The historical data of past inquiries used for the additional training is data accumulated in a highly specialized technical field, such as responding to failures in ICT systems, and is preferably data that is not publicly available, such as on the Internet. The first generation unit 20 receives a prompt P201 from the information input unit 10 as input to the first generation AI201. When the prompt P201 is input, the first generation AI201 generates an answer sentence A11. The first generation AI201 outputs the question sentence Q10 and the generated answer sentence A11 to the evaluation unit 40.

[0017] The second generation unit 30 includes a second generation AI301 (i.e., a large-scale language model) that has not undergone additional learning using performance data of past inquiries. The second generation unit 30 receives a prompt P201 from the information input unit 10 as input to the second generation AI301. When the prompt P201 is input, the second generation AI301 generates an answer sentence A12. The second generation AI301 outputs the question sentence Q10 and the generated answer sentence A12 to the evaluation unit 40.

[0018] The second generated AI301 is assumed to be no different from the first generated AI201 included in the first generation unit 20 except that additional learning using performance data of past inquiries has not been performed. In other words, the second generated AI301 may be considered to be the first generated AI201 that has not been additionally learned using performance data of past inquiries.

[0019] 3 is a diagram illustrating the difference between the first generation AI 201 and the second generation AI 301 according to some embodiments of the present disclosure. As shown in FIG. 3, the first generation AI 201 included in the first generation unit 20 is trained using a large amount of training data generally collected from an external source such as the Internet, and is additionally trained by additionally training using performance data of past inquiries. The second generation AI 301 included in the second generation unit 30 is trained by training using the same large amount of training data as the training data used by the first generation AI 201.

[0020] The evaluation unit 40 calculates a document similarity B11 indicating the degree of similarity between the conclusion sentence of the answer sentence A11 received from the first generation unit 20 and the conclusion sentence of the answer sentence A12 received from the second generation unit 30. The document similarity B11 here is a similarity score calculated using, for example, TF-IDF (Term Frequency-Inverse Document Frequency) vectorization and cosine similarity. The document similarity B11 is expressed as a similarity score ranging from 0 to 1. The closer the similarity score is to 1, the higher the similarity. For example, the document similarity B11 is divided into three levels: low (similarity score 0.0-0.4), medium (similarity score 0.4-0.6), and high (similarity score 0.6-1.0). Furthermore, the boundary values ​​of the similarity score, 0.4 and 0.6, are assumed to be included in the document similarity B11. In the embodiment of the present disclosure, the document similarity B11 is not limited to being calculated using TF-IDF vectorization and cosine similarity. For example, the document similarity B11 may be calculated using known WMD (Word Mover's Distance) or WRD (Word Rotator's Distance).

[0021] The evaluation unit 40 creates a prompt P202 using the reason sentence of the answer sentence A11 received from the first generation unit 20 as additional information for the question sentence Q10. The evaluation unit 40 outputs the created prompt P202 to the second generation unit 30. As a result, the evaluation unit 40 causes the second generation unit 30 to regenerate the answer sentence A13. When the second generation AI301 of the second generation unit 30 receives the prompt P202 from the evaluation unit 40, it generates the answer sentence A13. The evaluation unit 40 obtains the answer sentence A13 generated by the second generation AI301 from the second generation unit 30.

[0022] 4 is a diagram illustrating an example of a prompt P202 according to some embodiments of the present disclosure. As illustrated in FIG. 4, the prompt P202 includes a command statement that instructs the second generation AI 301 not to recognize the additional information as an absolute correct answer to the question, but to generate an answer in the format of a "conclusion statement" and a "reason statement" taking the additional information into consideration.

[0023] The evaluation unit 40 calculates a document similarity B12 indicating the degree of similarity between the conclusion sentence of the answer sentence A11 received from the first generation unit 20 and the conclusion sentence of the answer sentence A13 regenerated by the second generation unit 30.

[0024] Furthermore, the evaluation unit 40 compares the document similarity B11 and the document similarity B12 according to a determination table. Then, the evaluation unit 40 generates a determination result and a usage hint based on the comparison result. Fig. 5 is a diagram showing an example of the determination table according to some embodiments of the present disclosure. As shown in Fig. 5, the determination table associates a determination result and a usage hint for each combination of the document similarity B11 and the document similarity B12.

[0025] For example, when document similarity B11 is low and document similarity B12 is high, the judgment table associates a judgment result indicating a correct answer with the absence of a usage hint for the combination of document similarity B11 and document similarity B12. Also, when document similarity B11 is low and document similarity B12 is medium, the judgment table associates a judgment result indicating an incorrect answer with the usage hint, "The conclusion sentence of answer sentence A11 generated by the first generation AI may be correct but the reason sentence may be insufficient. It is recommended that you supplement the reason sentence to create a correct answer."

[0026] In an embodiment of the present disclosure, if the reason statement in the answer statement can sufficiently substantiate the conclusion statement of the answer statement, the answer statement is defined as a correct answer as a determination result, as shown in FIG. 5. In other cases (i.e., if the reason statement in the answer statement cannot sufficiently substantiate the conclusion statement of the answer statement), the answer statement is defined as an incorrect answer as a determination result, as shown in FIG. 5. When a first generation AI 201 that has undergone additional training using historical data from past inquiries outputs different answers to the same question statement, if the reason statement in the answer statement of the first generation AI 201 can sufficiently substantiate the conclusion statement of the answer statement itself, the reason statement can be provided to the second generation AI 301 as additional information, and the second generation AI 301 can be satisfied and re-output a new answer statement that is similar to the answer statement of the first generation AI 201. Based on this concept, inquiries from user U can be divided into seven patterns.

[0027] FIG. 6 is a diagram illustrating an example of an inquiry from a user U according to some embodiments of the present disclosure. For example, an inquiry from the user U can be divided into seven patterns as shown in FIG. 6. As a result, a combination of document similarity B11 and document similarity B12 can be identified for each of the seven patterns. Therefore, by identifying the identified combination of document similarity B11 and document similarity B12 in the determination table shown in FIG. 5, a determination process can be realized to obtain a determination result of correct or incorrect, and a usage hint for the determination result can be obtained.

[0028] The evaluation unit 40 determines whether the judgment result is correct or not. If the evaluation unit 40 determines that the judgment result is correct, it judges the answer sentence A11 to be the correct answer. In this case, the evaluation unit 40 outputs the judgment result that judges the answer sentence A11 to be the correct answer to the answer providing unit 60. On the other hand, if the evaluation unit 40 determines that the judgment result is incorrect, it determines that the answer sentence A11 is an incorrect answer. In this case, the judgment unit 40 outputs the question sentence Q10, the answer sentence A11, the answer sentence A12, the answer sentence A13, and the usage hint to the notification unit 50.

[0029] When data is input, the notification unit 50 creates an email. The notification unit 50 sends the created email to an operator. The email contents include the email address of the user U, which is a unique inquiry ID for inquiring with an operator and allowing the operator to take over the response, the question Q10, the answer A11, the answer A12, the answer A13, and a usage hint.

[0030] In the embodiment of the present disclosure, an example is given in which the notification unit 50 notifies the operator using email as a notification means. However, the notification unit 50 may notify the operator using a notification means other than email, and the notification means is not limited to email. The notification unit 50 may use any notification means as long as it can notify the operator of the question Q10, the answer A11, the answer A12, the answer A13, and the usage hint, and can allow the operator to take over the response based on the inquiry.

[0031] The answer providing unit 60 sends the answer sentence A11 received from the evaluation unit 40 to the email address of user U, which is the unique inquiry ID associated with the answer sentence A11. The answer providing unit 60 will be described as answering user U's inquiry using email. However, the answer providing unit 60 may answer user U's inquiry using an answering means other than email, and the answering means is not limited to email. The answer providing unit 60 may use any answering means as long as it can answer user U's inquiry.

[0032] The processing performed by the processing system 1 according to an embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0033] (Processing performed by the processing system) Fig. 7 is a diagram illustrating a first example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Fig. 8 is a diagram illustrating a second example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Fig. 9 is a diagram illustrating a third example of a processing flow of the processing system 1 according to some embodiments of the present disclosure.

[0034] (First process) First, the first processing performed by the processing system 1 will be described with reference to Fig. 7. The processing system 1 performs the following steps S1 to S6 as the first processing.

[0035] The information input unit 10 receives an inquiry from a user U (step S1). The information input unit 10 processes the received inquiry question Q10 into a prompt P201 (step S2). Then, the information input unit 10 outputs the prompt P201 to the first generation unit 20 and the second generation unit 30.

[0036] The first generation unit 20 receives the prompt P201 from the information input unit 10 as input to the first generation AI201. When the first generation AI201 receives the prompt P201, it generates an answer sentence A11 (step S3). The first generation AI201 outputs the question sentence Q10 and the generated answer sentence A11 to the evaluation unit 40 (step S4).

[0037] The second generation unit 30 receives the prompt P201 from the information input unit 10 and inputs it to the second generation AI301. When the second generation AI301 receives the prompt P201, it generates an answer sentence A12 (step S5). The second generation AI301 outputs the question sentence Q10 and the generated answer sentence A12 to the evaluation unit 40 (step S6).

[0038] (Second process) Next, the second process performed by the processing system 1 will be described with reference to Fig. 8. The processing system 1 performs the following steps S11 to S17 as the second process.

[0039] The evaluation unit 40 calculates a document similarity B11 indicating the degree of similarity between the conclusion sentence of the answer sentence A11 received from the first generation unit 20 and the conclusion sentence of the answer sentence A12 received from the second generation unit 30 (step S11). The evaluation unit 40 creates a prompt P202 using the reason sentence of the answer sentence A11 received from the first generation unit 20 as additional information for the question sentence Q10 (step S12). The evaluation unit 40 outputs the created prompt P202 to the second generation unit 30 (step S13). As a result, the evaluation unit 40 causes the second generation unit 30 to regenerate the answer sentence A13. When the second generation AI301 of the second generation unit 30 receives the prompt P202 from the evaluation unit 40, it generates the answer sentence A13. The evaluation unit 40 obtains the answer sentence A13 generated by the second generation AI301 from the second generation unit 30 (step S14).

[0040] The evaluation unit 40 calculates a document similarity B12 indicating the degree of similarity between the conclusion sentence of the answer sentence A11 received from the first generation unit 20 and the conclusion sentence of the answer sentence A13 regenerated by the second generation unit 30 (step S15).

[0041] The evaluation unit 40 also compares the document similarity B11 with the document similarity B12 in accordance with the decision table (step S16), and generates a decision result and a usage hint based on the comparison result (step S17).

[0042] (Third process) Next, the third process performed by the processing system 1 will be described with reference to Fig. 9. The processing system 1 performs the following steps S21 to S27 as the third process.

[0043] The evaluation unit 40 determines whether the judgment result is correct or not (step S21). If the evaluation unit 40 judges that the judgment result is correct (YES in step S21), it judges the answer sentence A11 to be the correct answer (step S22). In this case, the evaluation unit 40 outputs the judgment result that the answer sentence A11 is the correct answer to the answer providing unit 60. The answer providing unit 60 sends the answer sentence A11 received from the evaluation unit 40 to the email address of user U, which is the unique inquiry ID associated with the answer sentence A11 (step S23).

[0044] Furthermore, if the evaluation unit 40 determines that the judgment result is incorrect (NO in step S21), it judges the answer sentence A11 to be incorrect (step S24). In this case, the judgment unit 40 outputs the question sentence Q10, the answer sentence A11, the answer sentence A12, the answer sentence A13, and the usage hint to the notification unit 50 (step S25). The notification unit 50 creates an email including the email address of the user U, which is a unique inquiry ID, the question sentence Q10, the answer sentence A11, the answer sentence A12, the answer sentence A13, and the usage hint (step S26). The notification unit 50 sends the created email to an operator (i.e., an inquiry to the operator) to notify the operator to take over the response (step S27).

[0045] The operator may refer to the usage hint and use answer sentence A11, answer sentence A12, and answer sentence A13 to create a correct answer sentence to question sentence Q11. The operator may also send the created correct answer sentence to user U by email (i.e., answer).

[0046] Each of the first generation unit 20 and the second generation unit 30 may be realized by, for example, ChatGPT by OpenAI Corporation.

[0047] <Example> Here, an example of the processing system 1 according to an embodiment of the present disclosure will be described, in which ChatGPT by OpenAI Corporation is used for each of the first generation unit 20 and the second generation unit 30. In this example, the effect of the processing system 1 according to the present disclosure is shown. FIG. 10 is a diagram showing a first example of verification data according to some embodiments of the present disclosure. In the example of the processing system 1 according to an embodiment of the present disclosure, the verification data shown in FIG. 10 is used. The verification data shown in FIG. 10 includes a question Q10, an answer A11, and an answer A12.

[0048] The information input unit 10 receives an inquiry from a user U. The information input unit 10 processes a question Q10 of the inquiry into a prompt P201. The information input unit 10 then inputs the prompt P201 to the first generation unit 20 and the second generation unit 30.

[0049] The first generation unit 20 inputs the input prompt P201 into its own ChatGPT to generate an answer sentence A11. The second generation unit 30 inputs the input prompt P201 into its own ChatGPT to generate an answer sentence A12. As shown in Fig. 10, the conclusion sentence of the answer sentence A11 and the conclusion sentence of the answer sentence A12 have contradictory meanings.

[0050] The evaluation unit 40 creates a prompt P202 using the reason sentence of the answer sentence A11 as additional information. Then, the evaluation unit 40 outputs the created prompt P202 to the second generation unit 30. The ChatGPT included in the second generation unit 30 regenerates an answer sentence to the question sentence Q10. This answer sentence regenerated by the ChatGPT included in the second generation unit 30 becomes the answer sentence A13.

[0051] FIG. 11 is a diagram illustrating a second example of verification data according to some embodiments of the present disclosure. As illustrated in FIG. 11, the response by ChatGPT included in the second generation unit 30 has changed from response sentence A12 to response sentence A13 through regeneration. The conclusion sentence of response sentence A13 has the same meaning as the conclusion sentence of response sentence A11. Furthermore, the conclusion sentence of response sentence A13 has a meaning opposite to the conclusion sentence of response sentence A12. This may be interpreted as the response by ChatGPT included in the second generation unit 30 having changed to something closer to the response by ChatGPT included in the first generation unit 20.

[0052] 12 is a diagram illustrating a third example of verification data according to some embodiments of the present disclosure. As illustrated in FIG. 12, when the similarity score calculated by the evaluation unit 40 using TF-IDF (Term Frequency-Inverse Document Frequency) vectorization and cosine similarity is used as the document similarity, the document similarity B11 between the conclusion sentence of the answer sentence A11 and the conclusion sentence of the answer sentence A12 is 0.291, which is low. Furthermore, the document similarity B12 between the conclusion sentence of the answer sentence A11 and the conclusion sentence of the answer sentence A13 is 1.0, which is high. The evaluation unit 40 determines that the answer sentence A11 is correct according to the determination table shown in FIG. 5.

[0053] (advantage) The processing system 1 according to an embodiment of the present disclosure has been described above. The processing system 1 includes a first generation unit 20 (an example of a first generation means) that has undergone additional learning by performing additional learning on past performance data, a second generation unit 30 (an example of a second generation means) that has not performed the additional learning, and an evaluation unit 40 (an example of a first calculation means) that calculates a first document similarity between a first answer sentence generated by the first generation unit 20 when a question sentence is input to the first generation unit 20, the first answer sentence including a conclusion sentence and a reason sentence, and a second answer sentence generated by the second generation unit 30 when the same question sentence is input to the second generation unit 30, the second answer sentence including the conclusion sentence and the reason sentence. The first document similarity is the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence. This processing system 1 can evaluate the answer sentences generated in response to a question sentence from a user.

[0054] In addition, the above-mentioned processing system 1 can solve the problem that in highly specialized fields such as responding to ICT (Information and Communication Technology) system failures, it is not possible to automatically determine whether an answer generated by a generation AI to a question from a user is correct, and if the answer generated by the generation AI is incorrect, how to reuse the answer generated by the generation AI depends on the experience of the operator.

[0055] Next, a processing system 700 according to some embodiments of the present disclosure will be described. Fig. 13 is a diagram illustrating an example of the configuration of the processing system 700 according to some embodiments of the present disclosure. As shown in Fig. 13, the processing system 700 includes a first generating means 701, a second generating means 702, and a first calculating means 703.

[0056] The first generation means 701 is a means that has performed additional learning on past performance data. The second generation means 702 is a means that has not performed the additional learning. The first calculation means 703 calculates a first document similarity that is a similarity related to a first answer sentence that is an answer sentence generated by the first generation means 701 when a question sentence is input to the first generation means 701 and includes a conclusion sentence and a reason sentence, and a second answer sentence that is an answer sentence that is generated by the second generation means 702 when the same question sentence is input to the second generation means 702 and includes a conclusion sentence and a reason sentence, and is the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence.

[0057] The processing system 700 can be realized, for example, by using the functions of the processing system 1 illustrated in Fig. 1. The first generating means 701 can be realized, for example, by using the functions of the first generating unit 20 illustrated in Fig. 1. The second generating means 702 can be realized, for example, by using the functions of the second generating unit 30 illustrated in Fig. 1. The first calculating means 703 can be realized, for example, by using the functions of the evaluating unit 40 illustrated in Fig. 1.

[0058] Next, a description will be given of processing performed by the processing system 700 according to some embodiments of the present disclosure. Fig. 14 is a diagram showing an example of a processing flow of the processing system 700 according to some embodiments of the present disclosure. Here, the processing of the processing system 700 will be described with reference to Fig. 14.

[0059] In a processing system 700 including a first generation means 701 that has undergone additional learning by performing additional learning on past performance data, and a second generation means 702 that has not performed the additional learning, a first calculation means 703 calculates a first document similarity which is the similarity relating to a first answer sentence that is generated by the first generation means 701 when a question sentence is input to the first generation means 702 and includes a conclusion sentence and a reason sentence, and a second answer sentence that is generated by the second generation means 702 when the same question sentence is input to the second generation means 702 and includes a conclusion sentence and a reason sentence, and which is the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence (step S101).

[0060] The processing system 700 according to some embodiments of the present disclosure has been described above. The processing system 700 can evaluate answer sentences generated in response to question sentences from users.

[0061] The order of the processes in each embodiment of the present disclosure may be changed as long as the processes are performed appropriately.

[0062] Each embodiment of the present disclosure has been described, but the processing system 1, information input unit 10, first generation unit 20, second generation unit 30, evaluation unit 40, notification unit 50, answer providing unit 60, and other control devices may have a computer system inside. The above-mentioned processing steps are stored in the form of a program on a computer-readable recording medium, and the above processing is performed by reading and executing this program by a computer. Specific examples of computers are shown below.

[0063] 15 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 15, the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9.

[0064] For example, the above-described processing system 1, information input unit 10, first generation unit 20, second generation unit 30, evaluation unit 40, notification unit 50, answer providing unit 60, and other control devices are each implemented in a computer 5. The operations of each of the above-described processing units are stored in the form of a program in storage 8. CPU 6 reads the program from storage 8, loads it into main memory 7, and executes the above-described processing in accordance with the program. Furthermore, CPU 6 allocates storage areas in main memory 7 corresponding to each of the above-described storage units in accordance with the program.

[0065] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. In addition, when this program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-mentioned processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0066] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already recorded in the computer system, a so-called differential file (differential program).

[0067] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0068] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0069] (Appendix 1) a first generation means that has undergone additional learning by performing additional learning on past performance data; a second generation means that has not performed the additional learning; a first calculation means for calculating a first document similarity between a first answer sentence, which is an answer sentence generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence, which is an answer sentence generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; A processing system comprising:

[0070] (Appendix 2) The second generating means When the reason sentence included in the first answer sentence is input as additional information, an answer sentence including a conclusion sentence and a reason sentence is generated as a third answer sentence. 10. The processing system of claim 1.

[0071] (Appendix 3) a second calculation means for calculating a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; 3. The processing system of claim 2, comprising:

[0072] (Appendix 4) a determination means for determining whether the first answer sentence is correct or not based on a comparison result between the first document similarity and the second document similarity; 4. The processing system of claim 3, comprising:

[0073] (Appendix 5) a comparison means for comparing the first document similarity with the second document similarity; Equipped with The determination means determining whether the first answer sentence is correct or not based on the comparison result by the comparison means; 5. The processing system of claim 4.

[0074] (Appendix 6) A processing method executed by a processing system including a first generation means that has performed additional learning on past performance data and a second generation means that has not performed the additional learning, calculating a first document similarity between a first answer sentence which is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence which is generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; A processing method comprising:

[0075] (Appendix 7) generating a third answer sentence including a conclusion sentence and a reason sentence when the reason sentence included in the first answer sentence is input as additional information; 6. The method of claim 5, further comprising:

[0076] (Appendix 8) calculating a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; Attachment 7, a processing method comprising:

[0077] (Appendix 9) determining whether the first answer sentence is correct based on a comparison result between the first document similarity and the second document similarity; Attachment 8, a processing method comprising:

[0078] (Appendix 10) comparing the first document similarity with the second document similarity; determining whether the first answer sentence is correct based on a comparison result; Attachment 9, a processing method comprising:

[0079] (Appendix 11) A computer of a processing system including a first generation means that has performed additional learning on past performance data and a second generation means that has not performed the additional learning, calculating a first document similarity between a first answer sentence which is generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence which is generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; A program that executes the following.

[0080] (Appendix 12) generating a third answer sentence including a conclusion sentence and a reason sentence when the reason sentence included in the first answer sentence is input as additional information; 12. The program according to claim 11, which causes the computer to execute the above steps.

[0081] (Appendix 13) calculating a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; 13. The program according to claim 12, which causes the computer to execute the above steps.

[0082] (Appendix 14) determining whether the first answer sentence is correct based on a comparison result between the first document similarity and the second document similarity; 14. The program according to claim 13, which causes the computer to execute the above steps.

[0083] (Appendix 15) comparing the first document similarity with the second document similarity; determining whether the first answer sentence is correct based on a comparison result; 15. The program according to claim 14, which causes the computer to execute the above steps. [Explanation of symbols]

[0084] 1,700 Processing System 5. Computer 6 CPU 7. Main memory 8. Storage 9. Interface 10. Information input section 20...1st generation section 30...Second generation section 40. Evaluation section 50...Notification section 60...Answer provision department 701...first generation means 702...Second generation means 703...First calculation means

Claims

1. a first generation means that has undergone additional learning by performing additional learning on past performance data; a second generation means that has not performed the additional learning; a first calculation means for calculating a first document similarity between a first answer sentence, which is an answer sentence generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence, which is an answer sentence generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; second calculation means; A determination means; Equipped with The second generating means When the reason sentence included in the first answer sentence is input as additional information, a third answer sentence including a conclusion sentence and a reason sentence is generated; The second calculation means calculating a second document similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; The determination means determining whether the first answer sentence is correct based on a comparison result between the first document similarity and the second document similarity; Processing system.

2. a comparison means for comparing the first document similarity with the second document similarity; Equipped with The determination means determining whether the first answer sentence is correct or not based on the comparison result by the comparison means; The processing system of claim 1 .

3. A processing method executed by a processing system including: a first generation means that has performed additional learning on past performance data; a second generation means that has not performed the additional learning; a first calculation means; a second calculation means; and a determination means, the first calculation means calculates a first document similarity between a first answer sentence, which is an answer sentence generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence, which is an answer sentence generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; the second generation means generates a reply sentence including a conclusion sentence and a reason sentence as a third reply sentence when the reason sentence included in the first reply sentence is input as additional information; the second calculation means calculates a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; the determination means determines whether the first answer sentence is correct or not based on a comparison result between the first document similarity and the second document similarity; A processing method comprising:

4. A computer of a processing system including a first generation means that has performed additional learning on past performance data, a second generation means that has not performed the additional learning, a first calculation means, a second calculation means, and a determination means, the first calculation means calculates a first document similarity between a first answer sentence, which is an answer sentence generated by the first generation means when a question sentence is input to the first generation means and includes a conclusion sentence and a reason sentence, and a second answer sentence, which is an answer sentence generated by the second generation means when the same question sentence is input to the second generation means and includes a conclusion sentence and a reason sentence, the first document similarity being the similarity between the first answer sentence and the conclusion sentence included in the second answer sentence; the second generation means generates a reply sentence including a conclusion sentence and a reason sentence as a third reply sentence when the reason sentence included in the first reply sentence is input as additional information; the second calculation means calculates a second document similarity which is a similarity between the first answer sentence and the conclusion sentence included in the third answer sentence; the determination means determines whether the first answer sentence is correct or not based on a comparison result between the first document similarity and the second document similarity; A program that executes the following.

Images