Test case generation support system and test case generation support method

Abstract

It assists in generating test cases to accurately verify the quality of trained models. [Solution] A test case generation support system 30 comprising a computing device 31 that performs the following: an open answer generation process that generates an open answer corresponding to an open question by inputting an open question requesting an answer on a predetermined topic into a trained model 47; a first closed question generation process that generates a first closed question that asks whether the open answer is correct and requests that the open answer be correct by inputting the open question and the open answer into the trained model 47; and a second closed question generation process that modifies the topic portion of the first closed question using a predetermined algorithm and generates a second closed question that asks whether the modified topic portion is correct.

Description

Technical Field

[0001] The present invention relates to a test case generation support system and a test case generation support method.

Background Art

[0002] With the development of generative AI (Artificial Intelligence) related technologies, generative AI has been widely used in various fields of business. And in order to verify the quality of such generative AI, it is preferable to create test cases (pairs of questions and answers) using a database in the corresponding business field and use those test cases.

[0003] In order to verify the quality of generative AI with high accuracy, it is necessary to create various types of test cases, but creating test cases often requires a huge cost.

[0004] Therefore, for example, Patent Document 1 discloses a technique of generating test cases by a neural network and inputting the generated test cases into a target neural network.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, as one of the characteristics of generative AI, generative AI is often learned based on data that positively represents information, so there is a high need to verify with test cases whether it can accurately answer questions that require negative answers, for example, questions that should be answered with "no". However, the technology for creating such types of test cases has not been sufficiently developed.

[0007] This invention was made in view of the above background, and aims to provide a test case generation support system and a test case generation support method that can support the generation of test cases for verifying the quality of a trained model with high accuracy. [Means for solving the problem]

[0008] One of the present inventions for solving the above problems is a test case generation support system comprising: a storage device for storing open questions, which are natural language questions that ask for answers on a predetermined topic; an open answer generation process that generates open answers, which are natural language answers corresponding to the open questions, by inputting the open questions into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data; a first closed question generation process that generates a first closed question, which is a natural language question that asks whether the open answer is correct and requests that the open answer be correct, by inputting the open questions and the generated open answers into the trained model; and a second closed question generation process that modifies the topic portion of the generated first closed question using a predetermined algorithm and generates a second closed question, which is a question that asks whether the modified topic portion is correct. [Effects of the Invention]

[0009] According to the present invention, it is possible to support the generation of test cases for verifying the quality of a trained model with high accuracy.

[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention.

[0011] Other configurations and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0012] [Figure 1] It is a diagram showing an example of the configuration of the generative AI verification support system according to Example 1. [Figure 2] It is a diagram showing an example of the hardware that the test case generation support system has and the functions that the test case generation support system includes. [Figure 3] It is a diagram for explaining the details of the test case generation unit. [Figure 4] It is a flowchart for explaining the outline of the test case generation support process for assisting in creating test cases. [Figure 5] It is a flowchart for explaining the details of the first test case generation support process. [Figure 6] It is a diagram showing an example of an open question request prompt. [Figure 7] It is a diagram showing an example of an open answer request prompt. [Figure 8] It is a diagram showing an example of an affirmative statement request prompt. [Figure 9] It is a diagram showing an example of a closed question request prompt. [Figure 10] It is a diagram showing an example of a keyword request prompt. [Figure 11] It is a diagram showing an example of a synonym request prompt. [Figure 12] It is a flowchart for explaining the details of the second test case generation support process. [Figure 13] It is a flowchart for explaining the details of the first test case generation support process according to Example 2. [Figure 14] It is a flowchart for explaining the details of the first test case generation support process according to Example 3. [Figure 15] It is a diagram showing an example of a multiple-choice closed question request prompt. [Figure 16] It is a flowchart for explaining the details of the first test case generation support process according to Example 4. [Figure 17] It is a flowchart for explaining the details of the first test case generation support process according to Example 4.

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail based on the drawings.

[0014] [Example 1] FIG. 1 is a diagram showing an example of the configuration of a generative AI verification support system 1 according to Example 1. The generative AI verification support system 1 includes a model storage system 10 that stores a learned model (hereinafter referred to as generative AI 11 (Generative Artificial Intelligence)) described later, a model verification system 20 that verifies the quality of the generative AI 11 using test cases 36 (pairs of input data and output data), and a test case generation support system 30 that supports the generation of the test cases 36.

[0015] The generative AI 11 is a learned model that outputs corresponding natural language data for the input natural language text data. The generative AI 11 is a learned model that stores specialized knowledge (hereinafter referred to as a topic) related to a predetermined business field. The generative AI 11 is, for example, a large language model (LLM: Large language Models) such as BERT or GPT-3. The structure of the generative AI 11 is not particularly limited. For example, it is a neural network having an input layer to which input data is input, one or more intermediate layers (hidden layers) that extract and output feature amounts of the input data, and an output layer that outputs output data from the feature amounts. Examples of the neural network include an RNN (Recurrent Neural Network) and a CNN (Convolution Neural Network). In addition to the neural network, a model applied with an SVM (Support Vector Machine), a Bayesian network, a regression tree, or the like is not precluded from being adopted.

[0016] The test case generation support system 30 is an information processing system that assists in generating test cases 36 used by the model verification system 20. The test case generation support system 30 transmits the generated test cases 36 to the model verification system 20. The test cases 36 include data for a question (hereinafter referred to as a closed question) that asks for the correct answer from a set of multiple options (for example, multiple sentences or yes / no), and data for the correct answer.

[0017] The model verification system 20 evaluates the quality of the generating AI 11 by identifying whether the output value (answer) obtained by inputting the closed question in test case 36 received from the test case generation support system 30 into the generating AI 11 of the model memory system 10 matches or does not match the correct answer in test case 36.

[0018] The model memory system 10, the model verification system 20, and the test case generation support system 30 are connected by a wired or wireless communication network 5, such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), or a dedicated line.

[0019] Figure 2 shows an example of the hardware and functions of the test case generation support system 30.

[0020] The test case generation support system 30 comprises a computing device 31 such as a CPU (Central Processing Unit), a storage device 32 such as RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive), an input device 33 consisting of a keyboard, mouse, touch panel, etc., an output device 34 for screen display consisting of a monitor (display), etc., and a communication device 35 consisting of a NIC (Network Interface Card), wireless communication module, USB (Universal Serial Interface) module, or serial communication module, etc. Other information processing devices (systems) in the generation AI verification support system 1 also have similar hardware.

[0021] Next, the test case generation support system 30 stores proprietary knowledge information 41. Proprietary knowledge information 41 is a set of natural language data containing information about the business that the generating AI 11 is responsible for. Proprietary knowledge information 41 includes, for example, natural language data (text) such as various business documents, papers, meeting minutes, and memos.

[0022] Furthermore, the test case generation support system 30 includes the following functional units (programs): a proprietary knowledge input unit 42, a proprietary knowledge partitioning selection unit 43, a test case generation unit 44, an LLM call unit 45, and a test case output unit 46.

[0023] The unique knowledge input unit 42 acquires data for the unique knowledge information 41.

[0024] The proprietary knowledge segmentation selection unit 43 divides the data acquired by the proprietary knowledge input unit 42 into multiple texts (hereinafter also referred to as chunks) according to predetermined rules.

[0025] The test case generation unit 44 generates data consisting of closed questions and their correct answers based on each chunk divided by the proprietary knowledge partitioning selection unit 43.

[0026] Figure 3 is a diagram illustrating the details of the test case generation unit 44.

[0027] The test case generation unit 44 includes a filter combination definition unit 51, an open question generation filter 52, an open answer generation filter 53, a first closed question generation filter 54 which includes an affirmative sentence generation filter 55 as its components, a second closed question generation filter 56, a similarity calculation filter 60, an evaluation value calculation unit 61, and a filter evaluation filter 62.

[0028] The filter combination definition unit 51 records the combinations of each filter, which will be described later, in the filter combination recording unit 63.

[0029] The open question generation filter 52 generates natural language questions (open questions) that seek answers on a given topic. In contrast to closed questions, open questions have no restrictions on the scope (format and content) of the answers, and allow for answers in any format.

[0030] The open-response generation filter 53 generates open-response answers, which are natural language answers corresponding to open-response questions, by inputting the open-response questions into the trained model 47.

[0031] The first closed question generation filter 54 generates a first closed question, which is a natural language question that asks whether the open answer is correct or incorrect and requires that the open answer be correct, by inputting the open question and the open answer generated by the open answer generation filter 53 into a predetermined trained model. In other words, the first closed question is a question that asks whether the content indicated by the open answer is correct or incorrect, and the correct answer is "yes". This trained model may be the generation AI 11 described above, or any other trained model. Furthermore, this trained model may be stored in the test case generation support system 30, or it may be stored in an external information processing system. In this embodiment, it is assumed to be a trained model 47 stored in an external information processing system. The type of this trained model 47 is the same as that of the generation AI 11 described above.

[0032] The affirmative sentence generation filter 55 in the first closed question generation filter 54 generates an affirmative sentence (affirmative descriptive sentence) that explains the content of the open answer by inputting the open question and the open answer generated by the open question generation filter 52 into the trained model 47. The first closed question generation filter 54 generates the first closed question by inputting the generated affirmative sentence into the trained model 47.

[0033] Next, the second closed question generation filter 56 modifies the topic portion (the part asking for true or false) of the first closed question generated by the first closed question generation filter 54 using a predetermined algorithm (i.e., modifies it to false content), and generates a second closed question that asks for the true or false status of the modified topic portion. In other words, the second closed question is a question that asks for a "yes" or "no" answer regarding the true or false status of the question's content, and the correct answer is "no".

[0034] Specifically, the second closed question generation filter 56 includes a keyword extraction filter 57, a slang term generation filter 58, and a string replacement filter 59 as its components.

[0035] The keyword extraction filter 57 extracts keywords related to the above topic from the first closed question generated by the first closed question generation filter 54.

[0036] The antonym generation filter 58 retrieves other keywords (antonyms, etc.) with meanings different from the keywords extracted by the keyword extraction filter 57.

[0037] The string replacement filter 59 modifies the topic portion of the first closed question generated by the first closed question generation filter 54 by replacing the keywords in the first closed question generated by the first closed question generation filter 54 with the other keywords (antonyms, etc.) mentioned above. Then, the second closed question generation filter 56 generates a second closed question that asks whether the modified topic portion is correct or incorrect.

[0038] The similarity calculation filter 60 will be described in Example 3. Note that the similarity calculation filter 60 is optional.

[0039] The evaluation value calculation unit 61 and the filter evaluation filter 62 will be described in Example 4. Note that the evaluation value calculation unit 61 and the filter evaluation filter 62 are optional.

[0040] Next, as shown in Figure 2, the LLM calling unit 45 inputs various data received from the test case generation unit 44 into a predetermined trained model to obtain the data necessary for generating the closed question and its answer, and transmits the obtained data to the test case generation unit 44. In this embodiment, this trained model is a trained model 47 stored in an external information processing system, similar to the first closed question generation filter 54.

[0041] The test case output unit 46 outputs test case 36 based on the data generated by the test case generation unit 44. Specifically, the test case output unit 46 generates a test case that includes a second closing question and an answer to the second closing question stating that the part of the topic modified by the string replacement filter 59 is incorrect. The generated test case 36 is used by the model verification system 20.

[0042] The functions of the test case generation support system 30 described above are realized by the hardware of the test case generation support system 30, or by the arithmetic unit 31 of the test case generation support system 30 reading and executing each program stored in the storage device 32. These programs are stored in storage devices such as secondary storage devices, non-volatile semiconductor memory, hard disk drives, SSDs, or recording media readable by the test case generation support system 30, such as IC cards, SD cards, or DVDs. The test case generation support system 30 may be implemented in whole or in part using virtual information processing resources provided using virtualization technology, process space isolation technology, etc., such as a virtual server provided by a cloud system. Furthermore, all or part of the functions provided by these devices may be implemented by services provided by the cloud system via APIs (Application Programming Interfaces), etc. Next, we will explain the processes performed in the generation AI verification support system 1.

[0043] <Test case generation support process> Figure 4 is a flowchart illustrating the overview of the test case generation support process that assists in the generation of test cases. The generation AI creation support process is initiated, for example, when the test case generation support system 30 receives predetermined input from the user.

[0044] First, the proprietary knowledge input unit 42 of the test case generation support system 30 acquires proprietary knowledge information of the business in the relevant field from the proprietary knowledge information 41 (s1).

[0045] The proprietary knowledge segmentation selection unit 43 of the test case generation support system 30 segments each text contained in the proprietary knowledge information acquired in s1 based on predetermined rules (s3). Hereinafter, each segmented part will be referred to as a chunk.

[0046] For example, the unique knowledge segmentation selection unit 43 divides each sentence of the text into parts of a predetermined string length, and ensures that consecutive parts overlap by a predetermined number of characters (for example, the last string portion of one chunk overlaps with the string portion of the next chunk). By ensuring sufficient overlap, it is possible to prevent the eigenvalue expression from becoming fragmented and resulting in meaningless chunks. Alternatively, the unique knowledge segmentation selection unit 43 may divide the text into chunks by extracting meaningful parts from the document by analyzing its structure (chapters or sections, etc.) (morphological analysis, etc.) rather than by predetermined string length.

[0047] Subsequently, the test case generation unit 44 of the test case generation support system 30 executes a first test case generation support process s3, which generates a second closed question based on each chunk created in s3. Details of the first test case generation support process s3 will be described later.

[0048] Furthermore, the test case generation unit 44 of the test case generation support system 30 executes a second test case generation support process s4 that generates other closed questions based on each chunk created in s3. Details of the second test case generation support process s4 will be described later.

[0049] The test case output unit 46 of the test case generation support system 30 outputs test case data (s5) based on the data created in the first test case generation support process s3 and the second test case generation support process s4.

[0050] Specifically, the test case output unit 46 generates a test case that includes the second closing question generated in the first test case generation support process s3 and the correct answer corresponding to that second closing question (for example, "no"). The test case output unit 46 also generates a test case that includes the closing question generated in the second test case generation support process s4 and the correct answer corresponding to that closing question (for example, "yes").

[0051] Subsequently, the test case output unit 46 sends the generated test cases to the model verification system 20. This completes the test case generation support process.

[0052] Next, we will explain the details of each of the above processes.

[0053] <First Test Case Generation Support Process> Figure 5 is a flowchart illustrating the details of the first test case generation support process s3. The test case generation unit 44 first generates and stores open questions related to proprietary knowledge based on the chunks generated in s2 (hereinafter referred to as target chunks) (s31).

[0054] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as the open question request prompt) that requests the generation of an open question based on each target chunk, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the model to output an open question.

[0055] An example of an open-ended question output by the trained model 47 is: "This document details quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. What are the AI ​​product-specific quality characteristics and their relevance to data and models?"

[0056] (Open Question Request Prompt) Figure 6 shows an example of an open question request prompt. This open question request prompt 600 includes a statement 601 that requests the generation of an open question asking about the contents of a target chunk, including a specified keyword, and an example of input / output data. As an example of input / output data, three target chunks 602, 604, and 606 and three open questions 603, 605, and 607 corresponding to these target chunks are set.

[0057] Then, as shown in Figure 5, the test case generation unit 44 generates answers (open answers) to each open question generated in s31 (s32).

[0058] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as the open answer request prompt) that requests the generation of an open answer based on each open question, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the model to output an open answer.

[0059] The open-ended response output by the trained model 47 is, for example, a sentence answering an open-ended question, such as, "Data and models are internal properties."

[0060] (Open-ended response prompt) Figure 7 shows an example of an open-response request prompt. This open-response request prompt 700 includes a statement 701 that requests the generation of an open-response to an open-ended question using the target chunks, and an example of input / output data. As an example of input / output data, two target chunks 702 and 705, two corresponding open-ended questions 703 and 706, and two corresponding open-ended answers 704 and 707 are set.

[0061] As described above, by generating open questions and open answers in chunk units, clear questions and answers can be generated. If a summary of the content of proprietary knowledge information 41 is used, the questions and answers may become unclear.

[0062] Then, as shown in Figure 5, the test case generation unit 44 generates an affirmative explanatory sentence for each open question by describing the matter that the open question is asking based on the open answer generated in s32 (s33).

[0063] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as an affirmative statement request prompt) that requests the trained model 47 to convert the open question into a normal sentence format by rewriting the open question, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, causing the model to output an explanatory text for an affirmative statement.

[0064] An example of an affirmative statement explanation produced by the trained model 47 is: "This document provides a detailed explanation of quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. Quality characteristics specific to AI products refer to data and models as internal characteristics."

[0065] (Affirmative statement prompt) Figure 8 shows an example of an affirmative statement request prompt. This affirmative statement request prompt 800 includes a statement 801 that requests the conversion of an open question into an affirmative statement based on the open answer, and an example of input / output data. The example of input / output data includes two open questions 802 and 806, two corresponding target chunks 803 and 807, two corresponding open answers 804 and 808, and two corresponding affirmative statement explanations 805 and 809. The affirmative statement request prompt 800 may also include, for example, a statement 810 that instructs the output of an affirmative statement, a statement 811 that requests the correspondence between the open question and the open answer, and a statement 812 that requests the extraction of necessary information from the target chunk.

[0066] Then, as shown in Figure 5, the test case generation unit 44 generates a first closing question related to the explanatory text generated in s33 (s34).

[0067] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a closed question request prompt) that requests the description generated in s33 to be converted into a question asking whether or not it is correct, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the closed question to be output.

[0068] A closed question output by the trained model 47 might be: "This document details quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization / education, technology updates / innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. AI product-specific quality characteristics refer to data and models as internal characteristics. Please answer 'yes' or 'no'." This closed question is designed to elicit a "yes" or "no" answer to the open question.

[0069] (Close question prompt) Figure 9 shows an example of a closing question request prompt. This closing question request prompt 900 includes a statement 901 that requests the generation of a first closing question by rewriting the explanatory text of an affirmative statement, and an example of input / output data. As an example of input / output data, two explanatory texts 902 and 904 of affirmative statements and two corresponding first closing questions 903 and 905 are set. The closing question request prompt 900 may also include, for example, a statement 906 that requests the generation of a first closing question that can be answered with "yes" or "no," and a statement 907 that requests the first closing question to be expressed in such a way that "yes" is the correct answer.

[0070] Next, as shown in Figure 5, the test case generation unit 44 generates all the first closing questions and then extracts keywords from each of the first closing questions (s35).

[0071] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a keyword request prompt) that requests the extraction of important words or phrases from the first closed question generated in s34, and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the keyword.

[0072] The keywords output by the trained model 47 are, for example, "data and model".

[0073] (Keyword prompt) Figure 10 shows an example of a keyword request prompt. This keyword request prompt 1000 includes a statement 1001 requesting that the most suitable keyword for the second close question be extracted from the first close question, and an example of input / output data. As an example of input / output data, the first close question 1002 and the corresponding keyword 1003 are set. The keyword request prompt 1000 may also include, for example, a statement 1004 indicating that the keyword is used in a question that requires filling in a blank (in this case, a fill-in-the-blank format), and a statement 1005 indicating that the keyword is text present in the close question.

[0074] Next, as shown in Figure 5, the test case generation unit 44 extracts other keywords (words or phrases, etc.) with different content from the keyword extracted in s35 (s36).

[0075] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a synonym request prompt) that requests the keyword extracted in s35 to be converted into a word or phrase with a different meaning (including negation and opposite meanings) (hereinafter referred to as a synonym in this specification), and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the synonym.

[0076] The antonyms output by the trained model 47 include, for example, "data" and "model," as well as "emotions and experiences," which are words with different meanings.

[0077] (Prompt for requesting alternative meanings) Figure 11 shows an example of a antonym request prompt. This antonym request prompt 1100 includes a request 1101 to convert the keyword of the first closing question into an antonym, and an example of input / output data. As an example of input / output data, the keyword 1102 and antonym 1103 of the first closing question are set. The antonym request prompt 1100 may also include, for example, a statement 1104 requesting conversion to an antonym of the same level of abstraction (depth or breadth of vocabulary), a statement 1105 requesting that prefixes or suffixes expressing negation such as "not," "un," or "none" not be used, and a statement 1106 requesting that simple character or language conversion not be performed.

[0078] Next, as shown in Figure 5, the test case generation unit 44 generates a new closing question (i.e., a second closing question) for each of the first closing questions generated in s34, by replacing the keywords in the first closing question with the antonyms generated in s36 (s37).

[0079] This generates a closed question whose answer is "No". This concludes the first test case generation support process s3.

[0080] The second closing question generated is, for example, "It details the quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. The AI ​​product-specific quality characteristics mention emotions and experiences as internal characteristics. Answer with 'yes' or 'no'." Compared to the first closing question in s34 mentioned above, "data and models" has been changed to "emotions and experiences." The correct answer is "no."

[0081] <Second Test Case Generation Support Process> Figure 12 is a flowchart illustrating the details of the second test case generation support process s4. The test case generation unit 44 first generates open questions about specific knowledge based on the target chunk, similar to s31 (s41).

[0082] Then, the test case generation unit 44 generates open answers for each open question generated in s41, similar to s32 (s42).

[0083] The test case generation unit 44 generates a first closed question related to each open question generated in s42, similar to s33 (s43).

[0084] This generates a closed question that answers "yes". This concludes the second test case generation support process s4.

[0085] As described above, the test case generation support system 30 of this embodiment generates an open answer corresponding to an open question by inputting an open question that asks for an answer on a predetermined topic into a trained model 47, generates a first closed question that asks whether the open answer is correct and requests that the open answer be correct as the correct answer by inputting the open question and the open answer into the trained model 47, modifies the topic portion of the generated first closed question and generates a second closed question that asks whether the modified topic portion is correct or incorrect.

[0086] In other words, the test case generation support system 30 of this embodiment, based on an open question and an open answer, assumes a first closed question which is a question asking whether the open answer is correct or incorrect (for example, a question where "yes" is the correct answer), changes the topic part of this first closed question to a different content, and generates a second closed question which asks whether the changed topic part is correct or incorrect.

[0087] This ensures the generation of closed questions where the topic of the question is incorrect (for example, questions where "no" is the correct answer). For instance, directly attempting to generate a question where the correct answer is "no" may result in a nonsensical question due to the characteristics of the trained model (e.g., "Is it correct that XX is not OO?"). By using this second closed question to generate test cases to evaluate the quality of the generated AI11, it becomes possible to perform highly accurate validation of the trained model, which tends to learn from positive information.

[0088] Thus, the test case generation support system 30 of this embodiment can support the generation of test cases for verifying the quality of a trained model with high accuracy.

[0089] Furthermore, the test case generation support system 30 of this embodiment, in relation to generating the second closing question, extracts topic keywords from the first closing question, obtains other keywords with different meanings from the extracted keywords (such as antonyms), and replaces the keywords in the first closing question with the antonyms, thereby changing the topic portion of the first closing question and generating a second closing question that asks whether the changed topic portion is correct or incorrect.

[0090] In this way, by replacing the topic portion of the first closing question with an antonym to generate the second closing question, it is possible to reliably generate a closing question in which the topic portion (the part asking for true or false) is incorrect. In other words, by converting important words into words with different meanings, it is possible to reliably generate a question in which the correct answer is, for example, "no".

[0091] Furthermore, in this embodiment, the test case generation support system 30 generates an affirmative sentence explaining the content of the open answer by inputting the open question and open answer into the trained model 47, and then generates the first closed question by inputting the generated affirmative sentence into the trained model 47.

[0092] In this way, by generating the first closed question based on an affirmative statement that explains the content of the open answer, it is possible to reliably generate a closed question that requires the open answer to be correct as the correct answer.

[0093] Furthermore, the test case generation support system 30 of this embodiment generates a test case that includes a second closing question and an answer to the second closing question (for example, an answer of "no") indicating that the part in which the keyword has been changed to a synonym is incorrect.

[0094] This allows us to generate test cases for the trained model 47 where negative answers such as "no" are considered correct.

[0095] [Example 2] In Example 2, when generating the second closing question, the antonyms in the second closing question are limited to those that have a predetermined relationship with the corresponding keyword in the first closing question.

[0096] The configuration of the test case generation support system 30 in Example 2 is the same as in Example 1. In addition, the test case generation support system 30 has the following functional components in addition to those in Example 1.

[0097] In other words, the test case generation unit 44 includes a similarity calculation filter 60. The similarity calculation filter 60 calculates the similarity between the topic portion of the first closed question generated by the first closed question generation filter 54 and the modified topic portion (e.g., antonyms), determines whether the calculated similarity is within a predetermined range, and if it determines that the similarity is within the predetermined range, it generates a second closed question asking whether the modified topic portion is correct or incorrect.

[0098] The test case generation support process in Example 2 is the same as in Example 1. However, the first test case generation support process is as described below.

[0099] <First Test Case Generation Support Process> Figure 13 is a flowchart illustrating the details of the first test case generation support process s6 in Example 2. The processes s31 to s36 shown in the figure are the same as those in Example 1.

[0100] After processing in s36, the test case generation unit 44 calculates the similarity between the keyword generated in s35 and the corresponding antonym generated in s36 (s61). If the calculated similarity falls outside a predetermined range, the test case generation unit 44 repeats the processing in s36 to extract different antonyms.

[0101] For example, the test case generation unit 44 calculates vector values ​​for each keyword and antonym based on the proprietary knowledge information 41 (embedding), and determines the similarity by calculating the cosine similarity between the calculated vector values. Alternatively, the test case generation unit 44 may calculate the similarity based on the degree of matching of each character of the keyword and antonym, or based on a predetermined comparison database.

[0102] On the other hand, if the calculated similarity is within a predetermined range, the test case generation unit 44 generates a second closed question (s37), similar to Example 1. This completes the first test case generation support process s6.

[0103] Thus, in this embodiment, the test case generation support system 30 generates a second closing question that asks whether the modified topic is correct or incorrect, when the similarity between the topic portion of the first closing question and the modified topic portion is within a predetermined range.

[0104] This allows for the generation of questions of varying difficulty levels, adjusting the degree to which the modified topic deviates from the correct answer, and enabling the adjustment of the level of quality verification of the generated AI11 using test cases.

[0105] [Example 3] In Example 3, the second closing question is not a binary question with a "yes" or "no" answer, but a multiple-choice question from which the correct answer or no correct answer is selected.

[0106] The configuration of the test case generation support system 30 in Example 3 is the same as in Example 1. Furthermore, each functional unit in the test case generation support system 30 is the same, except for the second closed question generation filter.

[0107] In other words, the second closed question generation filter 56 generates multiple modified parts (such as antonyms) by changing the topic part of the first closed question generated by the first closed question generation filter 54, and uses the generated multiple modified parts and the topic part as question elements (choices), and generates a second closed question that asks whether each question element is correct or incorrect.

[0108] The test case generation support process in Example 3 is the same as in Example 1. However, the first test case generation support process is as described below.

[0109] <First Test Case Generation Support Process> Figure 14 is a flowchart illustrating the details of the first test case generation support process s7 in Example 3. The processes s31 to s33 shown in the figure are the same as those in Example 1.

[0110] After processing in s33, the test case generation unit 44 extracts one keyword (hereinafter referred to as the correct keyword) from the explanatory text of the affirmative statement generated in s33 (s71).

[0111] In this embodiment, the test case generation unit 44 generates a prompt requesting the extraction of keywords from the descriptive text of the affirmative statement generated in s33, and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the keywords.

[0112] The test case generation unit 44 obtains several other keywords that have different meanings from the correct keyword extracted in s71 (s72).

[0113] In this embodiment, the test case generation unit 44 generates a prompt requesting the output of multiple other keywords (keywords in options other than the correct answer option) that have different meanings from the keyword extracted in s71, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the multiple other keywords to be output. The extraction of other keywords may be performed, for example, based on the aforementioned antonym request prompt.

[0114] The test case generation unit 44 generates the lead portion of the second closed question (s73) by outputting a new explanatory text in which the correct keyword portion extracted in s71 from the explanatory text of the affirmative statement generated in s33 is masked with a predetermined string.

[0115] The test case generation unit 44 outputs a question (multiple-choice question) that has the lead portion generated in s73 and uses the multiple keywords obtained in s72 as answer options (s74).

[0116] In this embodiment, the test case generation unit 44 generates a prompt (multiple-choice closed question request prompt) that requests the output of a multiple-choice question having a lead portion that uses the correct answer keyword extracted in s71 and each keyword acquired in s72 as options, and requests that the user select the correct option from among these options, or select "no correct answer" if none of the options are correct. The generated prompt is input to the trained model 47 via the LLM call unit 45, causing the question to be output.

[0117] (Multiple-choice closed question prompt) Figure 15 shows an example of a multiple-choice closed question request prompt. This multiple-choice closed question request prompt 1500 includes a statement 1501 that requests the generation of a multiple-choice question sentence containing a masked lead portion and multiple answer choices, and an example of input / output data. As an example of input / output data, the lead portion 1502, options other than the correct keyword 1503, the correct keyword 1504, and the entire second closed question 1505 are set. The closed question request prompt 1500 may also include a statement 1506 that instructs the output of a multiple-choice question, a statement 1507 that requests the generation of options related to the correct keyword and other keywords within a predetermined number, etc.

[0118] Subsequently, as shown in Figure 14, the test case generation unit 44 obtains the correct keyword for the second closed question generated in s74, similar to s35 in Example 1.

[0119] The test case generation unit 44 acquires keywords (such as antonyms) that are different from the correct keywords generated in s35, similar to s36 in Example 1.

[0120] Then, the test case generation unit 44 generates a second closed question by replacing the correct keyword obtained in s37 with another keyword obtained in s36 (s75). This generates a second closed question with multiple choices, in which the option related to the original correct keyword is not considered the correct answer. This completes the first test case generation support process s7.

[0121] Subsequently, the test case output unit 46 generates a test case that includes the second closed question generated in the first test case generation support process s3 and the correct answer corresponding to that second closed question (for example, the correct answer options, or "no correct answer").

[0122] As described above, the test case generation support system 30 of this embodiment generates multiple modified parts (such as antonyms) by changing the topic part of the first closing question, and generates a second closing question that asks whether each of the generated antonyms and the topic part is correct or incorrect (which of the options is correct, or none of the options are correct) with respect to the generation of the second closing question.

[0123] In this way, by generating a second closed question with multiple-choice options, the accuracy of the knowledge generated by the AI11 using test cases can be verified with high precision.

[0124] [Example 4] In Example 4, the second closing question, which is appropriate as a test case, is identified by evaluating each question and answer generated by the test case generation unit 44.

[0125] The configuration of the test case generation support system 30 in Example 4 is the same as in Example 1. In addition to the functions of Example 1, the test case generation support system 30 also includes the following functions.

[0126] In other words, the test case generation support system 30 includes an evaluation value calculation unit 61 and a filter evaluation filter 62.

[0127] The evaluation value calculation unit 61 calculates evaluation values ​​for the open-ended questions, the first closed-ended questions, and the second closed-ended questions, respectively.

[0128] Similarly, the evaluation value calculation unit 61 calculates evaluation values ​​for the generated affirmative sentences and other elements such as keywords.

[0129] The filter evaluation filter 62 determines whether or not to generate a test case based on the evaluation values ​​calculated by the evaluation value calculation unit 61, and if it determines that a test case should be generated, it generates a test case that includes the second closed question.

[0130] The test case generation support process in Example 4 is the same as in Example 1. However, the first test case generation support process is as described below.

[0131] <First Test Case Generation Support Process> Figures 16 and 17 are flowcharts illustrating the details of the first test case generation support process s8 in Example 4 (divided into two figures due to space limitations).

[0132] First, as shown in Figure 16, the test case generation unit 44 performs the process in s31 in the same manner as in Example 1, and then the evaluation value calculation unit 61 evaluates the open question request prompt and the outputted open question (s81).

[0133] In this embodiment, the evaluation value calculation unit 61 generates an open question request prompt and a prompt requesting an evaluation of the outputted open question, and inputs the generated prompts to the trained model 47 to output an evaluation value for the open question.

[0134] Next, the test case generation unit 44 performs the processing in s32 in the same manner as in Example 1, and then evaluates the open response request prompt and the outputted open response (s82).

[0135] In this embodiment, the evaluation value calculation unit 61 generates an open response request prompt and a prompt requesting an evaluation of the output open response, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0136] Next, after the test case generation unit 44 performs the process in s33 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the affirmative statement request prompt and the outputted affirmative statement description (s83).

[0137] In this embodiment, the evaluation value calculation unit 61 generates an affirmative statement request prompt and a prompt requesting an evaluation of the explanatory text of the output affirmative statement, and outputs an evaluation value by inputting the generated prompts into the trained model 47.

[0138] Next, after the test case generation unit 44 performs the processing in s34 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the close question request prompt and the outputted first close question (s84).

[0139] In this embodiment, the evaluation value calculation unit 61 generates a closed question request prompt and a prompt requesting an evaluation of the outputted first closed question, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0140] Next, as shown in Figure 17, after the test case generation unit 44 performs the process in s35 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the keyword request prompt and the outputted keywords (s85).

[0141] In this embodiment, the evaluation value calculation unit 61 generates a keyword request prompt and a prompt requesting an evaluation of the output keyword, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0142] Next, after the test case generation unit 44 performs the process in s36 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the antinomial word request prompt and the output antinomial words (s86).

[0143] In this embodiment, the evaluation value calculation unit 61 generates a similar word request prompt and a prompt requesting an evaluation of the outputted similar words, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0144] Next, after the test case generation unit 44 performs the process in s37 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the generated second closed question (s87).

[0145] In this embodiment, the evaluation value calculation unit 61 generates a prompt requesting an evaluation of the second closed question, and outputs an evaluation value by inputting the generated prompt to the trained model 47.

[0146] Furthermore, the evaluation value calculation unit 61 may attach information on evaluation criteria to the prompt when calculating each of the above evaluation values. For example, the evaluation value calculation unit 61 may attach an evaluation request based on whether or not there is logical consistency between the prompt and the output text, or an evaluation request based on whether or not there is ambiguous expression in the wording of the prompt or the output text.

[0147] Next, the filter evaluation filter 62 calculates an overall evaluation value from the evaluation values ​​output by the above process (s88). For example, the filter evaluation filter 62 calculates the overall evaluation value by multiplying each evaluation value by a predetermined weighting value and summing up the weighted multipliers. Alternatively, for example, the filter evaluation filter 62 may use the lowest value of each evaluation value as the overall evaluation value.

[0148] Subsequently, the filter evaluation filter 62 determines whether the overall evaluation value calculated in s88 exceeds a predetermined threshold (s89). If the filter evaluation filter 62 determines that the overall evaluation value exceeds the predetermined threshold, it generates the corresponding test case in the process of s5 described above.

[0149] As described above, the test case generation support system 30 of this embodiment calculates an evaluation value for at least one of the open answer, the first closed question, and the second closed question, and generates a test case 36 if the calculated evaluation value exceeds a predetermined threshold.

[0150] This allows for accurate determination of whether an appropriate second closed question has been generated as a test case, and enables the generation AI 11 to use only appropriate test cases. In particular, when using the trained model 47 for outputting each data, there is a risk that low-quality data such as responses may be output due to its characteristics, making it highly necessary to evaluate based on evaluation values.

[0151] The present invention is not limited to the embodiments described above, and can be implemented using any components without departing from its spirit. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0152] For example, some of the hardware provided in each device of each embodiment may be provided in other devices.

[0153] Furthermore, each program of each device may be provided in other devices, a program may consist of multiple programs, or multiple programs may be integrated into a single program.

[0154] Furthermore, in this embodiment, an affirmative explanatory text is generated based on the open question and open answer, and a first closed question is generated based on the generated explanatory text. However, the first closed question may also be generated directly based on the open question and open answer using other algorithms such as predetermined string manipulation.

[0155] Furthermore, in this embodiment, when generating the second closed question, if a keyword is to be replaced with a synonym, the trained model 47 may be made to perform the replacement.

[0156] Furthermore, in this embodiment, when generating the second closing question from the first closing question, the question was modified by replacing keywords with antonyms, but modifications using other algorithms may also be performed. For example, modifications such as swapping the subject and predicate, or reversing the cause-and-effect relationships in a sentence may be performed. [Explanation of Symbols]

[0157] 30 Test case generation support system, 44 Test case generation unit, 47 Trained model

Claims

1. A memory device that stores open questions, which are natural language questions that elicit answers on a given topic, and An open answer generation process generates an open answer, which is a natural language response to the open question, by inputting the aforementioned open question into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data. A first closed question generation process generates a first closed question, which is a natural language question that asks whether the open question is correct or incorrect and requests that the open question be correct as the correct answer, by inputting the open question and the generated open answer into the trained model. A computing device that performs a second closed question generation process, which involves modifying the topic portion of the first closed question generated using a predetermined algorithm, and generating a second closed question that asks whether the modified topic portion is correct or incorrect. A test case generation support system equipped with the following features.

2. The aforementioned computing device is In the second closed question generation process, A keyword extraction process is performed to extract keywords of the topic from the first closed question generated, The process involves obtaining other keywords with different meanings from the extracted keywords, and A string replacement process is performed to change the topic portion of the generated first closed question by replacing the keyword in the generated first closed question with the other keyword. A second closed question is generated to inquire about the correctness of the modified topic portion. The test case generation support system according to claim 1.

3. The aforementioned computing device is In the first closed question generation process, the open question and the generated open answer are input to the trained model to perform an affirmative sentence generation process that generates an affirmative sentence explaining the content of the open answer, and the first closed question is generated by inputting the generated affirmative sentence to the trained model. The test case generation support system according to claim 1.

4. The aforementioned computing device is In the second closed question generation process, the similarity between the topic portion of the first closed question generated and the modified topic portion is calculated, it is determined whether the calculated similarity falls within a predetermined range, and if it is determined that the similarity falls within a predetermined range, a second closed question is generated that asks whether the modified topic portion is correct or incorrect. The test case generation support system according to claim 1.

5. Multiple-choice format The aforementioned computing device is In the second closed question generation process, Multiple modified parts are generated by changing the topic portion of the first closed question generated above, and the multiple generated modified parts and the topic portion are each used as question elements, and a second closed question is generated that asks whether each of the question elements is true or false. The test case generation support system according to claim 1.

6. The aforementioned computing device is The system executes a test case output process that generates a test case including the second closing question generated and an answer to the second closing question stating that the modified topic is incorrect. The test case generation support system according to claim 1.

7. The aforementioned computing device is An evaluation value calculation process is performed to calculate an evaluation value for at least one of the open-ended questions, the first closed question, and the second closed question. If the calculated evaluation value exceeds a predetermined threshold, the test case is generated. The test case generation support system according to claim 6.

8. A method for supporting the generation of test cases by an information processing device comprising a memory device for storing open questions, which are natural language questions that require answers on a predetermined topic, and an arithmetic unit, wherein the arithmetic unit is An open answer generation process generates an open answer, which is a natural language response to the open question, by inputting the aforementioned open question into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data. A first closed question generation process generates a first closed question, which is a natural language question that asks whether the open question is correct or incorrect and requests that the open question be correct as the correct answer, by inputting the open question and the generated open answer into the trained model. The process involves modifying the topic portion of the first closed question generated using a predetermined algorithm, and then generating a second closed question, which is a question asking whether the modified topic portion is correct or incorrect. A method for supporting the generation of test cases.

Images