Test case generation support system and test case generation support method

Abstract

A test case generation support system 30 comprising an arithmetic unit 31 that executes: open-ended response generation processing for generating an open-ended response corresponding to an open-ended question for a predetermined topic which requires a response by inputting the open-ended question to a trained model 47; first closed-ended question generation processing for generating a first closed-ended question by inputting the open-ended question and the open-ended response to the trained model 47, the first closed-ended question asking whether the open-ended response is correct or incorrect and requiring that the open-ended response is correct as a correct answer; and second closed-ended question generation processing for changing a topic portion of the first closed-ended question by a predetermined algorithm and generating a second closed-ended question asking whether the changed topic portion is correct or incorrect.

Description

Test Case Generation Support System and Test Case Generation Support Method

[0001] The present invention relates to a test case generation support system and a test case generation support method.

[0002] === Incorporation by Reference === This application claims the priority of Japanese Patent Application No. 2024-203116 filed on November 21, 2024, and incorporates its content by reference into this application. With the development of generative AI (Artificial Intelligence) related technologies, generative AI has been widely used in various fields of business. And in order to verify the quality of such generative AI, it is preferable to create test cases (pairs of questions and answers) using a database in the corresponding business field and use those test cases.

[0003] In order to verify the quality of generative AI with high precision, it is necessary to create various types of test cases, but creating test cases often requires a huge cost.

[0004] Therefore, for example, Patent Document 1 discloses a technique of generating test cases by a neural network and inputting the generated test cases into the target neural network.

[0005] Japanese Patent Application Laid-Open No. 2023-109726

[0006] By the way, as one of the characteristics of generative AI, generative AI is often learned based on data that positively represents information. Therefore, there is a high need to verify with test cases whether it can accurately answer questions that require negative answers, for example, questions that should be answered with "no". However, the technology for creating such types of test cases has not been sufficiently developed.

[0007] The present invention has been made in view of such a background, and an object thereof is to provide a test case generation support system and a test case generation support method capable of supporting the generation of test cases for verifying the quality of a learned model with high precision.

[0008] One of the present inventions for solving the above problems is a test case generation support system comprising: a storage device for storing open questions, which are natural language questions that ask for answers on a predetermined topic; an open answer generation process that generates open answers, which are natural language answers corresponding to the open questions, by inputting the open questions into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data; a first closed question generation process that generates a first closed question, which is a natural language question that asks whether the open answer is correct and requests that the open answer be correct, by inputting the open questions and the generated open answers into the trained model; and a second closed question generation process that modifies the topic portion of the generated first closed question using a predetermined algorithm and generates a second closed question, which is a question that asks whether the modified topic portion is correct.

[0009] According to the present invention, it is possible to support the generation of test cases for verifying the quality of a trained model with high accuracy.

[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention.

[0011] Other configurations and effects will be clarified by the following description of the embodiments.

[0012] This figure shows an example of the configuration of the AI-generated verification support system according to Example 1. This figure shows an example of the hardware and functions of the test case generation support system. This figure explains the details of the test case generation unit. This is a flowchart explaining the outline of the test case generation support process that assists in the creation of test cases. This is a flowchart explaining the details of the first test case generation support process. This figure shows an example of an open question request prompt. This figure shows an example of an open answer request prompt. This figure shows an example of an affirmative sentence request prompt. This figure shows an example of a closed question request prompt. This figure shows an example of a keyword request prompt. This figure shows an example of an anti-word request prompt. This is a flowchart explaining the details of the second test case generation support process. This is a flowchart explaining the details of the first test case generation support process according to Example 2. This is a flowchart explaining the details of the first test case generation support process according to Example 3. This figure shows an example of a multi-choice closed question request prompt. This is a flowchart explaining the details of the first test case generation support process according to Example 4. This is a flowchart explaining the details of the first test case generation support process according to Example 4.

[0013] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0014] [Example 1] Figure 1 shows an example of the configuration of the Generative AI Verification Support System 1 according to Example 1. The Generative AI Verification Support System 1 includes a model storage system 10 that stores a trained model (hereinafter referred to as Generative AI 11 (Generative Artificial Intelligence)) described later, a model verification system 20 that verifies the quality of Generative AI 11 using test cases 36 (pairs of input data and output data), and a test case generation support system 30 that assists in the generation of test cases 36.

[0015] The Generative AI 11 is a trained model that outputs corresponding natural language data in response to input natural language text data. The Generative AI 11 is a trained model that memorizes specialized knowledge (hereinafter referred to as "topics") related to a predetermined business field. The Generative AI 11 is, for example, a Large Language Model (LLM) such as BERT or GPT-3. The structure of the Generative AI 11 is not particularly limited, but for example, it is a neural network having an input layer into which input data is input, one or more intermediate layers (hidden layers) that extract and output features from the input data, and an output layer that outputs output data from the features. Examples of neural networks include RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network). Furthermore, there is no prerequisite for adopting models that apply SVM (Support Vector Machine), Bayesian networks, or regression trees, etc., in addition to neural networks.

[0016] The test case generation support system 30 is an information processing system that assists in generating test cases 36 used by the model verification system 20. The test case generation support system 30 transmits the generated test cases 36 to the model verification system 20. The test cases 36 include data for a question (hereinafter referred to as a closed question) that asks for the correct answer from a set of multiple options (for example, multiple sentences or yes / no), and data for the correct answer.

[0017] The model verification system 20 evaluates the quality of the generating AI 11 by identifying whether the output value (answer) obtained by inputting the closed question in test case 36 received from the test case generation support system 30 into the generating AI 11 of the model memory system 10 matches or does not match the correct answer in test case 36.

[0018] The model memory system 10, the model verification system 20, and the test case generation support system 30 are connected by a wired or wireless communication network 5, such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), or a dedicated line.

[0019] Figure 2 shows an example of the hardware and functions of the test case generation support system 30.

[0020] The test case generation support system 30 comprises a computing device 31 such as a CPU (Central Processing Unit), a storage device 32 such as RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive), an input device 33 consisting of a keyboard, mouse, touch panel, etc., an output device 34 for screen display consisting of a monitor (display), etc., and a communication device 35 consisting of a NIC (Network Interface Card), wireless communication module, USB (Universal Serial Interface) module, or serial communication module, etc. Other information processing devices (systems) in the generation AI verification support system 1 also have similar hardware.

[0021] Next, the test case generation support system 30 stores proprietary knowledge information 41. Proprietary knowledge information 41 is a set of natural language data containing information about the business that the generating AI 11 is responsible for. Proprietary knowledge information 41 includes, for example, natural language data (text) such as various business documents, papers, meeting minutes, and memos.

[0022] Furthermore, the test case generation support system 30 includes the following functional units (programs): a proprietary knowledge input unit 42, a proprietary knowledge partitioning selection unit 43, a test case generation unit 44, an LLM call unit 45, and a test case output unit 46.

[0023] The unique knowledge input unit 42 acquires data from the unique knowledge information 41.

[0024] The proprietary knowledge division selection unit 43 divides the data acquired by the proprietary knowledge input unit 42 into multiple texts (hereinafter also referred to as chunks) according to predetermined rules.

[0025] The test case generation unit 44 generates data consisting of closed questions and their correct answers based on each chunk divided by the proprietary knowledge partitioning selection unit 43.

[0026] Figure 3 is a diagram illustrating the details of the test case generation unit 44.

[0027] The test case generation unit 44 includes a filter combination definition unit 51, an open question generation filter 52, an open answer generation filter 53, a first closed question generation filter 54 which includes an affirmative sentence generation filter 55 as a component, a second closed question generation filter 56, a similarity calculation filter 60, an evaluation value calculation unit 61, and a filter evaluation filter 62.

[0028] The filter combination definition unit 51 records the combinations of each filter, which will be described later, in the filter combination recording unit 63.

[0029] The open question generation filter 52 generates natural language questions (open questions) that seek answers on a predetermined topic. In contrast to closed questions, open questions have no restrictions on the scope (format and content) of the answers, and allow for answers in any format.

[0030] The open-response generation filter 53 generates open-response answers, which are natural language answers corresponding to open-response questions, by inputting the open-response questions into the trained model 47.

[0031] The first closed question generation filter 54 generates a first closed question, which is a natural language question that asks whether the open answer is correct or incorrect and requires that the open answer be correct, by inputting the open question and the open answer generated by the open answer generation filter 53 into a predetermined trained model. In other words, the first closed question is a question that asks whether the content indicated by the open answer is correct or incorrect, and the correct answer is "yes". This trained model may be the generation AI 11 described above, or any other trained model. This trained model may be stored in the test case generation support system 30, or it may be stored in an external information processing system. In this embodiment, it is assumed to be a trained model 47 stored in an external information processing system. The type of this trained model 47 is the same as that of the generation AI 11 described above.

[0032] The affirmative sentence generation filter 55 in the first closed question generation filter 54 generates an affirmative sentence (affirmative descriptive sentence) that explains the content of the open answer by inputting the open question and the open answer generated by the open question generation filter 52 into the trained model 47. The first closed question generation filter 54 generates the first closed question by inputting the generated affirmative sentence into the trained model 47.

[0033] Next, the second closed question generation filter 56 modifies the topic portion (the part asking for true or false) of the first closed question generated by the first closed question generation filter 54 using a predetermined algorithm (i.e., modifies it to false content), and generates a second closed question that asks for true or false about the modified topic portion. In other words, the second closed question is a question that asks for a "yes" or "no" answer regarding the true or false content of the question, and the correct answer is "no".

[0034] Specifically, the second closed question generation filter 56 includes a keyword extraction filter 57, a antonym generation filter 58, and a string replacement filter 59 as its components.

[0035] The keyword extraction filter 57 extracts keywords related to the above topic from the first closed question generated by the first closed question generation filter 54.

[0036] The antonym generation filter 58 retrieves other keywords (antonyms, etc.) with meanings different from the keywords extracted by the keyword extraction filter 57.

[0037] The string replacement filter 59 modifies the topic portion of the first closed question generated by the first closed question generation filter 54 by replacing the keyword in the first closed question generated by the first closed question generation filter 54 with the other keyword (antonym, etc.). Then, the second closed question generation filter 56 generates a second closed question that asks whether the modified topic portion is correct or incorrect.

[0038] The similarity calculation filter 60 will be described in Example 3. Note that the similarity calculation filter 60 can be provided at will.

[0039] The evaluation value calculation unit 61 and the filter evaluation filter 62 will be described in Embodiment 4. Note that the evaluation value calculation unit 61 and the filter evaluation filter 62 are optional.

[0040] Next, as shown in Figure 2, the LLM call unit 45 inputs various data received from the test case generation unit 44 into a predetermined trained model to obtain the data necessary for generating closed questions and their answers, and transmits the obtained data to the test case generation unit 44. In this embodiment, this trained model is a trained model 47 stored in an external information processing system, similar to the first closed question generation filter 54.

[0041] The test case output unit 46 outputs a test case 36 based on the data generated by the test case generation unit 44. Specifically, the test case output unit 46 generates a test case that includes a second closing question and an answer to the second closing question stating that the part of the topic modified by the string replacement filter 59 is incorrect. The generated test case 36 is used by the model verification system 20.

[0042] The functions of the test case generation support system 30 described above are realized by the hardware of the test case generation support system 30, or by the arithmetic unit 31 of the test case generation support system 30 reading and executing each program stored in the storage device 32. Further, these programs are stored in a storage device such as a secondary storage device, a nonvolatile semiconductor memory, a hard disk drive, an SSD, or a recording medium readable by the test case generation support system 30 such as an IC card, an SD card, a DVD, etc. Note that the test case generation support system 30 may be realized using virtual information processing resources provided using virtualization technology, process space separation technology, etc., such as a virtual server provided by a cloud system, for example. Also, all or part of the functions provided by these devices may be realized by a service provided by a cloud system via an API (Application Programming Interface) or the like. Next, the processing performed in the generation AI verification support system 1 will be described.

[0043] <Test Case Generation Support Processing> Figure 4 is a flowchart for explaining the outline of the test case generation support processing for supporting the generation of test cases. The generation AI creation support processing is started, for example, when the test case generation support system 30 receives a predetermined input from the user.

[0044] First, the domain knowledge input unit 42 of the test case generation support system 30 acquires information on the domain knowledge of the business in the relevant field from the domain knowledge information 41 (s1).

[0045] The domain knowledge division selection unit 43 of the test case generation support system 30 divides each text included in the domain knowledge information acquired in s1 based on a predetermined rule (s3). Hereinafter, each divided part will be referred to as a chunk.

[0046] For example, for each sentence in the text, the proprietary knowledge segmentation and selection unit 43 divides the sentence into parts with a predetermined character string length, and at that time, consecutive parts overlap by a predetermined number of characters (between a certain chunk and the next chunk, the last character string part of a certain chunk and the character string part of the next chunk overlap). By taking a sufficient overlap part, it is possible to prevent the eigenvalue formula from becoming a fragmented chunk that does not make sense. Note that the proprietary knowledge segmentation and selection unit 43 may divide the sentence into each chunk by extracting each meaningful part content-wise by performing analysis (such as morphological analysis) on the structure of the document (chapters or sections, etc.) instead of a predetermined character string length.

[0047] After that, the test case generation unit 44 of the test case generation support system 30 executes a first test case generation support process s3 for generating second closed questions based on each chunk created in s3. The details of the first test case generation support process s3 will be described later.

[0048] In addition, the test case generation unit 44 of the test case generation support system 30 executes a second test case generation support process s4 for generating other closed questions based on each chunk created in s3. The details of the second test case generation support process s4 will be described later.

[0049] The test case output unit 46 of the test case generation support system 30 outputs test case data based on each data created in the first test case generation support process s3 and the second test case generation support process s4 (s5).

[0050] Specifically, the test case output unit 46 generates a test case including the second closed question generated in the first test case generation support process s3 and the correct answer corresponding to the second closed question (for example, "No"). In addition, the test case output unit 46 generates a test case including the closed question generated in the second test case generation support process s4 and the correct answer corresponding to the closed question (for example, "Yes").

[0051] Subsequently, the test case output unit 46 transmits the generated test cases to the model verification system 20. This completes the test case generation support process.

[0052] Next, we will explain the details of each of the above processes.

[0053] <First Test Case Generation Support Process> Figure 5 is a flowchart illustrating the details of the first test case generation support process s3. The test case generation unit 44 first generates and stores open questions related to proprietary knowledge based on the chunks generated in s2 (hereinafter referred to as target chunks) (s31).

[0054] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as the open question request prompt) that requests the generation of an open question based on each target chunk, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the model to output an open question.

[0055] An example of an open-ended question output by the trained model 47 is: "This document details quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. What are the relationship between AI product-specific quality characteristics and data and models?"

[0056] (Open Question Request Prompt) Figure 6 shows an example of an open question request prompt. This open question request prompt 600 includes a statement 601 that requests the generation of an open question asking about the contents of a target chunk, including a specified keyword, and an example of input / output data. As an example of input / output data, three target chunks 602, 604, and 606 and three open questions 603, 605, and 607 corresponding to these target chunks are set.

[0057] Then, as shown in Figure 5, the test case generation unit 44 generates answers (open answers) to each open question generated in s31 (s32).

[0058] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as the open answer request prompt) that requests the generation of an open answer based on each open question, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the model to output an open answer.

[0059] The open-ended response output by the trained model 47 is, for example, a sentence answering an open-ended question, such as, "Data and models are internal properties."

[0060] (Open-Answer Request Prompt) Figure 7 shows an example of an open-answer request prompt. This open-answer request prompt 700 includes a statement 701 that requests the generation of an open answer to an open question using the target chunk, and an example of input / output data. As an example of input / output data, two target chunks 702 and 705, two corresponding open questions 703 and 706, and two corresponding open answers 704 and 707 are set.

[0061] As described above, by generating open questions and open answers in chunk units, clear questions and answers can be generated. If a summary of the content of proprietary knowledge information 41 is used, the questions and answers may become unclear.

[0062] Then, as shown in Figure 5, the test case generation unit 44 generates an affirmative explanatory sentence for each open question by describing the matter that the open question is asking based on the open answer generated in s32 (s33).

[0063] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as an affirmative statement request prompt) that requests the trained model 47 to convert the open question into a normal sentence format by rewriting the open question, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, causing the model to output an explanatory text for an affirmative statement.

[0064] An example of an affirmative statement explanation produced by the trained model 47 is: "This document provides a detailed explanation of quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. Quality characteristics specific to AI products refer to data and models as internal characteristics."

[0065] (Affirmative Statement Request Prompt) Figure 8 shows an example of an affirmative statement request prompt. This affirmative statement request prompt 800 includes a statement 801 that requests that an open question be converted into an affirmative statement based on the open answer, and an example of input / output data. The example of input / output data includes two open questions 802 and 806, two corresponding target chunks 803 and 807, two corresponding open answers 804 and 808, and two corresponding affirmative statement explanations 805 and 809. The affirmative statement request prompt 800 may also include, for example, a statement 810 that instructs the output of an affirmative statement, a statement 811 that requests the correspondence between the open question and the open answer, and a statement 812 that requests the extraction of necessary information from the target chunk.

[0066] Then, as shown in Figure 5, the test case generation unit 44 generates a first closing question related to the explanatory text generated in s33 (s34).

[0067] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a closed question request prompt) that requests the description generated in s33 to be converted into a question asking whether or not it is correct, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the closed question to be output.

[0068] A closed question output by the trained model 47 might be: "This document details quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. AI product-specific quality characteristics refer to data and models as internal characteristics. Please answer 'yes' or 'no'." This closed question is designed to elicit a "yes" or "no" answer to the open question.

[0069] (Closing Question Request Prompt) Figure 9 shows an example of a closing question request prompt. This closing question request prompt 900 includes a statement 901 that requests the generation of a first closing question by rewriting the explanatory text of an affirmative statement, and an example of input / output data. As an example of input / output data, two explanatory texts 902 and 904 of affirmative statements and two corresponding first closing questions 903 and 905 are set. The closing question request prompt 900 may also include, for example, a statement 906 that requests the generation of a first closing question that can be answered with "yes" or "no", and a statement 907 that requests the first closing question to be expressed in such a way that "yes" is the correct answer.

[0070] Next, as shown in Figure 5, the test case generation unit 44 generates all of the first closing questions and then extracts keywords from each of the first closing questions (s35).

[0071] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a keyword request prompt) that requests the extraction of important words or phrases from the first closed question generated in s34, and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the keyword.

[0072] The keywords output by the trained model 47 are, for example, "data and model".

[0073] (Keyword Request Prompt) Figure 10 shows an example of a keyword request prompt. This keyword request prompt 1000 includes a statement 1001 that requests the extraction of the most suitable keyword for the second close question from the first close question, and an example of input / output data. As an example of input / output data, the first close question 1002 and the corresponding keyword 1003 are set. The keyword request prompt 1000 may also include, for example, a statement 1004 that indicates the keyword is used in a question of the form of a fill-in-the-blank format (in this case, a fill-in-the-blank format), and a statement 1005 that indicates the keyword is text present in the close question.

[0074] Next, as shown in Figure 5, the test case generation unit 44 extracts other keywords (words or phrases, etc.) with different content from the keyword extracted in s35 (s36).

[0075] In this embodiment, the test case generation unit 44 generates a prompt (hereinafter referred to as a synonym request prompt) that requests the keyword extracted in s35 to be converted into a word or phrase with a different meaning (including negation and opposite) (hereinafter referred to as a synonym in this specification), and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the synonym.

[0076] The antonyms output by the trained model 47 include, for example, "data" and "model," as well as "emotions and experiences," which are words with different meanings.

[0077] (Antonym Request Prompt) Figure 11 shows an example of an antionym request prompt. This antionym request prompt 1100 includes a request 1101 to convert the keyword of the first closing question into an antionym, and an example of input / output data. As an example of input / output data, the keyword 1102 and antionym 1103 of the first closing question are set. The antionym request prompt 1100 may also include, for example, a statement 1104 that requests conversion to an antionym of the same level of abstraction (depth or breadth of vocabulary), a statement 1105 that requests not to use prefixes or suffixes that express negation such as "not", "not", or "none", and a statement 1106 that requests not to simply convert characters or language.

[0078] Next, as shown in Figure 5, the test case generation unit 44 generates a new closing question (i.e., a second closing question) for each of the first closing questions generated in s34, by replacing the keyword in the first closing question with the antonym generated in s36 (s37).

[0079] This generates a closed question whose answer is "No". This concludes the first test case generation support process s3.

[0080] The second closing question generated is, for example, "It describes in detail the quality characteristics, internal characteristics, development processes, risk analysis, security, ethics, organization and education, technology updates and innovation, fair competition, a list of techniques, and case studies that engineers of AI-powered systems should consider. The quality characteristics specific to AI products mention emotions and experiences as internal characteristics. Please answer 'yes' or 'no'." Compared to the first closing question in s34 mentioned above, "data and models" has been changed to "emotions and experiences." The correct answer is "no."

[0081] <Second Test Case Generation Support Process> Figure 12 is a flowchart illustrating the details of the second test case generation support process s4. First, the test case generation unit 44 generates open questions about specific knowledge based on the target chunk, similar to s31 (s41).

[0082] Then, the test case generation unit 44 generates open answers for each open question generated in s41, similar to s32 (s42).

[0083] The test case generation unit 44 generates a first closed question related to each open question generated in s42, similar to s33 (s43).

[0084] This generates a closed question that will be answered "yes". This concludes the second test case generation support process s4.

[0085] As described above, the test case generation support system 30 of this embodiment generates an open answer corresponding to an open question by inputting an open question that asks for an answer on a predetermined topic into a trained model 47, generates a first closed question that asks whether the open answer is correct and requests that the open answer be correct as the correct answer by inputting the open question and the open answer into the trained model 47, modifies the topic portion of the generated first closed question and generates a second closed question that asks whether the modified topic portion is correct or incorrect.

[0086] In other words, the test case generation support system 30 of this embodiment, based on an open question and an open answer, assumes a first closed question which is a question asking whether the open answer is correct or incorrect (for example, a question where "yes" is the correct answer), changes the topic part of this first closed question to a different content, and generates a second closed question which asks whether the changed topic part is correct or incorrect.

[0087] This ensures the generation of closed questions where the topic of the question is incorrect (for example, questions where "no" is the correct answer). For instance, directly attempting to generate a question where the correct answer is "no" may result in a nonsensical question due to the characteristics of the trained model (e.g., "Is it correct that XX is not OO?"). By using this second closed question to generate test cases to evaluate the quality of the generated AI 11, it becomes possible to perform highly accurate validation of the trained model, which tends to learn from positive information.

[0088] Thus, the test case generation support system 30 of this embodiment can support the generation of test cases for verifying the quality of a trained model with high accuracy.

[0089] Furthermore, the test case generation support system 30 of this embodiment, in relation to generating the second closing question, extracts topic keywords from the first closing question, obtains other keywords (antonyms, etc.) with meanings different from the extracted keywords, and replaces the keywords in the first closing question with the antonyms, etc. thereby changing the topic portion of the first closing question, and generates a second closing question that asks whether the changed topic portion is correct or incorrect.

[0090] In this way, by replacing the topic portion of the first closed question with an antonym to generate the second closed question, it is possible to reliably generate a closed question in which the topic portion (the part asking for true or false) is incorrect. In other words, by converting important words into words with different meanings, it is possible to reliably generate a question in which the correct answer is, for example, "no".

[0091] Furthermore, in this embodiment, the test case generation support system 30 generates an affirmative sentence explaining the content of the open answer by inputting the open question and open answer into the trained model 47, and then generates the first closed question by inputting the generated affirmative sentence into the trained model 47.

[0092] In this way, by generating the first closed question based on an affirmative statement that explains the content of the open answer, it is possible to reliably generate a closed question that requires the open answer to be correct as the correct answer.

[0093] Furthermore, the test case generation support system 30 of this embodiment generates a test case that includes a second closing question and an answer to the second closing question (for example, an answer of "no") indicating that the part in which the keyword has been changed to a synonym is incorrect.

[0094] This makes it possible to generate test cases for questions where a negative answer such as "no" is considered the correct answer for the trained model 47.

[0095] [Example 2] In Example 2, when generating the second closing question, the antonyms of the second closing question are limited to those that have a predetermined relationship with the corresponding keyword of the first closing question.

[0096] The configuration of the test case generation support system 30 in Example 2 is the same as in Example 1. In addition, the test case generation support system 30 has the following functional components in addition to those in Example 1.

[0097] In other words, the test case generation unit 44 includes a similarity calculation filter 60. The similarity calculation filter 60 calculates the similarity between the topic portion of the first closed question generated by the first closed question generation filter 54 and the modified topic portion (e.g., antonyms), determines whether the calculated similarity is within a predetermined range, and if it determines that the similarity is within the predetermined range, it generates a second closed question asking whether the modified topic portion is correct or incorrect.

[0098] The test case generation support process in Example 2 is the same as in Example 1. However, the first test case generation support process is the process described below.

[0099] <First Test Case Generation Support Process> Figure 13 is a flowchart illustrating the details of the first test case generation support process s6 according to Example 2. The processes s31 to s36 shown in the figure are the same as the processes s31 to s36 in Example 1.

[0100] After processing in s36, the test case generation unit 44 calculates the similarity between the keyword generated in s35 and the corresponding antonym generated in s36 (s61). If the calculated similarity is outside a predetermined range, the test case generation unit 44 repeats the processing in s36 to extract different antonyms.

[0101] For example, the test case generation unit 44 calculates vector values ​​for each keyword and antonym based on the proprietary knowledge information 41 (embedding), and determines the similarity by calculating the cosine similarity between the calculated vector values. Alternatively, the test case generation unit 44 may calculate the similarity based on the degree of matching of each character of the keyword and antonym, or based on a predetermined comparison database.

[0102] On the other hand, if the calculated similarity is within a predetermined range, the test case generation unit 44 generates a second closed question (s37), similar to the first example. This completes the first test case generation support process s6.

[0103] Thus, in this embodiment, the test case generation support system 30 generates a second closing question that asks whether the modified topic is correct or incorrect if the similarity between the topic portion of the first closing question and the modified topic portion is within a predetermined range.

[0104] This allows for the generation of questions of varying difficulty levels, adjusting the degree to which the modified topic deviates from the correct answer, and enabling adjustment of the level of quality verification of the generating AI 11 using test cases.

[0105] [Example 3] In Example 3, the second closed question is not a binary question with a "yes" or "no" answer, but a multiple-choice question from which the correct answer or no correct answer is selected.

[0106] The configuration of the test case generation support system 30 in Example 3 is the same as in Example 1. Furthermore, the functional units in the test case generation support system 30 are the same, except for the second closed question generation filter.

[0107] In other words, the second closed question generation filter 56 generates multiple modified parts (such as antonyms) by changing the topic part of the first closed question generated by the first closed question generation filter 54, and uses the generated multiple modified parts and the topic part as question elements (choices), and generates a second closed question that asks whether each question element is correct or incorrect.

[0108] The test case generation support process in Example 3 is the same as in Example 1. However, the first test case generation support process is the process described below.

[0109] <First Test Case Generation Support Process> Figure 14 is a flowchart illustrating the details of the first test case generation support process s7 according to Example 3. The processes s31 to s33 shown in the figure are the same as the processes s31 to s33 in Example 1.

[0110] After processing in s33, the test case generation unit 44 extracts one keyword (hereinafter referred to as the correct keyword) from the explanatory text of the affirmative statement generated in s33 (s71).

[0111] In this embodiment, the test case generation unit 44 generates a prompt requesting the extraction of keywords from the descriptive text of the affirmative statement generated in s33, and inputs the generated prompt to the trained model 47 via the LLM call unit 45 to output the keywords.

[0112] The test case generation unit 44 obtains several other keywords that have different meanings from the correct keyword extracted in s71 (s72).

[0113] In this embodiment, the test case generation unit 44 generates a prompt requesting the output of multiple other keywords (keywords in options other than the correct answer option) that have different meanings from the keyword extracted in s71, and inputs the generated prompt to the trained model 47 via the LLM call unit 45, thereby causing the multiple other keywords to be output. The extraction of other keywords may be performed, for example, based on the aforementioned antonym request prompt.

[0114] The test case generation unit 44 generates the lead portion of the second closed question (s73) by outputting a new explanatory text in which the correct keyword portion extracted in s71 from the explanatory text of the affirmative statement generated in s33 is masked with a predetermined string.

[0115] The test case generation unit 44 has the lead portion generated in s73 and outputs a question (multiple-choice question) in which the multiple keywords obtained in s72 are each selected as answer options (s74).

[0116] In this embodiment, the test case generation unit 44 generates a prompt (multiple-choice closed question request prompt) that requests the output of a multiple-choice question having a lead portion that requires selecting the correct option from among the options, or selecting "no correct answer" if none of the options are correct. The generated prompt is then input to the trained model 47 via the LLM call unit 45, causing the question to be output.

[0117] (Multiple-Choice Closed Question Request Prompt) Figure 15 shows an example of a multiple-choice closed question request prompt. This multiple-choice closed question request prompt 1500 includes a phrase 1501 that requests the generation of a multiple-choice question sentence including a masked lead portion and multiple answer choices, and an example of input / output data. As an example of input / output data, a lead portion 1502, an option other than the correct keyword 1503, the correct keyword 1504, and the entire second closed question 1505 are set. The closed question request prompt 1500 may also include a phrase 1506 that instructs the output of a multiple-choice question, a phrase 1507 that requests the generation of each option related to the correct keyword and other keywords within a predetermined number, etc.

[0118] Subsequently, as shown in Figure 14, the test case generation unit 44 obtains the correct keyword for the second closed question generated in s74, similar to s35 in Example 1.

[0119] The test case generation unit 44 acquires keywords (synonyms, etc.) that are different from the correct keywords generated in s35, similar to s36 in Example 1.

[0120] Then, the test case generation unit 44 generates a second closed question by replacing the correct keyword obtained in s37 with another keyword obtained in s36 (s75). This generates a second closed question with multiple choices in which the option related to the original correct keyword is not the correct answer. This completes the first test case generation support process s7.

[0121] Subsequently, the test case output unit 46 generates a test case that includes the second closing question generated in the first test case generation support process s3 and the correct answer corresponding to the second closing question (for example, the correct answer options, or "no correct answer").

[0122] As described above, the test case generation support system 30 of this embodiment generates a second closing question by creating multiple modified parts (such as antonyms) by changing the topic part of the first closing question, and using the generated antonyms and the topic part as options, it generates a second closing question that asks whether each option is correct or incorrect (which of the options is correct, or none of the options are correct).

[0123] In this way, by generating a second closed question with multiple-choice options, the accuracy of the knowledge generated by the test case AI 11 can be verified with high precision.

[0124] [Example 4] In Example 4, the second closing question, which is appropriate as a test case, is identified by evaluating each question and answer generated by the test case generation unit 44.

[0125] The configuration of the test case generation support system 30 in Example 4 is the same as in Example 1. In addition to the functions of Example 1, the test case generation support system 30 also includes the following functions.

[0126] In other words, the test case generation support system 30 includes an evaluation value calculation unit 61 and a filter evaluation filter 62.

[0127] The evaluation value calculation unit 61 calculates evaluation values ​​for the open-ended questions, the first closed-ended questions, and the second closed-ended questions, respectively.

[0128] Similarly, the evaluation value calculation unit 61 calculates evaluation values ​​for the generated affirmative sentences and other elements such as keywords.

[0129] The filter evaluation filter 62 determines whether or not to generate a test case based on the evaluation values ​​calculated by the evaluation value calculation unit 61, and if it determines that a test case should be generated, it generates a test case that includes a second closed question.

[0130] The test case generation support process in Example 4 is the same as in Example 1. However, the first test case generation support process is as described below.

[0131] <First Test Case Generation Support Process> Figures 16 and 17 are flowcharts illustrating the details of the first test case generation support process s8 related to Example 4 (divided into two figures for space limitations).

[0132] First, as shown in Figure 16, the test case generation unit 44 performs the process in s31 in the same manner as in Embodiment 1, and then the evaluation value calculation unit 61 evaluates the open question request prompt and the outputted open question (s81).

[0133] In this embodiment, the evaluation value calculation unit 61 generates an open question request prompt and a prompt requesting an evaluation of the outputted open question, and inputs the generated prompts to the trained model 47 to output an evaluation value for the open question.

[0134] Next, the test case generation unit 44 performs the process in s32 in the same manner as in Example 1, and then evaluates the open response request prompt and the outputted open response (s82).

[0135] In this embodiment, the evaluation value calculation unit 61 generates an open response request prompt and a prompt requesting an evaluation of the outputted open response, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0136] Next, after the test case generation unit 44 performs the process in s33 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the affirmative statement request prompt and the outputted explanation of the affirmative statement (s83).

[0137] In this embodiment, the evaluation value calculation unit 61 generates a prompt requesting an affirmative statement and a prompt requesting an evaluation of the explanatory text of the output affirmative statement, and outputs an evaluation value by inputting the generated prompts into the trained model 47.

[0138] Next, after the test case generation unit 44 performs the process in s34 in the same manner as in Embodiment 1, the evaluation value calculation unit 61 evaluates the close question request prompt and the outputted first close question (s84).

[0139] In this embodiment, the evaluation value calculation unit 61 generates a closed question request prompt and a prompt requesting an evaluation of the outputted first closed question, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0140] Next, as shown in Figure 17, after the test case generation unit 44 performs the process in s35 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the keyword request prompt and the outputted keywords (s85).

[0141] In this embodiment, the evaluation value calculation unit 61 generates a keyword request prompt and a prompt requesting an evaluation of the output keyword, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0142] Next, after the test case generation unit 44 performs the process in s36 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the antinomial word request prompt and the output antinomial words (s86).

[0143] In this embodiment, the evaluation value calculation unit 61 generates a similar word request prompt and a prompt requesting an evaluation of the outputted similar words, and outputs an evaluation value by inputting the generated prompts to the trained model 47.

[0144] Next, after the test case generation unit 44 performs the process in s37 in the same manner as in Example 1, the evaluation value calculation unit 61 evaluates the generated second closed question (s87).

[0145] In this embodiment, the evaluation value calculation unit 61 generates a prompt requesting an evaluation of the second closed question, and outputs an evaluation value by inputting the generated prompt to the trained model 47.

[0146] Furthermore, the evaluation value calculation unit 61 may attach information on evaluation criteria to the prompt when calculating each of the above evaluation values. For example, the evaluation value calculation unit 61 may attach an evaluation request based on whether or not there is logical consistency between the prompt and the output text, or an evaluation request based on whether or not there is ambiguous expression in the wording of the prompt or the output text.

[0147] Next, the filter evaluation filter 62 calculates an overall evaluation value from the evaluation values ​​output by the above process (s88). For example, the filter evaluation filter 62 calculates an overall evaluation value by multiplying each evaluation value by a predetermined weighting value and summing up the weighted multipliers. Alternatively, for example, the filter evaluation filter 62 may use the lowest value of each evaluation value as the overall evaluation value.

[0148] Subsequently, the filter evaluation filter 62 determines whether the overall evaluation value calculated in s88 exceeds a predetermined threshold (s89). If the filter evaluation filter 62 determines that the overall evaluation value exceeds the predetermined threshold, it generates a corresponding test case in the process of s5 described above.

[0149] As described above, the test case generation support system 30 of this embodiment calculates an evaluation value for at least one of the open answer, the first closed question, and the second closed question, and generates a test case 36 if the calculated evaluation value exceeds a predetermined threshold.

[0150] This allows for a proper determination of whether an appropriate second closed question has been generated as a test case, and enables the generation AI 11 to use only appropriate test cases. In particular, when a trained model 47 is used to output each data, there is a risk that low-quality data such as responses may be output due to its characteristics, so it is highly necessary to evaluate based on evaluation values.

[0151] The present invention is not limited to the embodiments described above, and can be implemented using any components without departing from its spirit. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0152] For example, some of the hardware provided in each device of each embodiment may be provided in other devices.

[0153] Furthermore, each program of each device may be provided in other devices, a program may consist of multiple programs, or multiple programs may be integrated into a single program.

[0154] Furthermore, in this embodiment, an affirmative explanatory text is generated based on the open question and open answer, and a first closed question is generated based on the generated explanatory text. However, the first closed question may also be generated directly using other algorithms such as predetermined string manipulation based on the open question and open answer.

[0155] Furthermore, in this embodiment, when generating the second closed question, if a keyword is to be replaced with a synonym, the trained model 47 may be made to perform the replacement.

[0156] Furthermore, in this embodiment, when generating the second closing question from the first closing question, the question was modified by replacing keywords with antonyms, but modifications using other algorithms may also be performed. For example, modifications such as swapping the subject and predicate, or reversing the cause-and-effect relationships in a sentence may be performed.

[0157] 30 Test case generation support system, 44 Test case generation unit, 47 Trained model

Claims

1. A test case generation support system comprising: a storage device that stores open questions, which are natural language questions that request answers on a predetermined topic; an open answer generation process that generates open answers, which are natural language answers corresponding to the open questions, by inputting the open questions into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data; a first closed question generation process that generates a first closed question, which is a natural language question that asks whether the open answer is correct and requests that the open answer be correct, by inputting the open questions and the generated open answers into the trained model; and a second closed question generation process that modifies the topic portion of the generated first closed question using a predetermined algorithm and generates a second closed question, which is a question that asks whether the modified topic portion is correct.

2. The arithmetic unit, in the second closed question generation process, performs a keyword extraction process to extract keywords of the topic from the first closed question generated; an anti-keyword acquisition process to obtain other keywords with different meanings from the extracted keywords; and a string replacement process to change the topic portion of the first closed question generated by replacing the keywords in the first closed question generated with the other keywords, thereby generating a second closed question that asks whether the changed topic portion is correct or incorrect, the test case generation support system according to claim 1.

3. The test case generation support system according to claim 1, wherein the computing device, in the first closed question generation process, inputs the open question and the generated open answer to the trained model to generate an affirmative sentence generation process that generates an affirmative sentence explaining the content of the open answer, and inputs the generated affirmative sentence to the trained model to generate the first closed question.

4. The test case generation support system according to claim 1, wherein the calculation device, in the second closed question generation process, calculates the similarity between the topic portion of the first closed question generated and the modified topic portion, determines whether the calculated similarity is within a predetermined range, and if it determines that the similarity is within a predetermined range, generates a second closed question asking whether the modified topic portion is correct or incorrect.

5. The test case generation support system according to claim 1, wherein the multi-choice calculation device generates a plurality of modified parts by changing the topic part of the first closed question generated in the second closed question generation process, and generates a second closed question that asks whether each of the generated plurality of modified parts and the topic part is a question element.

6. The test case generation support system according to claim 1, wherein the computing device performs a test case output process that generates a test case including the generated second closing question and an answer to the second closing question stating that the modified topic is incorrect.

7. The test case generation support system according to claim 6, wherein the calculation device performs an evaluation value calculation process to calculate an evaluation value for at least one of the open answer, the first closed question, and the second closed question, and generates the test case when the calculated evaluation value exceeds a predetermined threshold.

8. A test case generation support method by an information processing device comprising a storage device for storing open questions, which are natural language questions that ask for answers on a predetermined topic, and an arithmetic unit, wherein the arithmetic unit performs: an open answer generation process that generates an open answer, which is a natural language answer corresponding to the open question, by inputting the open question into a trained model that takes natural language data as input and outputs natural language data corresponding to the input data; a first closed question generation process that generates a first closed question, which is a natural language question that asks whether the open answer is correct and requests that the open answer be correct, by inputting the open question and the generated open answer into the trained model; and a second closed question generation process that modifies the topic portion of the generated first closed question using a predetermined algorithm and generates a second closed question, which is a question that asks whether the modified topic portion is correct.

Images