Program, method, information processing apparatus, and system

Abstract

To reduce labor for creating a test case.SOLUTION: A program for causing a computer including a processor and a memory to execute: The program causes a processor to execute a step of inputting information related to test content to a first trained model trained to output an abstract expected value and causing the first trained model to output the abstract expected value, a step of inputting the abstract expected value and information related to a specification to a second trained model trained to output a specific expected value and causing the second trained model to output the specific expected value, and a step of presenting the specific expected value output from the second trained model or both the abstract expected value and the specific expected value to a user.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] To ensure the quality of software, its functionality is tested before it is released. Tests are designed based on the software specifications to ensure the desired quality is achieved within the development schedule and budget set by the software development company.

[0003] A technology has been proposed for obtaining appropriate test items for each specification item from multiple perspectives (see Patent Document 1). In Patent Document 1, the appearance trends of words in new specification data including multiple specification items that describe the specifications of the development target using one or more words are analyzed for each specification item, and each specification item is classified into one of multiple test categories, each of which is a test category, based on the analysis results of the appearance trends of words for each specification item. Furthermore, past classification result information that associates test items that describe test content with test categories is referenced, and for each specification item, test items that are associated in the past classification result information with the test category into which the specification item was classified are presented. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-115926 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, past test items classified into test categories are presented to the test designer for each new specification item. However, Patent Document 1 does not create test content (expected values) that reflects the contents of the specification. Therefore, when a user tries to create a test case corresponding to the specification, the user must modify the presented past test items to match the specification.

[0006] The purpose of this disclosure is to reduce the effort required to create test cases. [Means for solving the problem]

[0007] A program to be executed by a computer having a processor and a memory, the program instructs the processor to execute the following steps: inputting information related to test content into a first trained model trained to output an abstract expected value and causing the first trained model to output the abstract expected value; inputting information related to the abstract expected value and specifications into a second trained model trained to output a concrete expected value and causing the second trained model to output the concrete expected value; and presenting the concrete expected value output from the second trained model, or both the abstract expected value and the concrete expected value, to a user. [Effects of the Invention]

[0008] According to the present disclosure, the effort required to create test cases can be reduced. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Figure 2] 2 is a block diagram illustrating an example of the configuration of a terminal device 10 shown in FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of a server 20. [Figure 4] FIG. 2 is a diagram showing the data structure of a user information table 2021. [Figure 5]FIG. 20 is a diagram showing the data structure of a specification table 2022. [Figure 6] FIG. 10 is a diagram showing the data structure of a prompt table 2023. [Figure 7] FIG. 2 is a diagram showing the data structure of a test design document table 20241. [Figure 8] FIG. 20 is a diagram showing the data structure of a pattern table 20242. [Figure 9] FIG. 10 is a diagram showing the data structure of a test viewpoint table 20243. [Figure 10] 3 is a schematic diagram illustrating the operation of the server 20. FIG. [Figure 11] 10 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document. FIG. [Figure 12] 10 is a schematic diagram showing an example of a test design document creation screen displayed on a display 141. FIG. [Figure 13] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 14] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 15] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 16] FIG. 10 is a block diagram illustrating an example of the operation of the server 20 when outputting an expected value. [Figure 17] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 18] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 19] FIG. 10 is a block diagram illustrating an example of the operation of the server 20 when outputting an expected value. [Figure 20] FIG. 10 is a block diagram illustrating an example of the operation of the server 20 when outputting an expected value. [Figure 21]FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. The names and functions of the components are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0011] <Summary> The system according to this embodiment uses a trained model in each step of creating test cases based on specifications. Specifically, for example, in the step of identifying test targets, the system uses the trained model to output test categories. In the step of selecting test perspectives, the system uses the trained model to output test perspectives and check items. In the step of determining specific test content, the system uses the trained model to output test patterns. In the step of determining specific test content, the system uses the trained model to output expected values ​​of the test. For example, large language models (LLMs) may be used as the trained model.

[0012] To improve the accuracy of the expected values ​​to be output, the system uses, for example, two types of trained models to output the expected values. Specifically, for example, the system inputs information related to the test content into a first trained model for expected values ​​that has been trained to output abstract expected values, and causes the first trained model to output the abstract expected values. The system inputs information related to the abstract expected values ​​and specifications into a second trained model for expected values ​​that has been trained to output concrete expected values, and causes the second trained model to output concrete expected values.

[0013] In this embodiment, the test section may be expressed by other expressions. For example, the test section may use expressions that represent the target of the test, such as test target item or test target function. The test perspective may be expressed by other expressions. For example, the test perspective may use expressions that represent the content of the test, such as the viewpoint (way of thinking) on ​​which the test is based. The check items may be expressed by other expressions. For example, the check items may use expressions that represent information for specifying the test to be performed. The expected values ​​may be expressed by other expressions. For example, the expected values ​​may use expressions that represent the behavior or state that is expected to occur during the check.

[0014] <1 Overall system configuration> Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

[0015] 1 shows an example in which the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two. The number of terminal devices 10 included in the system 1 may be one, or three or more.

[0016] 1 shows an example in which the system 1 includes one LLM system 30, but the number of LLM systems 30 included in the system 1 is not limited to one. The number of LLM systems 30 included in the system 1 may be two or more.

[0017] In this embodiment, a collection of multiple devices may be considered as one server. The allocation of multiple functions required to realize the server 20 according to this embodiment to one or more pieces of hardware may be determined appropriately in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20. Specifically, for example, a server having a function for creating test cases based on specifications and a server on which a trained model is built may be different devices.

[0018] The terminal device 10 shown in Fig. 1 is an information processing device used by a user who uses a service provided by the server 20. For example, the terminal device 10 is an information processing device operated by a test designer who creates a test design document using the test design document creation support provided by the server 20. The terminal device 10 is realized by, for example, a desktop personal computer (PC), a laptop PC, a head-mounted display, or the like. The terminal device 10 may also be a portable computer such as a smartphone or a tablet terminal.

[0019] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.).

[0020] The server 20 is an information processing device that provides a service to support the creation of a test design document. For example, the server 20 uses a plurality of trained models to support the creation of a test design document by a user.

[0021] The server 20 is, for example, an information processing device realized by a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a user and as an interface with an output device for outputting information to the user.

[0022] The LLM system 30 is a system built with a large-scale artificial intelligence model (LLM) used in the field of natural language processing (NLP), for example. By learning from large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns in human language and effectively perform natural language generation (NLG) tasks.

[0023] LLMs are used in many NLP tasks, such as generating answers to specific questions, automatic sentence generation, text summarization, translation, and sentiment analysis. They can also be used in a variety of applications, including education, entertainment, customer service, and product development. Examples of LLMs include: GPT-4 (registered trademark) (OpenAI) PaLM2 (registered trademark) (Google Inc.) StableLM (StableAI) Llama 2 (Meta)

[0024] The LLM system 30 inputs test-related information and prompts output from the server 20 into the LLM, and causes the LLM to output answers based on the input text data and prompts. The LLM system 30 transmits the answers output from the LLM to the server 20.

[0025] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0026] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.

[0027] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0028] The input device 13 is a device for inputting instructions or information by a user operating the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like, which inputs instructions by touching an operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, or the like. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

[0029] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0030] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0031] The camera 160 is a device that receives light with a light receiving element and outputs the light as an image capturing signal.

[0032] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0033] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0034] The user information 181 includes, for example, information about the user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, contact information, and the like.

[0035] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating in accordance with the program.

[0036] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131 or the like.

[0037] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. For example, the operation reception unit 191 analyzes the received voice signal and extracts a predetermined noun, thereby acquiring an instruction from the user.

[0038] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits information input by a user or instructions input by a user to the server 20. In addition, the transmitting / receiving unit 192 receives information provided by the server 20.

[0039] The presentation control unit 193 controls the output device 14 to present information provided from the server 20 to the user. Specifically, for example, the presentation control unit 194 causes the information transmitted from the server 20 to be displayed on the display 141. In addition, the presentation control unit 194 causes the information transmitted from the server 20 to be output from the speaker 172.

[0040] <1.2 Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0041] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0042] The storage unit 202 includes, for example, a user information table 2021, a specification table 2022, a prompt table 2023, a test information table 2024, and a plurality of trained models. The plurality of trained models includes, for example, a trained model used in at least one step when designing a test.

[0043] Specifically, for example, the multiple trained models include a first trained model 2025 used when selecting a test perspective. The multiple trained models also include, for example, a second trained model 2026 (a first trained model for expected values, an expected behavior generation model) and a third trained model 2027 (a second trained model for expected values, an expected behavior instantiation model) used when determining the specific content of a test. In this embodiment, for example, an LLM constructed in the LLM system 30 is used when identifying test subjects and determining the specific content of a test.

[0044] The information stored in the storage unit 202 is not limited to these.

[0045] The user information table 2021 is a table that stores information about users, and will be described in detail later.

[0046] The specification table 2022 is a table that stores information related to specifications. Details will be described later. In this embodiment, the specification is data that describes the specifications of the functions of the test target. A test designer, for example, creates a test design document while referring to the specification. The specification includes, for example, an area where detailed specifications of the functions are stored and an area where message definitions are stored. If the specification is created using spreadsheet software, these areas are realized, for example, on different sheets. The detailed specifications of the functions include, for example, the specification name, object type, requirements, and control.

[0047] The prompt table 2023 is a table that stores information about prompts (instructions), and will be described in detail later.

[0048] The test information table 2024 is a table that stores information accumulated in various past debugging processes. For example, the test information table 2024 stores test designs created in various past debugging processes and elements decomposed from these test designs. That is, the test information table 2024 is a table that generalizes accumulated test information and shares it as shared knowledge. The elements decomposed from the test designs represent, for example, information for specifying required tests and information in which specific test cases are set. In the test information table 2024, for example, identifiers representing attributes are assigned to the stored test designs and elements. Multiple types of attributes can be set, such as business type / industry, system type, and process type. Using identifiers representing attributes makes it possible to classify test designs or elements according to the attributes.

[0049] In this embodiment, the test design document includes, for example, meta information, information about the test to be performed, and a pattern table. The meta information includes, for example, the title of the test design document, the project name, the product name, the control number, the version, the creation date, the creator, the last update date, the updater, and other information. Note that the information included in the meta information is not limited to these. Any of these pieces of information may be absent, or other information may be included.

[0050] Information about the tests to be conducted plays the role of, for example, a so-called basic design document that defines the general framework of the test. Information about the test includes, for example, "test target items / target functions." "Test target items / target functions" represent, for example, the items / functions that are the subject of the test. "Test target items / target functions" can also be referred to as test categories. Test categories may include sub-concepts such as major functions (category 1), medium functions (category 2), and minor functions (category 3).

[0051] Information about a test includes, for example, a test perspective. A test perspective represents, for example, the standpoint (idea) on which a test is based. A test perspective is associated with a test section. An arrangement of test perspectives, for example, specifies the order of tests. A test perspective is associated with a check item. A check item represents information for identifying a test to be performed.

[0052] The information about the test includes, for example, an expected value. The expected value represents, for example, a behavior or state that is expected to occur during the check. Note that the information about the test is not limited to these. Any of these pieces of information may be absent, or other information may be included.

[0053] The pattern table defines, as patterns, for example, the content to be confirmed, the conditions for the behavior manifested by the confirmation process, and specific test cases that combine the confirmation content and conditions. In the pattern table, for example, at least one pattern is set according to the function to be inspected. In the pattern table, the content to be confirmed is defined, for example, as factors and levels. The factors represent, for example, factors that can affect changes in the state of a computer. The levels represent, for example, specific elements of the factors. In the pattern table, the conditions for the behavior manifested by the confirmation process are defined, for example, by conditions stored as expected values. In the pattern table, specific test cases are defined, for example, using a matrix associated with the factors and levels.

[0054] The test design document is broken down into test perspectives, for example, as information for specifying required tests, and into patterns, for example, as information for specifying specific test cases.

[0055] The first trained model 2025 is a model for outputting a test perspective estimated to be appropriate based on a test section and at least one check item associated with the test perspective. Specifically, the first trained model 2025 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional training. The additional training involves teaching the correct answer for each task, or so-called fine tuning, to specialize the model for creating test design documents. Specifically, for example, the first trained model 2025 is trained to output a test perspective estimated to be appropriate for an input test section and at least one check item associated with the test perspective. In this case, the learning data for training the first trained model 2025 is, for example, the test section stored in the test information table 2024 as input data, and the test perspective and check item stored in the test information table 2024 that correspond to the input test section as correct output data. The information input as input data may be a major function (category 1), a medium function (category 2), or a minor function (category 3) as sub-concepts included in the test category.

[0056] Context tuning may also be performed on the first trained model 2025. In context tuning, for example, an example answer, an answer format, etc. are included in the instruction sentence so that information is output from the first trained model 2025 in a format desired by the user.

[0057] The first trained model 2025 may be retrained at a predetermined timing based on the test category, test perspective, and confirmation items included in the test design document newly added to the test information table 2024.

[0058] The first trained model 2025 may be trained to output test perspectives estimated to be suitable for an input test section and at least one check item associated with the test perspectives, as well as to output an index value indicating the suitability of the output test perspectives. The first trained model 2025 may also be trained to output test perspectives and check items, as well as to output an index value indicating the suitability of the output test perspectives and check items. Depending on the test design document, different test perspectives and check items may be set even for the same test section. The index value is output, for example, by training the first trained model 2025 with learning data in which the test perspectives and check items associated with the test section vary.

[0059] The first trained model 2025 may be trained to output a test perspective estimated to be appropriate for an input test section and at least one check item associated with the test perspective, as well as to output an index value representing the risk of a malfunction occurring in the output test perspective. The first trained model 2025 may also be trained to output a test perspective and check item, as well as to output an index value representing the risk of a malfunction occurring in the output test perspective and check item. For example, the test information table 2024 stores test results, such as the occurrence of a malfunction, for each test perspective, etc. The learning data for training the first trained model 2025 may, for example, use the test section stored in the test information table 2024 as input data, and use the test perspective, check items, and test results stored in the test information table 2024 corresponding to the input test section as correct output data. As a result, an index value representing the malfunction risk is output along with the test perspective and check items.

[0060] The second trained model 2026 is a model for outputting abstract expected values ​​based on test sections, test perspectives, and check items. The second trained model 2026 is referred to as an expected behavior generation model, for example. Specifically, the second trained model 2026 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional learning. In the additional learning, the second trained model 2026 is specialized for creating test design documents by teaching correct answers for each task, i.e., fine-tuning. Specifically, for example, the second trained model 2026 is trained to output abstract expected values ​​for test sections, test perspectives, and check items. In this case, the learning data for training the second trained model 2026 is, for example, the test sections, test perspectives, and check items stored in the test information table 2024 as input data, and the abstract expected values ​​corresponding to the input information are used as correct output data. In this case, for example, the abstract expected values ​​are, for example, concrete numerical values ​​included in the expected values ​​stored in the test information table 2024 that have been abstracted into a "default numerical value." Also, the abstract expectation is <specification> 、 <number>It may be abstracted as follows.

[0061] More specifically, for example, the second trained model 2026 may use MPT-7B and then be fine-tuned using QLoRA to further improve performance. Using the pre-trained MPT-7B as a base model provides a powerful foundation capable of handling the complexities of text generation. Subsequent fine-tuning using QLoRA adapts the model to the expected behavior prediction task or existing domain, achieving more context-sensitive predictions. Note that instead of MPT-7B, other models such as T5 or Swallow from the llama family can also be used.

[0062] Furthermore, depending on the test perspective and check item, a pattern table may exist. In such cases, the second trained model 2026 is trained to output abstract expected values ​​for, for example, the test section, test perspective, check item, and test pattern. In this case, the learning data for training the second trained model 2026 uses, for example, the test section, test perspective, check item, and test pattern stored in the test information table 2024 as input data, and the abstract expected value corresponding to the input information as correct output data.

[0063] Context tuning may also be performed on the second trained model 2026. In context tuning, for example, an instruction sentence may include example answers, answer formats, etc. so that information is output from the second trained model 2026 in a format desired by the user.

[0064] When the test category, test perspective, and check items are used as input data and the abstract expected value is used as correct output data, information that can identify the company, etc. that conducted the test may be masked. The second trained model 2026 may be retrained as needed based on the test category, test perspective, check items, and expected value included in a newly added test design document.

[0065] The third trained model 2027 is a model for outputting specific expected values ​​based on information related to specifications and abstract expected values. The third trained model 2027 is, for example, referred to as an expected behavior instantiation model. Specifically, the third trained model 2027 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional learning. The additional learning involves teaching the correct answers for each task, or so-called fine tuning, to specialize the model for creating test design documents. Specifically, for example, the third trained model 2027 is trained to output specific expected values ​​for information related to test sections, test perspectives, and check items included in the specifications, and abstract expected values. In this case, the learning data for training the third trained model 2027 is, for example, information related to test sections, test perspectives, and check items included in the specifications, and abstract expected values, which are input data, and the expected values ​​stored in the test information table 2024 corresponding to the input information are used as correct output data.

[0066] The data used includes terminology specific to the software, and there are test cases in which the software's functional requirements are essential for predicting expected behavior. Since the lack of this information leads to incomplete predictions, the expected behavior concretization model incorporates information from the specification document into the abstract expected values ​​output by the second trained model 2026. Specifically, the LLM is given specification information and the generated expected behavior prediction as directives, and a process is introduced to concretize the predictions. This makes it possible to reflect specification information in abstract expected values, enabling the output of more reliable and specific expected values.

[0067] Furthermore, depending on the test perspective and check items, a pattern table may exist. In such cases, the third trained model 2027 is trained to output specific expected values ​​for, for example, information related to the test category, test perspective, check items, and pattern table included in the specifications, and abstract expected values. In this case, the learning data for training the third trained model 2027 uses, for example, information related to the test category, test perspective, check items, and pattern table included in the specifications, and abstract expected values, as input data, and the expected values ​​stored in the test information table 2024 corresponding to the input information are used as correct output data.

[0068] Context tuning may also be performed on the third trained model 2027. In context tuning, for example, example answers, answer formats, etc. are included in the instruction sentence so that information is output from the third trained model 2027 in a format desired by the user.

[0069] When information related to the test category, test perspective, and confirmation items included in the specifications, and abstract expected values ​​are used as input data and specific expected values ​​are used as correct output data, information that can identify the company, etc. that conducted the test may be masked. The third trained model 2027 may be retrained as needed based on the expected values ​​included in newly added specifications and test design documents created based on the specifications.

[0070] The additional learning may also include instruction tuning and reinforcement learning from human feedback (RLHF). Instruction tuning, for example, uses a data set consisting of various instructions and their corresponding desirable responses, and a data set consisting of various instructions and their corresponding undesirable responses, as learning data, making it possible to output information that is in line with the test engineer's intentions.

[0071] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the terminal device 10. The control unit 203 operates in accordance with the program to perform functions shown as a reception control module 2031, a transmission control module 2032, a creation assistance module 2033, and a presentation control module 2034.

[0072] The reception control module 2031 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.

[0073] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0074] The creation support module 2033 supports the user in creating a test design document. Specifically, for example, the creation support module 2033 uses a trained model at each step in test design to suggest to the user effective options for test design.

[0075] Specifically, for example, the creation support module 2033 uses the LLM system 30 when identifying test targets. For example, the creation support module 2033 transmits specification data selected by the user and an instruction statement for outputting a test division appropriate for the specification data to the LLM system 30. The LLM system 30 inputs the specification data and instruction statement transmitted from the server 20 into the LLM, and causes the LLM to output a test division appropriate for the specification data. The test division output from the LLM may be one or more. The test division output from the LLM may be assigned division 1, division 2, or division 3 as a sub-concept of the test division.

[0076] The creation support module 2033 issues an instruction statement for outputting a test section in response to, for example, a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting a test section, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the specification data. The instruction statement may also include content for specifying the output format of the test section. The creation support module 2033 may convert the specification data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format may include, for example, Markdown format.

[0077] Furthermore, for example, the creation assistance module 2033 uses the first trained model 2025 when selecting a test perspective. For example, the creation assistance module 2033 inputs a test section related to a test design document and an instruction statement for outputting a test perspective and check items corresponding to the test section into the first trained model 2025, and causes the first trained model 2025 to output the test perspective and check items corresponding to the input test section. The number of test perspectives output from the first trained model 2025 may be one or more. The number of check items associated with the test perspective output from the first trained model 2025 may be one or more.

[0078] When the first trained model 2025 outputs a test viewpoint, a check item, and an index value for the test viewpoint, the creation assistance module 2033 may determine whether to output the test viewpoint and the check item to a subsequent stage based on the index value. For example, when the index value of the test viewpoint output from the first trained model 2025 is less than a preset threshold, the creation assistance module 2033 may not output the test viewpoint and the check item associated with the test viewpoint to a subsequent stage.

[0079] If there are multiple types of index values, such as compatibility and defect risk, the test perspective and check items may be output when either one of the index values ​​exceeds a threshold, or when both index values ​​exceed a threshold. Alternatively, one index value may be given priority.

[0080] For example, the creation support module 2033 issues an instruction statement for outputting test perspectives and check items in response to a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting test perspectives and check items, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the test section. The instruction statement may also include content for specifying the output format of the test perspectives and check items.

[0081] Furthermore, the creation support module 2033 utilizes the LLM system 30 when determining the specific content of the test. For example, the creation support module 2033 transmits to the LLM system 30 the specification data, test division, test viewpoint, and check items selected by the user, as well as instructions for outputting appropriate factors and levels based on this information. The LLM system 30 inputs the specification data, test division, test viewpoint, check items, and instructions transmitted from the server 20 into the LLM, and causes the LLM to output factors and levels appropriate for the specification data, test division, test viewpoint, and check items selected by the user. The factors and levels output from the LLM may be one or more.

[0082] For example, the creation support module 2033 issues an instruction statement for outputting factors and levels in response to a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting factors and levels, and use the received text information as the instruction statement. The instruction statement may include content explaining what each piece of information to be input is. The instruction statement may also include content specifying an output format for the factors and levels. The creation support module 2033 may convert the specification data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format may include, for example, Markdown format.

[0083] Furthermore, for example, when generating a test pattern, the creation support module 2033 uses rules for generating patterns from factors and levels that are stored in advance in the storage unit 202. For example, the creation support module 2033 applies the rules to factors and levels to generate a pattern.

[0084] Furthermore, for example, when generating an expected value, the creation assistance module 2033 uses the second trained model 2026 and the third trained model 2027. For example, the creation assistance module 2033 inputs the test category, test perspective, confirmation item, and instructions for outputting an appropriate and abstract expected value for this information into the second trained model 2026, and causes the second trained model 2026 to output an appropriate and abstract expected value for the input information.

[0085] Furthermore, for example, when a pattern table is associated with a test perspective and a check item, the creation support module 2033 inputs the test category, test perspective, check item, pattern table, and an instruction statement for outputting an appropriate and abstract expected value for this information into the second trained model 2026, and causes the second trained model 2026 to output an appropriate and abstract expected value for the input information.

[0086] Furthermore, for example, the creation support module 2033 inputs information regarding the test category, test perspective, and confirmation items contained in the specification, as well as abstract expected values ​​and instructions for outputting appropriate and specific expected values ​​for this information, into the third trained model 2027, and causes the third trained model 2027 to output appropriate and specific expected values ​​for the input information.

[0087] Furthermore, for example, when a pattern table is associated with a test perspective and a check item, the creation support module 2033 inputs information related to the test category, test perspective, check item, and pattern table contained in the specification, as well as abstract expected values ​​and instructions for outputting appropriate and specific expected values ​​for this information, into the third trained model 2027, and causes the third trained model 2027 to output appropriate and specific expected values ​​for the input information.

[0088] For example, the creation assistance module 2033 issues a command statement for outputting an expected value in response to a user pressing a predetermined button. The creation assistance module 2033 may also receive text information from the user regarding a command for outputting an expected value, and use the received text information as the command statement. The command statement may include content that explains what each piece of information to be input is. The command statement may also include content that specifies the output format of the expected value.

[0089] The presentation control module 2034 controls the process of presenting information to the user. For example, the presentation control module 2034 presents information output from the trained model to the user. Specifically, for example, the presentation control module 2034 presents the test section output from the LLM system 30 to the user. The presentation control module 2034 also presents the test perspectives and check items output from the first trained model 2025 to the user. The presentation control module 2034 also presents the factors and levels output from the LLM system 30 to the user. The presentation control module 2034 also presents the patterns generated by applying predetermined rules to the factors and levels to the user. The presentation control module 2034 also presents the abstract expected values ​​output from the second trained model 2026 to the user. The presentation control module 2034 also presents the concrete expected values ​​output from the third trained model 2027 to the user.

[0090] <2 Data Structure> 4 to 9 are diagrams showing the data structures of tables stored in the server 20. Note that Figures 4 to 9 are merely examples and do not exclude data that is not listed. Furthermore, even data that is listed in the same table may be stored in separate storage areas in the storage unit 202.

[0091] Fig. 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with the user ID as a key.

[0092] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The gender is an item that stores the gender of the user. The date of birth is an item that stores the date of birth of the user. The contact information is an item that stores the contact information (e.g., telephone number, email address, etc.) of the terminal device 10 that the user has.

[0093] Fig. 5 is a diagram showing the data structure of the specification table 2022. The specification table 2022 shown in Fig. 5 is a table having columns such as client ID, project ID, creation date and time, specification name, specification data, and latest update, with specification ID as a key. Note that the information stored in Fig. 5 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0094] The specification ID is an item that stores an identifier for uniquely identifying a specification. The client ID is an item that stores an identifier for uniquely identifying a client who requires testing based on the specification. The project ID is an item that stores an identifier for uniquely identifying a project to which a function that is the target of testing based on the specification belongs. The creation date / time is an item that stores the date and time the specification was created. The specification name is an item that stores the name of the specification. The specification name may be given by extracting a portion of the content of the specification, or may be given by the creator of the specification, etc. Specification data includes data regarding detailed specifications of the function, data regarding message definitions, etc. The item "specification data" may store reference information (path) for a specification data file located in another location. The last update is an item that stores the date and time the specification data was last updated.

[0095] FIG. 6 is a diagram showing the data structure of the prompt table 2023. The prompt table 2023 shown in FIG. 6 is a table having columns such as prompt data, with a prompt ID as a key. Note that the information stored in FIG. 6 is not limited to this. For example, further information may be stored. For example, a column may be provided for storing an ID that identifies an industry. This allows the creation assistance module 2033 to select instructions appropriate for each industry, for example.

[0096] The prompt ID is an item that stores an identifier for uniquely identifying a prompt. The prompt data is an item that stores data related to the prompt, which is the instruction sentence to be input to the LLM. The data related to the prompt is, for example, text information related to the prompt. In the text information related to the prompt, for example, the content that specifies the information is left blank, and predetermined content is inserted when the prompt is used. In other words, it can be interpreted that the instruction sentence is generated by combining it with predetermined information.

[0097] The text information related to the prompt is, for example, an instruction sentence for outputting a test section based on the specification data, and includes, for example, the following content: "The input document is a specification {...}. Please output the test category for testing the functions related to this specification in the format..."

[0098] The text information related to the prompt is, for example, an instruction sentence for outputting test viewpoints and check items corresponding to the test category, and includes, for example, the following content. "The information you entered is test category {...}. Please output the test perspectives and check items corresponding to this test category in the format..."

[0099] The text information related to the prompt is, for example, an instruction to output factors and levels based on specification data, test category, test viewpoint, and confirmation item, and includes, for example, the following content: "The information you entered is the specification {...}, the test category {...} for which you would like to test this specification, the test perspective {...}, and the check items {...}. Please output the factors and levels for testing based on this information in the format of ..."

[0100] Furthermore, the text information related to the prompt is, for example, an instruction statement for outputting an abstract expected value based on the test category, test viewpoint, and confirmation item, and includes, for example, the following content: "The information you entered is the desired test category {...}, test perspective {...}, and check items {...}. Please output the abstract expectations for conducting the test based on this information in the format..."

[0101] In addition, the text information related to the prompt is, for example, an instruction statement for outputting specific expected values ​​based on the test category, test perspective, and confirmation items included in the specification and abstract expected values, and includes, for example, the following content: "The information you have entered is the information about the test category {...}, the information about the test perspective {...}, and the information about the check items {...} contained in the specification {...}. The information you have entered is also the abstract expectations for conducting the test. Please output the specific expectations for conducting the test based on this information in the format of ..."

[0102] The item "prompt data" may store a reference (path) to a prompt data file located elsewhere.

[0103] 7 to 9 are diagrams showing the data structure of the test information table 2024. The test information table 2024 includes, for example, a test design document table 20241, a pattern table 20242, and a test viewpoint table 20243. The pattern table 20242 and the test viewpoint table 20243 are tables that store elements decomposed from the test design document table 20241.

[0104] FIG. 7 is a diagram showing the data structure of the test design document table 20241. Each record of the test design document table 20241 shown in FIG. 7 has columns for file name, tag, designer ID, project ID, service ID, attribute, link, and latest update. The file name is an item that stores the name of the test design document. The tag is an item that stores information that represents the characteristics of the test. Information in the "tag" item is stored, for example, based on information set for the project related to the specification. Furthermore, the information in the "tag" item may be set arbitrarily by the user.

[0105] The designer ID is an item that stores an identifier for uniquely identifying the person who created the test design document. The project ID is an item that stores an identifier for uniquely identifying the project for which the test design document was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document belongs. Multiple types of attributes can be set in the "attribute" item, and for example, one or more attributes can be set from business type / industry, system type, process type, etc. The information in the "attribute" item is set, for example, based on the information stored in the "tag" item. Furthermore, the information in the "attribute" item may be set arbitrarily by the user.

[0106] The link is an item that stores specification data associated with the test design document. The specification data itself may be associated with the item "link", but the relevant section of the specification data may also be associated. For example, if the specification data is a spreadsheet, the relevant section of the specification data may be stored in the form of a slide, row, cell, or the like. The last update is an item that stores the date and time when the test design document was last updated.

[0107] Fig. 8 is a diagram showing the data structure of the pattern table 20242. Each record of the pattern table 20242 shown in Fig. 8 has columns for pattern name, pattern number, tag, file name, project ID, service ID, attribute, and latest update. The pattern name is an item that stores the name of the pattern. The pattern number is an item that stores a unique number assigned to the pattern. The tag is an item that stores information that is attached as a tag to the test design document that is the source of decomposition.

[0108] The file name is an item that stores the name of the test design document to be decomposed. The project ID is an item that stores an identifier for uniquely identifying the project in which the test design document to be decomposed was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document to be decomposed belongs. The last update is an item that stores the date and time when the pattern was last updated.

[0109] FIG. 9 is a diagram showing the data structure of the test perspective table 20243. Each record of the test perspective table 20243 shown in FIG. 9 has columns for test perspective, check item, tag, category 1, category 2, attribute, and latest update. The test perspective is an item for storing a test perspective. The check item is an item for storing information for identifying the test to be performed. The tag is an item for storing information attached as a tag to the test design document from which the test perspective is decomposed. Category 1 is an item for storing information about the first category attached to the test perspective. The information stored in the item "category 1" is set by the user when inputting the test perspective, for example, during the creation stage of the test design document. The information stored in the item "category 1" is set by the user, for example, by selecting from a plurality of pre-set items. The information stored in the item "category 1" is, for example, information related to the test category.

[0110] Classification 2 is an item that stores information about the second classification assigned to a test perspective. The information stored in the item "Classification 2" is set by the user, for example, when inputting a test perspective during the creation stage of a test design document. The information stored in the item "Classification 2" is set by the user, for example, by selecting from a plurality of pre-set items. The information stored in the item "Classification 2" is, for example, information related to the test division. The number of classifications stored in the test perspective table 20243 is not limited to two, and may be three or more. The attribute is an item that stores information indicating the type to which the test design document from which the test perspective is decomposed belongs. The last update is an item that stores the most recent date and time the test perspective was updated.

[0111] <3 operations> The operation of the server 20 when assisting in the creation of a test design document will now be described.

[0112] FIG. 10 is a schematic diagram showing the operation of the server 20. FIG. 11 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document.

[0113] First, a user (test engineer) who wants to create a test design document accesses the server 20. The user logs in to a service that assists in creating a test design document, provided by the server 20, for example, by entering his or her own user ID and password.

[0114] In step S11, the terminal device 10 accepts specification data designation from the user. Specifically, for example, the display 141 displays a list of specifications stored in the specification table 2022. The user selects a specification for which a test design document needs to be created from the specifications displayed in the list. After selecting a specification, the user inputs an instruction to the terminal device 10 to output a test section corresponding to the specification. The transmitting / receiving unit 192 transmits information related to the designated specification data and an instruction to output the test section to the server 20.

[0115] In step S12, the server 20 obtains a test division appropriate for the specified specification data based on the specified specification data. Specifically, for example, the creation support module 2033 reads an instruction statement from the prompt table 2023 for outputting a test division appropriate for the specification data. The creation support module 2033 outputs information about the specification data specified by the user and the read instruction statement to the LLM system 30. The LLM system 30 inputs the information about the specification data and the instruction statement transmitted from the server 20 into the LLM, causing the LLM to output a test division appropriate for the specification data. At this time, the LLM outputs, for example, text information indicating the items / functions to be tested. The LLM assigns, for example, hierarchical information such as "test division," "division 1," "division 2," "division 3," or any combination thereof to the text information. The LLM system 30 transmits the information output from the LLM to the server 20.

[0116] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test section transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test section transmitted from the LLM system 30, to the terminal device 10.

[0117] In step S13, the terminal device 10 displays a screen for creating a test design document including the test division transmitted from the LLM system 30 to the user.

[0118] 12 is a schematic diagram showing an example of a test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0119] The display area 1411 is an area where a list of specification data is displayed. In Fig. 12, the presentation control unit 193 displays one piece of specification data in the display area 1411. In Fig. 12, the presentation control unit 193 marks the square to the left of the specification data to indicate that the specification data has been specified by the user. Note that when a test design document creation screen is created for a single piece of specification data, the display area 1411 does not necessarily have to be provided.

[0120] The operation object 1412 is an object for inputting an instruction to request a proposal for a test section.

[0121] 12, the presentation control unit 193 displays text information in the display area 1413 based on the hierarchical information of the test section, section 1, section 2, and section 3.

[0122] The operation object 1414 is an object for inputting an instruction to request the proposal of a test viewpoint and confirmation items. The presentation control unit 193 displays the operation object 1414, for example, in an area associated with a predetermined test section. Specifically, for example, the presentation control unit 193 displays the operation object 1414 in association with a line displaying text information marked with section 3, which indicates that the test section is the lowest level.

[0123] 12 , the user specifies specification data in the display area 1411 and presses the operation object 1412. Then, the terminal device 10 transmits an instruction to the server 20 requesting a test section proposal. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output a test section suitable for the specification data specified by the user. The presentation control unit 193 displays text information related to the test section output for the specified specification data in the display area 1413.

[0124] In step S14, the terminal device 10 accepts the selection of a test subject. Specifically, for example, the display 141 displays a list of multiple test sections provided by the server 20. The user selects a test section for which the user wants to check test perspectives and check items from the test sections displayed in the list. The user inputs an instruction to the terminal device 10 to output test perspectives and check items corresponding to the selected test section. The transmitting / receiving unit 192 transmits information about the selected test section and an instruction to output the test perspectives and check items to the server 20.

[0125] In step S15, the server 20 outputs test perspectives and check items appropriate for the test section based on the selected test section. Specifically, for example, the creation assistance module 2033 reads, from the prompt table 2023, an instruction statement for outputting test perspectives and check items corresponding to the test section. The creation assistance module 2033 inputs the selected test section and the read instruction statement into the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items appropriate for the test section. At this time, the first trained model 2025 outputs, for example, text information representing the test perspective and text information representing the check items.

[0126] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test viewpoints and check items output from the first trained model 2025. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test viewpoints and check items output from the first trained model 2025, to the terminal device 10.

[0127] In step S16, the terminal device 10 displays a test design document creation screen including the outputted test viewpoints and check items to the user.

[0128] 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0129] The display area 1415 is an area where the test perspectives and check items for the selected test section are displayed. In Fig. 13, the presentation control unit 193 displays the test perspectives and check items output for two test sections in the display area 1415. In Fig. 13, the presentation control unit 193 displays a test section for which one set of test perspectives and check items is proposed, and a test section for which two sets of test perspectives and check items are proposed.

[0130] 12, the user presses the operation object 1414 corresponding to the test section for which the user desires test perspectives and check items. The terminal device 10 then transmits an instruction to the server 20 requesting the proposal of test perspectives and check items. In response to the instruction, the creation assistance module 2033 causes the first trained model 2025 to output test perspectives and check items suitable for the test section selected by the user. The presentation control unit 193 displays text information related to the test perspectives and check items output for the selected test section in the display area 1415.

[0131] In step S17, the terminal device 10 accepts the selection of a test perspective and a check item. Specifically, for example, the display 141 displays the test perspective and check items provided by the server 20. The user selects a test perspective and check item for which the user wants to check factors and levels from the displayed test perspectives and check items. The user inputs an instruction to the terminal device 10 to output factors and levels corresponding to the selected test perspective and check item. The transmitting / receiving unit 192 transmits information related to the selected test perspective and check item and an instruction to output the factors and levels to the server 20.

[0132] In step S18, the server 20 acquires factors and levels appropriate for the specification data specified by the user and the test division, test perspective, and check items selected by the user, based on the information. Specifically, for example, the creation support module 2033 reads instructions for outputting factors and levels appropriate for the specification data, test division, test perspective, and check items from the prompt table 2023. The creation support module 2033 outputs information regarding the specification data specified by the user, the test division, test perspective, and check items selected by the user, and the read instructions to the LLM system 30. The LLM system 30 inputs the specification data, information regarding the test division, test perspective, and check items transmitted from the server 20 into the LLM, and causes the LLM to output factors and levels appropriate for the selected test perspective and check items. At this time, the LLM outputs, for example, text information representing the factors and text information representing the levels. The LLM may also output remarks for the levels. The LLM system 30 transmits the information output from the LLM to the server 20.

[0133] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the factors and levels transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the factors and levels transmitted from the LLM system 30, to the terminal device 10.

[0134] In step S19, the terminal device 10 displays a test design document creation screen including the factors and levels transmitted from the LLM system 30 to the user.

[0135] 14 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0136] The operation object 1416 is an object for inputting an instruction to request a proposal of factors and levels.

[0137] The display area 1417 is an area where the factors and levels for the selected test perspective and confirmation item are displayed. In Fig. 14, the presentation control unit 193 displays the factors and levels for the test perspective: "display / hide" and the confirmation item: "the text is displayed according to the pattern" in the display area 1417.

[0138] 14 , the user selects the test perspective and check item for which the user desires factors and levels, and presses the operation object 1416. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of factors and levels. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output factors and levels appropriate for the specification data specified by the user, and the test category, test perspective, and check item selected by the user. The presentation control unit 193 displays text information related to the factors and levels output from the LLM system 30 in the display area 1417.

[0139] In step S110, the terminal device 10 accepts an instruction to create a test pattern. Specifically, for example, the display 141 displays a screen for creating a pattern table based on factors and levels provided by the server 20 in response to an instruction from the user. The user selects a rule for creating a test pattern displayed on the creation screen. The rule for creating a test pattern is, for example, "two-factor coverage," "full coverage," or "constraint + full coverage." When a rule is selected, the transmitter / receiver 192 transmits information about the factors and levels, information about the selected rule, and an instruction to create a test pattern to the server 20.

[0140] In step S111, the server 20 creates a test pattern from factors and levels based on the selected rule. Specifically, for example, the creation support module 2033 applies the selected rule to the factors and levels to create a test pattern. At this time, the creation support module 2033, for example, expands predetermined marks in accordance with the selected rule in a matrix with multiple columns, each row representing a factor and a level. The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the created test pattern. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the created test pattern, to the terminal device 10.

[0141] In step S112, the terminal device 10 displays a screen for creating a test design document including a test pattern to the user.

[0142] 15 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0143] The operation object 1418 is an object for selecting a rule for generating a test pattern. In Fig. 15, "bifactorial coverage" is selected as the rule.

[0144] The created test pattern is displayed in a display area 1419. In Fig. 15, the presentation control unit 193 displays a matrix containing "circles" in the display area 1419 as a test pattern.

[0145] 15, the user presses an operation object 1418 for selecting a rule for creating a test pattern. The terminal device 10 then transmits an instruction to the server 20 requesting the creation of a test pattern. The creation assistance module 2033 creates a test pattern in accordance with the instruction and the selected rule. The presentation control unit 193 displays the created test pattern in the display area 1419.

[0146] In step S113, the terminal device 10 accepts an instruction to create an expected value. Specifically, for example, the display 141 displays test perspectives and check items provided by the server 20. The user selects a test perspective and check item for which the user wants to check the expected value from the displayed test perspectives and check items. The user inputs an instruction to the terminal device 10 to output an expected value corresponding to the selected test perspective and check item. The transmitter / receiver 192 transmits information related to the selected test perspective and check item and an instruction to output the expected value to the server 20. Note that the display 141 displays factors and levels provided by the server 20 and test patterns created by the server 20, and the user may input an instruction to the terminal device 10 to output an expected value corresponding to a case defined by the test pattern.

[0147] In step S114, the server 20 outputs the expected value based on the information specified by the user. Specifically, for example, the server 20 outputs the expected value using the second trained model 2026 and the third trained model 2027.

[0148] Fig. 16 is a block diagram illustrating an example of the operation of the server 20 when outputting expected values. In the example shown in Fig. 16, for example, the creation assistance module 2033 inputs a test category selected by the user and an instruction statement for outputting test perspectives and check items corresponding to the test category to the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items suitable for the test category. This process is performed in step S15 of Fig. 11. The user selects test perspectives and check items for which they wish to output expected values ​​from the output test perspectives and check items.

[0149] The creation assistance module 2033 reads an instruction statement for outputting an abstract expected value from the prompt table 2023. The creation assistance module 2033 inputs the test category, test perspective, and confirmation item selected by the user and the instruction statement read from the prompt table 2023 into the second trained model 2026, and causes the second trained model 2026 to output an appropriate and abstract expected value for the input information.

[0150] The creation support module 2033 searches the specification data for information necessary to realize the expected values. Specifically, the creation support module 2033 searches the specification data for information related to the test to be performed by, for example, executing a predetermined search algorithm. The predetermined search algorithm is, for example, an algorithm that searches the specification data for documents containing a search query and calculates the degree of match between the searched document and the search query to search for the necessary information. Note that the search for information from the specification data is not limited to a search based on the degree of match, and other known methods may be used. The creation support module 2033, for example, executes a predetermined search algorithm to search the specification data for information related to the test classification, information related to the test perspective, and information related to the confirmation items. For example, the specification data may contain a list of processing targets, processing contents, etc., and the list may be configured to allow navigation to a page describing details of the processing targets and processing contents. The creation support module 2033, for example, executes a search algorithm to retrieve items in the list based on the degree of correspondence between words contained in the test category, test perspective, confirmation items, and abstract expectations and words contained in the specifications, and retrieves detailed information from a specified page that can be navigated to via the retrieved items.

[0151] The creation support module 2033 reads an instruction statement for outputting a specific expected value from the prompt table 2023. The creation support module 2033 inputs information related to the test category, information related to the test viewpoint, and information related to the confirmation item searched from the specification data, the abstract expected value output from the second trained model 2026, and the instruction statement read from the prompt table 2023 into the third trained model 2027, and causes the third trained model 2027 to output an appropriate and specific expected value for the input information. At this time, the third trained model 2027 outputs, for example, text information related to the expected value.

[0152] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including specific expected values ​​output from the third trained model 2027. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the expected values ​​output from the third trained model 2027, to the terminal device 10.

[0153] In step S115, the terminal device 10 displays a test design document creation screen including the output expected values ​​to the user.

[0154] 17 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0155] The operation object 14110 is an object for inputting an instruction to request a proposal of an expected value.

[0156] The display area 14111 is an area where the expected values ​​for the selected test perspective and confirmation item are displayed. In FIG. 17 , the presentation control unit 193 displays the expected value for the test perspective: "Display / Hide" and the confirmation item: "The text is displayed according to the pattern" in the display area 14111: "If the total amount of regular products is less than 2,000 yen, the message 'The total amount of the products in the cart is 'XX yen'. Add '△ yen' and get free shipping!!' will be displayed." The creation assistance module 2033 may output multiple expected values ​​from the second trained model 2026 and the third trained model 2027.

[0157] 17, the user selects the test perspective and confirmation item for which the user desires an expected value, and presses the operation object 14110. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of an expected value. In response to the instruction, the creation support module 2033 causes the second trained model 2026 to output an abstract expected value for the selected information. The creation support module 2033 causes the third trained model 2027 to output a concrete expected value based on the information searched for in the specification data and the abstract expected value. The presentation control unit 193 displays text information related to the output abstract expected value in the display area 14111.

[0158] 18 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0159] The operation object 14112 is an object for inputting an instruction to request a proposal of an expected value.

[0160] The display area 14113 is an area where the cases for which the expected values ​​are to be created are displayed. In Fig. 18, the presentation control unit 193 displays a matrix in which the factors and levels are rows and each case is a column in the display area 14113 as the target for which the expected values ​​are to be created. In Fig. 18, for example, the presentation control unit 193 displays the case where the factor is "product type" and the level is "regular products only" as the first case of the test pattern, with the factor being "total product price" and the level being "less than 2000 yen" in the matrix.

[0161] The display area 14114 is an area where the expected values ​​and the cases for which the expected values ​​were created are displayed. In Fig. 18, the presentation control unit 193 displays black circles representing the targets for which the expected values ​​are to be created and the created expected values ​​in the display area 14114. In Fig. 18, the presentation control unit 193 displays, for example, the expected values ​​created for the first case, but the expected values ​​for the other cases may also be created collectively.

[0162] 18, the user presses the operation object 14112 in the display area 14114. In response to this, the terminal device 10 transmits an instruction to the server 20 requesting the proposal of an expected value. In response to the instruction, the creation assistance module 2033 causes the second trained model 2026 to output an abstract expected value for a case defined by a test pattern. The creation assistance module 2033 causes the third trained model 2027 to output a concrete expected value based on information searched from the specification data and the abstract expected value. The presentation control unit 193 displays text information related to the output abstract expected value in the display area 14112.

[0163] As described above, in the above embodiment, the creation assistance module 2033 inputs information related to the test content to the second trained model 2026, which has been trained to output an abstract expected value, and causes the second trained model 2026 to output the abstract expected value. The creation assistance module 2033 inputs the abstract expected value and information related to the specification to the third trained model 2027, which has been trained to output a concrete expected value, and causes the third trained model 2027 to output the concrete expected value. The presentation control module 2034 presents to the user the concrete expected value output from the third trained model 2027, or both the abstract expected value and the concrete expected value. This makes it possible to output a concrete expected value by reflecting information related to the specification in the abstract expected value that is initially output.

[0164] When a specific expected value is output directly from a trained model, the learning cost tends to be high. As in the present embodiment, by outputting a final product through multiple stages, it is possible to reduce the learning cost. Furthermore, when a specific expected value is output directly from a trained model, incorrect information may be output. As in the present embodiment, by outputting a final product through multiple stages, it is possible to prevent incorrect information from being output.

[0165] Furthermore, when a specific expected value is output directly from a trained model, the correct information may not be output unless the training data contains similar information. By first outputting an abstract expected value and then outputting a specific expected value, it is possible to output a more reliable and specific expected value even if the training data does not contain similar information.

[0166] Therefore, the system according to this embodiment can reduce the effort required to create test cases.

[0167] In the above embodiment, at least one of the second trained model 2026 and the third trained model 2027 is a large-scale language model. This enables the creation assistance module 2033 to provide the user with information corresponding to input information with high accuracy.

[0168] Furthermore, in the above embodiment, the information about the specifications is information obtained by searching for information about the test from the specification data. This makes it possible to obtain, from the specification data that includes a large amount of content, information necessary and sufficient to output specific expected values ​​from the third trained model 2027. This improves the accuracy with which specific expected values ​​are output from the third trained model 2027.

[0169] <Modification> In the above embodiment, a case has been described as an example in which a specific expected value output from the third trained model 2027 is presented to the user. The server 20 according to this embodiment may also present an evaluation of the specific expected value to the user.

[0170] 19 is a block diagram illustrating an example of the operation of the server 20 when outputting an expected value. In the example shown in FIG. 19, the creation assistance module 2033 inputs the specific expected value output from the third trained model 2027 into the fourth trained model 2028 (evaluation model), and causes the fourth trained model 2028 to output an evaluation index for the specific expected value.

[0171] The storage unit 202 stores, for example, a fourth trained model 2028.

[0172] The fourth trained model 2028 is a model for evaluating the specific expected value output from the third trained model 2027. The fourth trained model 2028 is referred to as an evaluation model, for example. Specifically, the fourth trained model 2028 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has been subjected to prompt tuning. Specifically, for example, a prompt is designed to assign a four-level index value to the input expected value based on the definition of each index value and samples related to each index value. For example, the index values ​​are set as S, A, B, and C. For example, S represents employable and excellent, A represents employable, B represents unemployable but relatively good, and C represents unemployable.

[0173] For example, the creation assistance module 2033 inputs into the fourth trained model 2028 a specific expected value output from the third trained model 2027 and an instruction statement designed to assign a four-level index value to the input expected value, and causes the fourth trained model 2028 to output an index value for the input specific expected value.

[0174] FIG. 19 shows an example in which the fourth trained model 2028 evaluates the specific expected value output from the third trained model 2027, but the fourth trained model 2028 may evaluate the abstract expected value output from the second trained model 2026. The fourth trained model 2028 may also evaluate both the specific expected value output from the third trained model 2027 and the abstract expected value output from the second trained model 2026. The fourth trained model 2028 may also evaluate the specific expected value output from the third trained model 2027 based on the specific expected value and the abstract expected value output from the second trained model 2026.

[0175] The expected value may be evaluated using an existing method, such as BERTScore or ROUGE. The evaluation model in this case does not have to be, for example, LLM. BERTScore is a method for calculating the similarity between each token of a model-generated sentence and a teacher-label sentence using an embedded representation obtained by BERT. There are multiple types of ROUGE, and one method is to compare the average values ​​of ROUGE-1, ROUGE-2, and ROUGE-L. Evaluations using BERTScore or ROUGE, for example, do not completely match human evaluations in actual operation. The fourth trained model 2028 according to this embodiment performs prompt tuning on LLM, enabling it to provide evaluations closer to humans than existing evaluation methods.

[0176] In the above embodiment, a case has been described in which a specific expected value output from the third trained model 2027 is presented to the user. The server 20 according to this embodiment may regenerate the output specific expected value according to the evaluation index.

[0177] Figure 20 is a block diagram illustrating an example of the operation of the server 20 when outputting an expected value. In the example shown in Figure 20, the creation assistance module 2033 inputs the specific expected value output from the third trained model 2027 and the evaluation index value output from the fourth trained model 2028 into the fifth trained model 2029 (the third trained model for expected values, the regenerated model), and causes the fifth trained model 2029 to output a specific expected value regenerated so as to increase the index value.

[0178] The storage unit 202 stores, for example, a fifth trained model 2029.

[0179] The fifth trained model 2029 is a model for outputting a specific expected value with a higher evaluation based on a specific expected value and an evaluation index value for the specific expected value. The fifth trained model 2029 is, for example, referred to as a regenerative model. Specifically, the fifth trained model 2029 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional learning. In the additional learning, the correct answer for each task is taught, so-called fine tuning, to specialize the model for creating test design documents. Specifically, for example, the fifth trained model 2029 is trained to output a specific expected value with an improved index value for a predetermined index value and a specific expected value. The fifth trained model 2029 may be generated for each index value, for example. That is, for example, the fifth trained model 2029 is trained to output a specific expected value of index value S for index value B and the specific expected value of index value B. Furthermore, for example, the fifth trained model 2029 is trained to output a specific expected value of the index value S for an index value C and a specific expected value of the index value C.

[0180] In this case, the learning data for training the fifth trained model 2029 has, for example, the expected value of a predetermined index value as input data, and the expected value corrected to make the expected value the index value S as correct output data. In other words, for example, the learning data for training the fifth trained model 2029 has, as input data, the expected value of index value B, and the expected value corrected to make the expected value the index value S as correct output data. Also, for example, the learning data for training the fifth trained model 2029 has, as input data, the expected value of index value C, and the expected value corrected to make the expected value the index value S as correct output data.

[0181] It is not necessary to prepare a fifth trained model 2029 for each index value. For example, an evaluation may be set for each dataset, and fine-tuning may be performed with the evaluation included in the weight. This enables the fifth trained model 2029 to output an expected value of a specific index value based on the expected value of the specific index value.

[0182] For example, the creation assistance module 2033 determines whether the index value output from the fourth trained model 2028 is B or C (an evaluation indicating rejection). If the index value is S or A, the creation assistance module 2033 does not input the specific expected value output from the third trained model 2027 to the fifth trained model 2029. If the index value is B or C, the creation assistance module 2033 inputs the index value B or C, the specific expected value at which the index value was output, and an instruction statement for outputting a higher-evaluated expected value based on the specific expected value to the fifth trained model 2029, and causes the fifth trained model 2029 to output the higher-evaluated specific expected value.

[0183] 20 shows an example in which the fourth trained model 2028 evaluates the specific expected value output from the third trained model 2027, but the fourth trained model 2028 may also evaluate the abstract expected value output from the second trained model 2026. Furthermore, the fourth trained model 2028 may also evaluate both the specific expected value output from the third trained model 2027 and the abstract expected value output from the second trained model 2026.

[0184] 20 shows an example in which the fifth trained model 2029 outputs a more highly evaluated specific expected value based on the specific expected value output from the third trained model 2027, but the fifth trained model 2029 may output a more highly evaluated abstract expected value based on the abstract expected value output from the second trained model 2026. Furthermore, the fifth trained model 2029 may output a more highly evaluated specific expected value and an abstract expected value based on both the specific expected value output from the third trained model 2027 and the abstract expected value output from the second trained model 2026.

[0185] In this way, by outputting highly evaluated specific expected values ​​from the fifth trained model 2029, expected values ​​that are estimated to be adoptable are presented to the user. This reduces the need for the user to redo the creation of expected values ​​and to correct the presented expected values, thereby reducing the effort required of the user when creating expected values.

[0186] In the above embodiment, a case has been described in which a test case is created by a user selecting information output by a trained model. However, the information output by the trained model does not necessarily have to be adopted as is. For example, a user may modify the information output from the trained model. That is, a user may modify and adopt the test section output from the LLM system 30. A user may also add a new test section and adopt the added test section. A user may also modify and adopt the test perspectives and check items output from the first trained model 2025. A user may also add new test perspectives and check items and adopt the added test perspectives and check items. A user may also modify and adopt the factors and levels output from the LLM system 30. A user may also add new factors and levels and adopt the added factors and levels. A user may also modify a test pattern created based on the factors and levels. A user may also add a new test pattern and adopt the added test pattern. A user may also modify and adopt the expected value output from the third trained model 2027. The user may also create a new expected value and adopt the created expected value.

[0187] In addition, in the above embodiment, a case is described in which information of a type expected to be input is input to a trained model, and the trained model is caused to output predetermined information. However, the information input to the trained model may be part of the information expected to be output. In other words, a user may be responsible for part of creating the predetermined elements, and the trained model may perform the rest.

[0188] For example, when setting a test section, the user creates an upper-level test section, such as test section 1, and instructs the server 20 to output a lower-level test section. The creation support module 2033 outputs the specification data, the test section created by the user, and an instruction statement for outputting the lower-level test section to the LLM system 30. The LLM system 30 causes the LLM to output the lower-level test section and transmits the output information to the server 20.

[0189] Also, for example, when setting test perspectives and check items, the user creates at least one test perspective and check item and instructs the server 20 to output other test perspectives and check items. The creation assistance module 2033 outputs the test category, the test perspectives and check items created by the user, and an instruction statement for outputting the other test perspectives and check items to the first trained model 2025. The first trained model 2025 outputs the other test perspectives and check items.

[0190] Furthermore, for example, when setting factors and levels, the user creates at least one factor and level and instructs the server 20 to output other factors and levels. The creation support module 2033 outputs to the LLM system 30 instruction statements for outputting other appropriate factors and levels based on the specification data, test category, test viewpoint, check item, factors and levels created by the user, and this information. The LLM system 30 causes the LLM to output the factors and levels and transmits the output information to the server 20.

[0191] Also, for example, when setting an expected value, the user creates at least one expected value and instructs the server 20 to output other expected values. The creation assistance module 2033 outputs to the second trained model 2026 the test section, test perspective, check items, expected values ​​created by the user, and instructions for outputting other appropriate and abstract expected values ​​for this information. The second trained model 2026 outputs other abstract perspectives. The creation assistance module 2033 also outputs to the third trained model 2027 information related to the test section, test perspective, and check items included in the specification, the abstract expected values, expected values ​​created by the user, and instructions for outputting other appropriate and concrete expected values ​​for this information. The third trained model 2027 outputs other concrete perspectives.

[0192] In the above embodiment, the LLM system 30 is configured to output the test divisions, factors, and levels. However, the test divisions, factors, and levels are not limited to being output from the LLM system 30, and may be output from trained models that have undergone additional training on the LLM, such as the first trained model 2025 and the second trained model 2026. Additional training, for example, involves fine-tuning, which teaches the correct answer for each task, to specialize in the creation of test design documents. Specifically, for example, by training a test division appropriate for a specification as the correct answer, a test division appropriate for the input specification data is output in response to input of a prompt for outputting the specification data and a test division appropriate for the specification data. Furthermore, for example, by training a factor and level corresponding to the specification data, test division, test perspective, and check item as the correct answer, a factor and level appropriate for the input information is output in response to input of a prompt for outputting the specification data, test division, test perspective, check item, and a factor and level corresponding to this information.

[0193] Furthermore, in the above embodiment, a case has been described in which various information is input to the trained model. However, the information input to the trained model is not necessarily limited to the above. For example, in the above embodiment, the check items are input to the trained model together with the test viewpoints. However, if the check items are not included in the training data, the check items do not necessarily have to be input to the trained model.

[0194] Furthermore, in the above embodiment, for example, the creation assistance module 2033 is configured to output test perspectives and check items that are plausibly set among the test perspectives and check items output from the first trained model 2025. However, the test perspectives and check items output from the first trained model 2025 are not limited to test perspectives and check items that are plausibly set. For example, the test information table 2024 stores the results of tests conducted for the set test perspectives and check items. Then, as learning data for training the first trained model 2025, for example, the test division stored in the test information table 2024 is used as input data, and the test perspectives, check items, and information on defects as test results stored in the test information table 2024 that correspond to the input test division are used as correct output data. As a result, the first trained model 2025 outputs a test perspective that is estimated to be appropriate for the input test division, at least one check item associated with the test perspective, and information on defects. This makes it possible to identify test perspectives and check items where many defects are likely to occur, thereby improving test efficiency. Note that the first trained model 2025 may be configured to preferentially output test perspectives and check items where many defects are likely to occur from among test perspectives estimated to be appropriate for the input test category and at least one check item associated with the test perspectives.

[0195] Furthermore, the first trained model 2025 may be trained to output a test perspective estimated to be appropriate for an input test category and at least one check item associated with the test perspective, as well as to output an index value representing the defect occurrence rate for the output test perspective. Furthermore, the first trained model 2025 may be trained to output a test perspective and check items, as well as to output an index value representing the defect occurrence rate for the output test perspective and check item.

[0196] Furthermore, in the above embodiment, the user can select, modify, or add to the suggestions output by the trained model. The creation assistance module 2033 stores, for example, in the test information table 2024, items set through modifications or additions by the user. The creation assistance module 2033 also stores, for example, in the test information table 2024, items proposed by the trained model and selected by the user. When fine-tuning the trained model, the server 20 includes in the training data the history of the adoption of the item (whether it was a selected item, a modified item, or an added item). This allows the trained model to prioritize output of content created by the user, making it possible to output information desired by the user with high accuracy.

[0197] Furthermore, in the above embodiment, an example has been described in which, when proposing test perspectives and check items, a test category is input to the first trained model 2025. However, if specification data is included in the learning data, the specification data may be input to the first trained model 2025 in addition to the test category.

[0198] In the above embodiment, the server 20 has the prompt table 2023. However, the server 20 does not necessarily have to have the prompt table 2023. For example, the server 20 may have a predetermined link module, and the link module may include a prompt template. The link module is, for example, an API (Application Programming Interface). Note that the link module may not be provided within the server 20, but may be provided on a server different from the server 20.

[0199] Specifically, for example, in step S12, the creation support module 2033 outputs information about the specification data specified by the user to the linking module. The linking module creates an instruction statement by combining information held as a template with information about the specification data. The linking module outputs the information about the specification data specified by the user and the created instruction statement to the LLM system 30.

[0200] Also, for example, in step S15, the creation assistance module 2033 outputs information about the test category selected by the user to the collaboration module. The collaboration module creates an instruction sentence by combining information held as a template with information about the test category. The collaboration module inputs the information about the test category selected by the user and the created instruction sentence into the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items appropriate for the test category.

[0201] Also, for example, in step S18, the creation support module 2033 outputs the specification data specified by the user and information regarding the test division, test perspective, and check items selected by the user to the collaboration module. The collaboration module creates an instruction statement by combining information held as a template with the specification data, test division, test perspective, and check items. The collaboration module outputs the specification data specified by the user, information regarding the test division, test perspective, and check items selected by the user, and the created instruction statement to the LLM system 30.

[0202] Also, for example, in step S114, the server 20 outputs an appropriate and abstract expected value for the selected information based on the test section, test perspective, and check items selected by the user. Specifically, for example, the creation assistance module 2033 outputs the test section, test perspective, and check items selected by the user to the linking module. The linking module creates an instruction statement by combining information held as a template with the test section, test perspective, and check items selected by the user. The linking module inputs the test section, test perspective, and check items selected by the user and the created instruction statement into the second trained model 2026, and causes the second trained model 2026 to output an appropriate and abstract expected value for the input information.

[0203] Furthermore, the server 20 outputs appropriate and specific expected values ​​based on information related to the test division, test perspective, and check items included in the specifications, and abstract expected values. Specifically, for example, the creation assistance module 2033 outputs information related to the test division, test perspective, and check items included in the specifications, and abstract expected values, to the linking module. The linking module creates an instruction statement by combining information held as a template, information related to the test division, test perspective, and check items included in the specifications, and abstract expected values. The linking module inputs the information related to the test division, test perspective, and check items included in the specifications, the abstract expected values, and the created instruction statement into the third trained model 2027, and causes the third trained model 2027 to output appropriate and specific expected values ​​for the input information.

[0204] In the above embodiment, the LLM of the LLM system 30 outputs the test section and also outputs the factors and levels, but this is not limited to this. The LLM that outputs the test section and the LLM that outputs the factors and levels may be different LLMs.

[0205] In addition, in the above embodiment, the case where the multiple trained models are LLMs has been described as an example. However, the multiple trained models according to this embodiment may be trained models that are not LLMs.

[0206] In the above embodiment, an example of the mode in which the creation support module 2033 is provided in the server 20 has been described, but the present invention is not limited to this mode, and the terminal device 10 may be configured to have the functions of the creation support module 2033. In this case, for example, some or all of the information stored in the storage unit 202 may be stored in the terminal device 10.

[0207] <4 Basic computer hardware configuration> 21 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main memory device 92, an auxiliary memory device 93, and a communication IF (interface) 99. These are electrically connected to one another by a bus.

[0208] The processor 91 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0209] The main storage device 92 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0210] The auxiliary storage device 93 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0211] The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0212] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0213] <Basic functional configuration of computer 90> A description will be given of the functional configuration of a computer realized by the basic hardware configuration of a computer 90 shown in Fig. 21. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0214] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0215] The control unit is realized by the processor 91 reading various programs stored in the auxiliary storage device 93, expanding them in the main storage device 92, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that processes information.

[0216] The storage unit is realized by a main storage device 92 and an auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Furthermore, the processor 91 can allocate a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 in accordance with the programs. Furthermore, the control unit can cause the processor 91 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0217] A database refers to a relational database, which manages data sets called tables, which are structured by rows and columns, by relating them to each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated. Usually, each table has a column set as a key for uniquely identifying a record, but setting a key to a column is not essential. The control unit can cause the processor 91 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

[0218] The communication unit is realized by the communication IF 99. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 91 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0219] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0220] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0221] <Additional Notes> The matters described in the above embodiments will be supplemented below. (Appendix 1) A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the following steps: inputting information related to test content into a first trained model that has been trained to output an abstract expected value, and causing the first trained model to output the abstract expected value; inputting information related to the abstract expected value and specifications into a second trained model that has been trained to output a concrete expected value, and causing the second trained model to output a concrete expected value; and presenting to a user the concrete expected value output from the second trained model, or both the abstract expected value and the concrete expected value. (Appendix 2) The program according to claim 1, wherein at least one of the first trained model and the second trained model is a large-scale language model. (Appendix 3) A program described in (Appendix 1) or (Appendix 2) in which, in the step of outputting specific expected values, the information about the specification is information obtained by searching for information about the test from the specification data. (Appendix 4) A program described in any of (Appendix 1) to (Appendix 3), which causes a processor to execute a step of evaluating abstract expectations, concrete expectations, or both, and in a presentation step, presents the results of the evaluation to a user. (Appendix 5) A program described in Appendix 4, which causes a processor to execute a step of inputting a specific expected value output from the second trained model into a third trained model that has been trained to output an expected value that is estimated to have a high evaluation based on the specific expected value, and causing the third trained model to output an expected value with a higher evaluation. (Appendix 6) The program according to claim 5, wherein in the evaluating step, an abstract expectation, a concrete expectation, a more highly evaluated expectation, or at least two or more of these expectation values ​​are evaluated. (Appendix 7) A program described in (Appendix 5) or (Appendix 6), in which a processor is caused to execute a step of determining whether the result of the evaluation is an evaluation indicating rejection, and in a step of outputting an expected value with a higher evaluation, if the result of the evaluation is an evaluation indicating rejection, the evaluation indicating rejection and the expected value given the evaluation are input into a third trained model, and an expected value estimated to result in an evaluation indicating acceptance is output from the third trained model. (Appendix 8) A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 7). (Appendix 9) An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any of the inventions according to (Appendix 1) to (Appendix 7). (Appendix 10) A system comprising means for executing all steps performed in any of the inventions according to (Appendix 1) to (Appendix 7). [Explanation of symbols]

[0222] 1. System 10...Terminal device 120…Communications Department 13...Input device 131...Touch-sensitive devices 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output interface 25…Memory 26…Storage 29...Processor 30...LLM System< / number> < / specification>

Claims

1. A program to be executed by a computer having a processor and a memory, the program causing the processor to: A step of inputting information related to test content into a first trained model trained to output an abstract expected value and causing the first trained model to output an abstract expected value; A step of inputting the abstract expected value and information related to the specification into a second trained model trained to output a specific expected value, and outputting the specific expected value from the second trained model; presenting to a user the specific expected value or both the abstract expected value and the specific expected value output from the second trained model; A program that executes the following.

2. The program according to claim 1 , wherein at least one of the first trained model and the second trained model is a large-scale language model.

3. 2. The program according to claim 1, wherein in the step of outputting the specific expected value, the information about the specification is information obtained by searching for information about a test from specification data.

4. causing the processor to evaluate the abstract expectations, the concrete expectations, or both; 2. The program according to claim 1, wherein the step of presenting the results of the evaluation is performed to the user.

5. The program of claim 4, which causes the processor to execute a step of inputting a specific expected value output from the second trained model into a third trained model that has been trained to output an expected value that is estimated to have a high evaluation based on a specific expected value, and causing the third trained model to output an expected value with a higher evaluation.

6. 6. The program according to claim 5, wherein the evaluating step evaluates the abstract expected value, the concrete expected value, the higher expected value, or at least two or more of these expected values.

7. causing the processor to execute a step of determining whether the result of the evaluation is an evaluation indicating rejection; 6. The program according to claim 5, wherein, in the step of outputting the expected value of the higher evaluation, if the result of the evaluation is an evaluation indicating rejection, the evaluation indicating rejection and the expected value given the evaluation are input into the third trained model, and an expected value estimated to result in an evaluation indicating adoption is output from the third trained model.

8. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 7.

9. 8. An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any one of the inventions according to claims 1 to 7.

10. A system comprising means for executing all steps performed in the invention according to any one of claims 1 to 7.

Images