Program, method, information processing device and system

Abstract

To customize debugging processing using a learned model for every client while reducing labor.SOLUTION: A program causes a computer including a processor and a memory to execute processing. The program causes the processor to execute the steps of: acquiring test knowledge related to a customer; adjusting, on the basis of the acquired test knowledge, processing using any learned model out of a plurality of learned models being used when creating a test case on the basis of specifications, the learned models being learned to output a prescribed element constituting the test case in response to input of data related to the specifications; and providing processing using the adjusted learned model as processing for the customer who has provided the test knowledge.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] To ensure the quality of software, its functionality is tested before it is released. Tests are designed based on the software specifications to ensure the desired quality is achieved within the development schedule and budget set by the software development company.

[0003] Patent Document 1 describes a technique that makes it possible to easily perform software testing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-129520 Summary of the Invention [Problem to be solved by the invention]

[0005] The test support system described in Patent Document 1 comprises a model storage unit that stores a learning model that uses screen definition information of a system as input data and is trained using at least one of test cases, test programs, and test data as training data; a screen definition input unit that accepts input of the screen definition information of the system to be tested; a generation unit that provides the screen definition information to the learning model to generate at least one of the test cases, test programs, and test data; and an output unit that outputs the generated at least one of the test cases, test programs, and test data.

[0006] However, Patent Document 1 uses "a learning model in which the system's screen definition information is used as input data and at least one of test cases, test programs, and test data is used as training data," and creating a trained model for each client requires a lot of effort.

[0007] The objective of the present disclosure is to customize debugging processes using trained models for each client while reducing the effort required. [Means for solving the problem]

[0008] A program to be executed by a computer having a processor and a memory, the program causes the processor to execute the following steps: acquiring test knowledge related to a customer; adjusting, based on the acquired test knowledge, processing using one of multiple trained models used when creating test cases based on specifications, the trained models being trained to output predetermined elements that constitute test cases in response to input data related to the specifications; and providing the processing using the adjusted trained model as processing for the customer that provided the test knowledge. [Effects of the Invention]

[0009] According to the present disclosure, debugging processes using trained models can be customized for each client while reducing the effort required. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Figure 2] 2 is a block diagram illustrating an example of the configuration of a terminal device 10 shown in FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of a server 20. [Figure 4] FIG. 2 is a diagram showing the data structure of a user information table 2021. [Figure 5]FIG. 20 is a diagram showing the data structure of a specification table 2022. [Figure 6] FIG. 10 is a diagram showing the data structure of a prompt table 2023. [Figure 7] FIG. 2 is a diagram showing the data structure of a test design document table 20241. [Figure 8] FIG. 20 is a diagram showing the data structure of a pattern table 20242. [Figure 9] FIG. 10 is a diagram showing the data structure of a test viewpoint table 20243. [Figure 10] 3 is a schematic diagram illustrating the operation of the server 20. FIG. [Figure 11] 10 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document. FIG. [Figure 12] 10 is a schematic diagram showing an example of a test design document creation screen displayed on a display 141. FIG. [Figure 13] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 14] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 15] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 16] 10 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. FIG. [Figure 17] 10 is a flowchart showing an example of the operation of the server 20 when adjusting processing using a trained model. [Figure 18] FIG. 10 is a conceptual diagram for explaining parameter tuning by the adjustment module 2035. [Figure 19] FIG. 10 is a conceptual diagram for explaining parameter tuning by the adjustment module 2035. [Figure 20] FIG. 10 is a diagram illustrating another example of the functional configuration of the server 20 when RAG is adopted. [Figure 21]FIG. 10 is a block diagram showing another example of the overall configuration of the system 1. [Figure 22] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. The names and functions of the components are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0012] <Summary> The system according to this embodiment uses trained models at each step of test case creation based on specifications. The system acquires test knowledge related to the customer. The system adjusts, based on the acquired test knowledge, a process using a trained model that has been trained to output specific elements that make up the test case, from among multiple trained models used when creating test cases based on specifications. The system provides the adjusted process as a process for the customer that provided the test knowledge.

[0013] Specifically, for example, in this embodiment, the system uses a trained model to output a test category in the step of identifying test subjects. The system uses a trained model to output test viewpoints and check items in the step of selecting test viewpoints. The system uses a trained model to output a test pattern in the step of determining specific test content. The system uses a trained model to output an expected value of the test in the step of determining specific test content. For example, large language models (LLMs) may be used as the trained model.

[0014] The system may, for example, adjust a process using a trained model that outputs a test section based on the acquired test knowledge. The system may also, for example, adjust a process using a trained model that outputs a test perspective (and check items) based on the acquired test knowledge. The system may also, for example, adjust a process using a trained model that outputs a test pattern (factors and levels) based on the acquired test knowledge. The system may also, for example, adjust a process using a trained model that outputs an expected value based on the acquired test knowledge.

[0015] In this embodiment, the test section may be expressed by other expressions. For example, the test section may use expressions that represent the target of the test, such as test target items or test target functions. The test perspective may also be expressed by other expressions. For example, the test perspective may use expressions that represent the content of the test, such as the viewpoint (idea) on which the test is based. The check items may also be expressed by other expressions. For example, the check items may use expressions that represent information for identifying the test to be performed.

[0016] <1 Overall system configuration> Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

[0017] 1 shows an example in which the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two. The number of terminal devices 10 included in the system 1 may be one, or three or more.

[0018] 1 shows an example in which the system 1 includes one LLM system 30, but the number of LLM systems 30 included in the system 1 is not limited to one. The number of LLM systems 30 included in the system 1 may be two or more.

[0019] In this embodiment, a collection of multiple devices may be considered as one server. The allocation of multiple functions required to realize the server 20 according to this embodiment to one or more pieces of hardware may be determined appropriately in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20. Specifically, for example, a server having a function for creating test cases based on specifications and a server on which a trained model is built may be different devices.

[0020] The terminal device 10 shown in Fig. 1 is an information processing device used by a user who uses a service provided by the server 20. For example, the terminal device 10 is an information processing device operated by a test designer who creates a test design document using the test design document creation support provided by the server 20. The terminal device 10 is realized by, for example, a desktop personal computer (PC), a laptop PC, a head-mounted display, or the like. The terminal device 10 may also be a portable computer such as a smartphone or a tablet terminal.

[0021] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.).

[0022] The server 20 is an information processing device that provides a service to support the creation of a test design document. For example, the server 20 uses a plurality of trained models to support the creation of a test design document by a user.

[0023] The server 20 is, for example, an information processing device realized by a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a user and as an interface with an output device for outputting information to the user.

[0024] The LLM system 30 is a system built with a large-scale artificial intelligence model (LLM) used in the field of natural language processing (NLP), for example. By learning from large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns in human language and effectively perform natural language generation (NLG) tasks.

[0025] LLMs are used in many NLP tasks, such as generating answers to specific questions, automatic sentence generation, text summarization, translation, and sentiment analysis. They can also be used in a variety of applications, including education, entertainment, customer service, and product development. Examples of LLMs include: GPT-4 (registered trademark) (OpenAI) PaLM2 (registered trademark) (Google Inc.) StableLM (StableAI) Llama 2 (Meta)

[0026] The LLM system 30 inputs test-related information and prompts output from the server 20 into the LLM, and causes the LLM to output answers based on the input text data and prompts. The LLM system 30 transmits the answers output from the LLM to the server 20.

[0027] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0028] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.

[0029] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0030] The input device 13 is a device for inputting instructions or information by a user operating the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like, which inputs instructions by touching an operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, or the like. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

[0031] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0032] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0033] The camera 160 is a device that receives light with a light receiving element and outputs the light as an image capturing signal.

[0034] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0035] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0036] The user information 181 includes, for example, information about the user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, contact information, and the like.

[0037] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating in accordance with the program.

[0038] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131 or the like.

[0039] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. For example, the operation reception unit 191 analyzes the received voice signal and extracts a predetermined noun, thereby acquiring an instruction from the user.

[0040] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits information input by a user or instructions from a user to the server 20. In addition, the transmitting / receiving unit 192 receives information provided by the server 20.

[0041] The presentation control unit 193 controls the output device 14 to present information provided from the server 20 to the user. Specifically, for example, the presentation control unit 194 causes the information transmitted from the server 20 to be displayed on the display 141. In addition, the presentation control unit 194 causes the information transmitted from the server 20 to be output from the speaker 172.

[0042] <1.2 Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0043] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0044] The storage unit 202 includes, for example, a user information table 2021, a specification table 2022, a prompt table 2023, a test information table 2024, a client test information table 2027, and a plurality of trained models. The plurality of trained models includes, for example, a trained model used in at least one step of test design.

[0045] Specifically, for example, the multiple trained models include a first trained model 2025 used when selecting test perspectives. The multiple trained models also include a second trained model 2026 used when determining the specific content of a test. The multiple trained models also include a first trained model for customer 20251, which is a first trained model constructed for a specific customer. In this embodiment, for example, an LLM constructed in the LLM system 30 is used when identifying test subjects and when determining the specific content of a test.

[0046] The information stored in the storage unit 202 is not limited to these.

[0047] The user information table 2021 is a table that stores information about users, and will be described in detail later.

[0048] The specification table 2022 is a table that stores information related to specifications. Details will be described later. In this embodiment, the specification is data that describes the specifications of the functions of the test target. A test designer, for example, creates a test design document while referring to the specification. The specification includes, for example, an area where detailed specifications of the functions are stored and an area where message definitions are stored. If the specification is created using spreadsheet software, these areas are realized, for example, on different sheets. The detailed specifications of the functions include, for example, the specification name, object type, requirements, and control.

[0049] The prompt table 2023 is a table that stores information about prompts (instructions), and will be described in detail later.

[0050] The test information table 2024 is a table that stores information accumulated in various past debugging processes. For example, the test information table 2024 stores test designs created in various past debugging processes and elements decomposed from these test designs. That is, the test information table 2024 is a table that generalizes accumulated test information and shares it as shared knowledge. The elements decomposed from the test designs represent, for example, information for specifying required tests and information in which specific test cases are set. In the test information table 2024, for example, identifiers representing attributes are assigned to the stored test designs and elements. Multiple types of attributes can be set, such as business type / industry, system type, and process type. Using identifiers representing attributes makes it possible to classify test designs or elements according to the attributes.

[0051] In this embodiment, the test design document includes, for example, meta information, information about the test to be performed, and a pattern table. The meta information includes, for example, the title of the test design document, the project name, the product name, the control number, the version, the creation date, the creator, the last update date, the updater, and other information. Note that the information included in the meta information is not limited to these. Any of these pieces of information may be absent, or other information may be included.

[0052] Information about the tests to be conducted plays the role of, for example, a so-called basic design document that defines the general framework of the test. Information about the test includes, for example, "test target items / target functions." "Test target items / target functions" represent, for example, the items / functions that are the subject of the test. "Test target items / target functions" can also be referred to as test categories. Test categories may include sub-concepts such as major functions (category 1), medium functions (category 2), and minor functions (category 3).

[0053] Information about a test includes, for example, a test perspective. A test perspective represents, for example, the standpoint (idea) on which a test is based. A test perspective is associated with a test section. An arrangement of test perspectives, for example, specifies the order of tests. A test perspective is associated with a check item. A check item represents information for identifying a test to be performed.

[0054] The information about the test includes, for example, an expected value. The expected value represents, for example, a behavior or state that is expected to occur during the check. Note that the information about the test is not limited to these. Any of these pieces of information may be absent, or other information may be included.

[0055] The pattern table defines, as patterns, for example, the content to be confirmed, the conditions for the behavior manifested by the confirmation process, and specific test cases that combine the confirmation content and conditions. In the pattern table, for example, at least one pattern is set according to the function to be inspected. In the pattern table, the content to be confirmed is defined, for example, as factors and levels. The factors represent, for example, factors that can affect changes in the state of a computer. The levels represent, for example, specific elements of the factors. In the pattern table, the conditions for the behavior manifested by the confirmation process are defined, for example, by conditions stored as expected values. In the pattern table, specific test cases are defined, for example, using a matrix associated with the factors and levels.

[0056] The test design document is broken down into test perspectives, for example, as information for specifying required tests, and into patterns, for example, as information for specifying specific test cases.

[0057] The client test information table 2027 is a table that stores information accumulated during debugging performed by a customer. For example, the client test information table 2027 is a table that stores a test design document created during debugging performed by a customer, elements decomposed from this test design document, and test results. That is, the client test information table 2027 is a table that stores, for example, information about tests performed by a customer as the customer's test knowledge. The elements decomposed from the test design document represent, for example, information for specifying required tests and information in which specific test cases are set. The data stored in the client test information table 2027 may have the same data structure as the test information table 2024, or may have a data structure specific to the customer.

[0058] The client test information table 2027 is stored, for example, in a storage area set up for the corresponding client (a client-dedicated environment). In other words, the client test information table 2027 is stored in a storage area different from that of the test information table 2024. In other words, information about tests performed by clients is stored in a table (database) different from that of the test information table 2024.

[0059] The client test information table 2027 stores, for example, information provided by a customer and accumulated in debugging processes previously performed by the customer. The client test information table 2027 also stores information accumulated by debugging processes based on a test design document created using a service provided by the control unit 203. The client test information table 2027 may store only information output in processing using the first trained model for customer 20251, out of the information output in the debugging processes of the customer. In other words, at least a portion of the information output in the debugging processes of the customer is not stored in the test information table 2024, but is stored only in the client test information table 2027.

[0060] The first trained model 2025 is a model for outputting a test perspective estimated to be appropriate based on a test section and at least one check item associated with the test perspective. Specifically, the first trained model 2025 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional training. The additional training involves teaching the correct answer for each task, or so-called fine tuning, to specialize the model for creating test design documents. Specifically, for example, the first trained model 2025 is trained to output a test perspective estimated to be appropriate for an input test section and at least one check item associated with the test perspective. In this case, the learning data for training the first trained model 2025 is, for example, the test section stored in the test information table 2024 as input data, and the test perspective and check item stored in the test information table 2024 that correspond to the input test section as correct output data. The information input as input data may be a major function (category 1), a medium function (category 2), or a minor function (category 3) as sub-concepts included in the test category.

[0061] Context tuning may also be performed on the first trained model 2025. In context tuning, for example, an example answer, an answer format, etc. are included in the instruction sentence so that information is output from the first trained model 2025 in a format desired by the user.

[0062] The first trained model 2025 may be retrained at a predetermined timing based on the test category, test perspective, and confirmation items included in the test design document newly added to the test information table 2024.

[0063] The first trained model 2025 may be trained to output test perspectives estimated to be suitable for an input test section and at least one check item associated with the test perspectives, as well as to output an index value indicating the suitability of the output test perspectives. The first trained model 2025 may also be trained to output test perspectives and check items, as well as to output an index value indicating the suitability of the output test perspectives and check items. Depending on the test design document, different test perspectives and check items may be set even for the same test section. The index value is output, for example, by training the first trained model 2025 with learning data in which the test perspectives and check items associated with the test section vary.

[0064] The first trained model 2025 may be trained to output a test perspective estimated to be appropriate for an input test section and at least one check item associated with the test perspective, as well as to output an index value representing the risk of a malfunction occurring in the output test perspective. The first trained model 2025 may also be trained to output a test perspective and check item, as well as to output an index value representing the risk of a malfunction occurring in the output test perspective and check item. For example, the test information table 2024 stores test results, such as the occurrence of a malfunction, for each test perspective, etc. The learning data for training the first trained model 2025 may, for example, use the test section stored in the test information table 2024 as input data, and use the test perspective, check items, and test results stored in the test information table 2024 corresponding to the input test section as correct output data. As a result, an index value representing the malfunction risk is output along with the test perspective and check items.

[0065] The first trained model for customer 20251 is a first trained model constructed for a customer. The first trained model for customer 20251 may be the same model as the first trained model 2025, or may be adjusted for the customer by the adjustment module 2035. The first trained model for customer 20251 is stored, for example, in a storage area set for the corresponding customer. For example, when a customer for whom a dedicated trained model has been constructed uses a service, the control unit 203 uses the first trained model for customer 20251 instead of the first trained model 2025.

[0066] The second trained model 2026 is a model for outputting expected values ​​that are estimated to be appropriate based on specification data, test divisions, test perspectives, check items, and test patterns. Specifically, the second trained model 2026 is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional training. The additional training involves teaching the correct answers for each task, or so-called fine tuning, to specialize the model in creating test design documents. Specifically, for example, the second trained model 2026 is trained to output expected values ​​that are estimated to be appropriate for the input specification data, test divisions, test perspectives, check items, and test patterns. In this case, the learning data for training the second trained model 2026 is, for example, the test divisions, test perspectives, check items, and test patterns stored in the test information table 2024, which are used as input data, and the expected values ​​stored in the test information table 2024 that correspond to the input information are used as correct output data. In addition, depending on the test perspective and check items, a pattern table may not exist. In such a case, the second trained model 2026 is trained to output expected values ​​that are estimated to be appropriate for, for example, input specification data, test category, test perspective, and check item. In this case, the learning data for training the second trained model 2026 is, for example, the test category, test perspective, and check item stored in the test information table 2024 as input data, and the expected value stored in the test information table 2024 that corresponds to the input information is used as correct output data.

[0067] Context tuning may also be performed on the second trained model 2026. In context tuning, for example, an instruction sentence may include example answers, answer formats, etc. so that information is output from the second trained model 2026 in a format desired by the user.

[0068] When the test category, test perspective, check items, and test patterns are used as input data and the expected value is used as correct output data, information that can identify the company, etc. that conducted the test may be masked. The second trained model 2026 may be retrained as needed based on the test category, test perspective, check items, test patterns, and expected value included in a newly added test design document.

[0069] The additional learning may also include instruction tuning and reinforcement learning from human feedback (RLHF). Instruction tuning, for example, uses a data set consisting of various instructions and their corresponding desirable responses, and a data set consisting of various instructions and their corresponding undesirable responses, as learning data, making it possible to output information that is in line with the test engineer's intentions.

[0070] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the terminal device 10. The control unit 203 operates in accordance with the program to perform functions shown as a reception control module 2031, a transmission control module 2032, a creation assistance module 2033, a presentation control module 2034, and an adjustment module 2035.

[0071] The reception control module 2031 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.

[0072] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0073] The creation support module 2033 supports the user in creating a test design document. Specifically, for example, the creation support module 2033 uses a trained model at each step in test design to suggest to the user effective options for test design.

[0074] Specifically, for example, the creation support module 2033 uses the LLM system 30 when identifying test targets. For example, the creation support module 2033 transmits specification data selected by the user and an instruction statement for outputting a test division appropriate for the specification data to the LLM system 30. The LLM system 30 inputs the specification data and instruction statement transmitted from the server 20 into the LLM, and causes the LLM to output a test division appropriate for the specification data. The test division output from the LLM may be one or more. The test division output from the LLM may be assigned division 1, division 2, or division 3 as a sub-concept of the test division.

[0075] The creation support module 2033 issues an instruction statement for outputting a test section in response to, for example, a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting a test section, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the specification data. The instruction statement may also include content for specifying the output format of the test section. The creation support module 2033 may convert the specification data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format may include, for example, Markdown format.

[0076] Furthermore, for example, the creation assistance module 2033 uses the first trained model 2025 when selecting a test perspective. For example, the creation assistance module 2033 inputs a test section related to a test design document and an instruction statement for outputting a test perspective and check items corresponding to the test section into the first trained model 2025, and causes the first trained model 2025 to output the test perspective and check items corresponding to the input test section. The number of test perspectives output from the first trained model 2025 may be one or more. The number of check items associated with the test perspective output from the first trained model 2025 may be one or more.

[0077] When the first trained model 2025 outputs a test viewpoint, a check item, and an index value for the test viewpoint, the creation assistance module 2033 may determine whether to output the test viewpoint and the check item to a subsequent stage based on the index value. For example, when the index value of the test viewpoint output from the first trained model 2025 is less than a preset threshold, the creation assistance module 2033 may not output the test viewpoint and the check item associated with the test viewpoint to a subsequent stage.

[0078] If there are multiple types of index values, such as compatibility and defect risk, the test perspective and check items may be output when either one of the index values ​​exceeds a threshold, or when both index values ​​exceed a threshold. Alternatively, one index value may be given priority.

[0079] Furthermore, for example, the creation support module 2033 uses the first trained model for customer 20251 when selecting a test perspective for a specific customer. For example, the creation support module 2033 inputs a test category related to a test design document and an instruction statement for outputting test perspectives and check items corresponding to the test category into the first trained model for customer 20251, and causes the first trained model for customer 20251 to output the test perspectives and check items corresponding to the input test category. The number of test perspectives output from the first trained model for customer 20251 may be one or more. The number of check items associated with the test perspectives output from the first trained model for customer 20251 may be one or more.

[0080] When the first trained model for customer 20251 outputs test viewpoints, check items, and index values ​​for the test viewpoints, the creation assistance module 2033 may determine whether to output the test viewpoints and check items to a subsequent stage based on the index values. For example, when the index value of a test viewpoint output from the first trained model for customer 20251 is less than a preset threshold, the creation assistance module 2033 may not output the test viewpoint and the check items associated with the test viewpoint to a subsequent stage.

[0081] For example, the creation support module 2033 issues an instruction statement for outputting test perspectives and check items in response to a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting test perspectives and check items, and use the received text information as the instruction statement. The instruction statement may include content for explaining the structure of the test section. The instruction statement may also include content for specifying the output format of the test perspectives and check items.

[0082] Furthermore, the creation support module 2033 utilizes the LLM system 30 when determining the specific content of the test. For example, the creation support module 2033 transmits to the LLM system 30 the specification data, test division, test viewpoint, and check items selected by the user, as well as instructions for outputting appropriate factors and levels based on this information. The LLM system 30 inputs the specification data, test division, test viewpoint, check items, and instructions transmitted from the server 20 into the LLM, and causes the LLM to output factors and levels appropriate for the specification data, test division, test viewpoint, and check items selected by the user. The factors and levels output from the LLM may be one or more.

[0083] For example, the creation support module 2033 issues an instruction statement for outputting factors and levels in response to a user pressing a predetermined button. The creation support module 2033 may also receive text information from the user regarding an instruction for outputting factors and levels, and use the received text information as the instruction statement. The instruction statement may include content explaining what each piece of information to be input is. The instruction statement may also include content specifying an output format for the factors and levels. The creation support module 2033 may convert the specification data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format may include, for example, Markdown format.

[0084] Furthermore, for example, when generating a test pattern, the creation support module 2033 uses rules for generating patterns from factors and levels that are stored in advance in the storage unit 202. For example, the creation support module 2033 applies the rules to factors and levels to generate a pattern.

[0085] Furthermore, for example, when generating an expected value, the creation assistance module 2033 uses the second trained model 2026. For example, the creation assistance module 2033 inputs the specification data, test category, test viewpoint, confirmation item, and test pattern selected by the user, as well as instructions for outputting an expected value appropriate for this information, into the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information.

[0086] Also, for example, if a pattern table is not associated with the test perspective and confirmation items, the creation support module 2033 inputs the specification data, test category, test perspective, and confirmation items selected by the user, as well as instructions for outputting expected values ​​appropriate for this information, into the second trained model 2026, and causes the second trained model 2026 to output expected values ​​appropriate for the input information.

[0087] For example, the creation assistance module 2033 issues a command statement for outputting an expected value in response to a user pressing a predetermined button. The creation assistance module 2033 may also receive text information from the user regarding a command for outputting an expected value, and use the received text information as the command statement. The command statement may include content that explains what each piece of information to be input is. The command statement may also include content that specifies the output format of the expected value.

[0088] The presentation control module 2034 controls the process of presenting information to the user. For example, the presentation control module 2034 presents information output from the trained model to the user. Specifically, for example, the presentation control module 2034 presents the test section output from the LLM system 30 to the user. The presentation control module 2034 also presents the test perspective and check items output from the first trained model 2025 to the user. The presentation control module 2034 also presents the test perspective and check items output from the first trained model for customer 20251 to the user. The presentation control module 2034 also presents the factors and levels output from the LLM system 30 to the user. The presentation control module 2034 also presents the patterns generated by applying predetermined rules to the factors and levels to the user. The presentation control module 2034 also presents the expected values ​​output from the second trained model 2026 to the user.

[0089] The adjustment module 2035 adjusts the trained model and / or the creation support module 2033 for the customer. Specifically, for example, when test perspectives, confirmation items, and index values ​​for the test perspectives are output from the first trained model for customer 20251, the adjustment module 2035 adjusts the threshold for the index value output from the first trained model for customer 20251. For example, the adjustment module 2035 acquires attributes to which the customer belongs. The attributes include, for example, business type / industry, system type, and process type, and are stored in the client test information table 2027. The adjustment module 2035 adjusts the threshold in the creation support module 2033 based on, for example, the acquired attributes.

[0090] For example, the adjustment module 2035 sets a low suitability threshold for test perspectives and check items that are likely to be adopted for a predetermined attribute. Also, for example, the adjustment module 2035 sets a low suitability threshold for test perspectives and check items that are unlikely to be adopted for a predetermined attribute. Also, for example, the adjustment module 2035 sets a low defect risk threshold for test perspectives and check items that are likely to detect a defect for a predetermined attribute. Also, for example, the adjustment module 2035 sets a high defect risk threshold for test perspectives and check items that are unlikely to detect a defect for a predetermined attribute.

[0091] Furthermore, the adjustment module 2035 may set the threshold value by analyzing the information stored in the client test information table 2027. For example, the adjustment module 2035 may analyze the information stored in the client test information table 2027 to extract test perspectives and check items that are frequently adopted, and set the threshold based on the extracted results. For example, the adjustment module 2035 may analyze the information stored in the client test information table 2027 to extract test perspectives and check items that frequently detect defects, and set the threshold based on the extracted results. As a result, test perspectives and check items that are likely to be adopted by customers are output from the first trained model for customer 20251.

[0092] Furthermore, the adjustment module 2035 may adjust the customer-use first trained model 20251 for the customer by performing additional learning on the first trained model 2025. Specifically, for example, the adjustment module 2035 uses the test category stored in the client test information table 2027 as input data and the test perspectives and check items stored in the client test information table 2027 corresponding to the input test category as correct answer output data to fine-tune the customer-use first trained model 20251. The information input as input data may be large functions (category 1), medium functions (category 2), and small functions (category 3) as sub-concepts included in the test category. As a result, the customer-use first trained model 20251 outputs customer-specific test perspectives and check items for the input test category in addition to the test perspectives and check items based on the test information table 2024 or in place of the test perspectives and check items based on the test information table 2024. Furthermore, when adding a customer-specific test perspective, several dozen actual test cases linked to the test perspective to be added are required when training the first customer-specific trained model 20251.

[0093] Furthermore, the adjustment module 2035 may adjust the trained model and the creation support module 2033 for the client by adjusting a retriever. That is, the adjustment module 2035 may adjust the trained model and the creation support module 2033 for the client by employing a Retrieval-based Language Model (RAG). Specifically, for example, the adjustment module 2035 constructs a retriever in the storage unit 202 that searches the client test information table 2027 for information similar to input information. The retriever may be constructed in a memory area for the client. For example, the retriever calculates the similarity between the input test section vector and the test section vector stored in the client test information table 2027, and acquires test sections with high similarity as search results. For example, the creation support module 2033 inputs test sections related to the test design document into the retriever. The retriever acquires test sections similar to the input test section from the client test information table 2027. The creation support module 2033 inputs the test division related to the test design document, the test division acquired from the client test information table 2027, and an instruction statement for outputting the test viewpoint and check items corresponding to the test division into the first trained model for customer 20251. The first trained model for customer 20251 outputs the test viewpoint and check items corresponding to the input test division.

[0094] <2 Data Structure> 4 to 9 are diagrams showing the data structures of tables stored in the server 20. Note that Figures 4 to 9 are merely examples and do not exclude data that is not listed. Furthermore, even data that is listed in the same table may be stored in separate storage areas in the storage unit 202.

[0095] Fig. 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with the user ID as a key.

[0096] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The gender is an item that stores the gender of the user. The date of birth is an item that stores the date of birth of the user. The contact information is an item that stores the contact information (e.g., telephone number, email address, etc.) of the terminal device 10 that the user has.

[0097] Fig. 5 is a diagram showing the data structure of the specification table 2022. The specification table 2022 shown in Fig. 5 is a table having columns such as client ID, project ID, creation date and time, specification name, specification data, and latest update, with specification ID as a key. Note that the information stored in Fig. 5 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0098] The specification ID is an item that stores an identifier for uniquely identifying a specification. The client ID is an item that stores an identifier for uniquely identifying a client who requires testing based on the specification. The project ID is an item that stores an identifier for uniquely identifying a project to which a function that is the target of testing based on the specification belongs. The creation date / time is an item that stores the date and time the specification was created. The specification name is an item that stores the name of the specification. The specification name may be given by extracting a portion of the content of the specification, or may be given by the creator of the specification, etc. Specification data includes data regarding detailed specifications of the function, data regarding message definitions, etc. The item "specification data" may store reference information (path) for a specification data file located in another location. The last update is an item that stores the date and time the specification data was last updated.

[0099] FIG. 6 is a diagram showing the data structure of the prompt table 2023. The prompt table 2023 shown in FIG. 6 is a table having columns such as prompt data, with a prompt ID as a key. Note that the information stored in FIG. 6 is not limited to this. For example, further information may be stored. For example, a column may be provided for storing an ID that identifies an industry. This allows the creation assistance module 2033 to select instructions appropriate for each industry, for example.

[0100] The prompt ID is an item that stores an identifier for uniquely identifying a prompt. The prompt data is an item that stores data related to the prompt, which is an instruction to be input to the LLM. The data related to the prompt is, for example, text information related to the prompt. In the text information related to the prompt, for example, the content that identifies the information is left blank, and predetermined content is inserted when the prompt is used.

[0101] The text information related to the prompt is, for example, an instruction sentence for outputting a test section based on the specification data, and includes, for example, the following content: "The input document is a specification {...}. Please output the test category for testing the functions related to this specification in the format..."

[0102] The text information related to the prompt is, for example, an instruction sentence for outputting test viewpoints and check items corresponding to the test category, and includes, for example, the following content. "The input document is in the test category {...}. Please output the test perspectives and check items corresponding to this test category in the format..."

[0103] The text information related to the prompt is, for example, an instruction for outputting factors and levels based on specification data, test category, test viewpoint, and confirmation item, and includes, for example, the following content: "The input documents are the specification {...}, the test category {...} for which testing is desired for this specification, the test perspective {...}, and the confirmation items {...}. Please output the factors and levels for conducting testing based on this information in the format of ..."

[0104] The text information related to the prompt is, for example, an instruction for outputting an expected value based on specification data, test category, test viewpoint, confirmation item, and test pattern, and includes, for example, the following content: "The input documents are the specification {...}, the test category {...} for which testing is desired for this specification, the test perspective {...}, the confirmation items {...}, and the test pattern {...}. Please output the expected values ​​for conducting testing based on this information in the format..."

[0105] The item "prompt data" may store a reference (path) to a prompt data file located elsewhere.

[0106] 7 to 9 are diagrams showing the data structure of the test information table 2024. The test information table 2024 includes, for example, a test design document table 20241, a pattern table 20242, and a test viewpoint table 20243. The pattern table 20242 and the test viewpoint table 20243 are tables that store elements decomposed from the test design document table 20241.

[0107] FIG. 7 is a diagram showing the data structure of the test design document table 20241. Each record of the test design document table 20241 shown in FIG. 7 has columns for file name, tag, designer ID, project ID, service ID, attribute, link, and latest update. The file name is an item that stores the name of the test design document. The tag is an item that stores information that represents the characteristics of the test. Information in the "tag" item is stored, for example, based on information set for the project related to the specification. Furthermore, the information in the "tag" item may be set arbitrarily by the user.

[0108] The designer ID is an item that stores an identifier for uniquely identifying the person who created the test design document. The project ID is an item that stores an identifier for uniquely identifying the project for which the test design document was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document belongs. Multiple types of attributes can be set in the "attribute" item, and for example, one or more attributes can be set from business type / industry, system type, process type, etc. The information in the "attribute" item is set, for example, based on the information stored in the "tag" item. Furthermore, the information in the "attribute" item may be set arbitrarily by the user.

[0109] The link is an item that stores specification data associated with the test design document. The specification data itself may be associated with the item "link", but the relevant section of the specification data may also be associated. For example, if the specification data is a spreadsheet, the relevant section of the specification data may be stored in the form of a slide, row, cell, or the like. The last update is an item that stores the date and time when the test design document was last updated.

[0110] FIG. 8 is a diagram showing the data structure of the pattern table 20242. Each record of the pattern table 20242 shown in FIG. 8 has columns for pattern name, pattern number, tag, file name, project ID, service ID, attribute, and latest update. "Pattern name" is an item that stores the name of the pattern. "Pattern number" is an item that stores a unique number assigned to the pattern. "Tag" is an item that stores information that is attached as a tag to the test design document that is the source of decomposition.

[0111] The file name is an item that stores the name of the test design document to be decomposed. The project ID is an item that stores an identifier for uniquely identifying the project in which the test design document to be decomposed was created. The service ID is an item that stores an identifier for uniquely identifying the service in which the test target is used. The attribute is an item that stores information indicating the type to which the test design document to be decomposed belongs. The last update is an item that stores the date and time when the pattern was last updated.

[0112] FIG. 9 is a diagram showing the data structure of the test perspective table 20243. Each record of the test perspective table 20243 shown in FIG. 9 has columns for test perspective, check item, tag, category 1, category 2, attribute, and latest update. The test perspective is an item for storing a test perspective. The check item is an item for storing information for identifying the test to be performed. The tag is an item for storing information attached as a tag to the test design document from which the test perspective is decomposed. Category 1 is an item for storing information about the first category attached to the test perspective. The information stored in the item "category 1" is set by the user when inputting the test perspective, for example, during the creation stage of the test design document. The information stored in the item "category 1" is set by the user, for example, by selecting from a plurality of pre-set items. The information stored in the item "category 1" is, for example, information related to the test category.

[0113] Classification 2 is an item that stores information about the second classification assigned to a test perspective. The information stored in the item "Classification 2" is set by the user, for example, when inputting a test perspective during the creation stage of a test design document. The information stored in the item "Classification 2" is set by the user, for example, by selecting from a plurality of pre-set items. The information stored in the item "Classification 2" is, for example, information related to the test division. The number of classifications stored in the test perspective table 20243 is not limited to two, and may be three or more. The attribute is an item that stores information indicating the type to which the test design document from which the test perspective is decomposed belongs. The last update is an item that stores the most recent date and time the test perspective was updated.

[0114] <3 operations> The operation of the server 20 when assisting in the creation of a test design document will now be described.

[0115] (Test design document creation support 1) FIG. 10 is a schematic diagram showing the operation of the server 20. FIG. 11 is a diagram for explaining the operations of the terminal device 10 and the server 20 when the server 20 assists in creating a test design document.

[0116] First, a user (test engineer) who wants to create a test design document accesses the server 20. The user logs in to a service that assists in creating a test design document, provided by the server 20, for example, by entering his or her own user ID and password.

[0117] In step S11, the terminal device 10 accepts specification data designation from the user. Specifically, for example, the display 141 displays a list of specifications stored in the specification table 2022. The user selects a specification for which a test design document needs to be created from the specifications displayed in the list. After selecting a specification, the user inputs an instruction to the terminal device 10 to output a test section corresponding to the specification. The transmitting / receiving unit 192 transmits information related to the designated specification data and an instruction to output the test section to the server 20.

[0118] In step S12, the server 20 obtains a test division appropriate for the specified specification data based on the specified specification data. Specifically, for example, the creation support module 2033 reads an instruction statement from the prompt table 2023 for outputting a test division appropriate for the specification data. The creation support module 2033 outputs information about the specification data specified by the user and the read instruction statement to the LLM system 30. The LLM system 30 inputs the information about the specification data and the instruction statement transmitted from the server 20 into the LLM, causing the LLM to output a test division appropriate for the specification data. At this time, the LLM outputs, for example, text information indicating the items / functions to be tested. The LLM assigns, for example, hierarchical information such as "test division," "division 1," "division 2," "division 3," or any combination thereof to the text information. The LLM system 30 transmits the information output from the LLM to the server 20.

[0119] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test section transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test section transmitted from the LLM system 30, to the terminal device 10.

[0120] In step S13, the terminal device 10 displays a screen for creating a test design document including the test division transmitted from the LLM system 30 to the user.

[0121] 12 is a schematic diagram showing an example of a test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0122] The display area 1411 is an area where a list of specification data is displayed. In Fig. 12, the presentation control unit 193 displays one piece of specification data in the display area 1411. In Fig. 12, the presentation control unit 193 marks the square to the left of the specification data to indicate that the specification data has been specified by the user. Note that when a test design document creation screen is created for a single piece of specification data, the display area 1411 does not necessarily have to be provided.

[0123] The operation object 1412 is an object for inputting an instruction to request a proposal for a test section.

[0124] 12, the presentation control unit 193 displays text information in the display area 1413 based on the hierarchical information of the test section, section 1, section 2, and section 3.

[0125] The operation object 1414 is an object for inputting an instruction to request the proposal of a test viewpoint and confirmation items. The presentation control unit 193 displays the operation object 1414, for example, in an area associated with a predetermined test section. Specifically, for example, the presentation control unit 193 displays the operation object 1414 in association with a line displaying text information marked with section 3, which indicates that the test section is the lowest level.

[0126] 12 , the user specifies specification data in the display area 1411 and presses the operation object 1412. Then, the terminal device 10 transmits an instruction to the server 20 requesting a test section proposal. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output a test section suitable for the specification data specified by the user. The presentation control unit 193 displays text information related to the test section output for the specified specification data in the display area 1413.

[0127] In step S14, the terminal device 10 accepts the selection of a test subject. Specifically, for example, the display 141 displays a list of multiple test sections provided by the server 20. The user selects a test section for which the user wants to check test perspectives and check items from the test sections displayed in the list. The user inputs an instruction to the terminal device 10 to output test perspectives and check items corresponding to the selected test section. The transmitting / receiving unit 192 transmits information about the selected test section and an instruction to output the test perspectives and check items to the server 20.

[0128] In step S15, the server 20 outputs test perspectives and check items appropriate for the test section based on the selected test section. Specifically, for example, the creation assistance module 2033 reads, from the prompt table 2023, an instruction statement for outputting test perspectives and check items corresponding to the test section. The creation assistance module 2033 inputs the selected test section and the read instruction statement into the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items appropriate for the test section. At this time, the first trained model 2025 outputs, for example, text information representing the test perspective and text information representing the check items.

[0129] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test viewpoints and check items output from the first trained model 2025. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the test viewpoints and check items output from the first trained model 2025, to the terminal device 10.

[0130] In step S16, the terminal device 10 displays a test design document creation screen including the outputted test viewpoints and check items to the user.

[0131] 13 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0132] The display area 1415 is an area where the test perspectives and check items for the selected test section are displayed. In Fig. 13, the presentation control unit 193 displays the test perspectives and check items output for two test sections in the display area 1415. In Fig. 13, the presentation control unit 193 displays a test section for which one set of test perspectives and check items is proposed, and a test section for which two sets of test perspectives and check items are proposed.

[0133] 12, the user presses the operation object 1414 corresponding to the test section for which the user desires test perspectives and check items. The terminal device 10 then transmits an instruction to the server 20 requesting the proposal of test perspectives and check items. In response to the instruction, the creation assistance module 2033 causes the first trained model 2025 to output test perspectives and check items suitable for the test section selected by the user. The presentation control unit 193 displays text information related to the test perspectives and check items output for the selected test section in the display area 1415.

[0134] In step S17, the terminal device 10 accepts the selection of a test perspective and a check item. Specifically, for example, the display 141 displays the test perspective and check items provided by the server 20. The user selects a test perspective and check item for which the user wants to check factors and levels from the displayed test perspectives and check items. The user inputs an instruction to the terminal device 10 to output factors and levels corresponding to the selected test perspective and check item. The transmitting / receiving unit 192 transmits information related to the selected test perspective and check item and an instruction to output the factors and levels to the server 20.

[0135] In step S18, the server 20 acquires factors and levels appropriate for the specification data specified by the user and the test division, test perspective, and check items selected by the user, based on the information. Specifically, for example, the creation support module 2033 reads instructions for outputting factors and levels appropriate for the specification data, test division, test perspective, and check items from the prompt table 2023. The creation support module 2033 outputs information regarding the specification data specified by the user, the test division, test perspective, and check items selected by the user, and the read instructions to the LLM system 30. The LLM system 30 inputs the specification data, information regarding the test division, test perspective, and check items transmitted from the server 20 into the LLM, and causes the LLM to output factors and levels appropriate for the selected test perspective and check items. At this time, the LLM outputs, for example, text information representing the factors and text information representing the levels. The LLM may also output remarks for the levels. The LLM system 30 transmits the information output from the LLM to the server 20.

[0136] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the factors and levels transmitted from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the factors and levels transmitted from the LLM system 30, to the terminal device 10.

[0137] In step S19, the terminal device 10 displays a test design document creation screen including the factors and levels transmitted from the LLM system 30 to the user.

[0138] 14 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0139] The operation object 1416 is an object for inputting an instruction to request a proposal of factors and levels.

[0140] The display area 1417 is an area where the factors and levels for the selected test perspective and confirmation item are displayed. In Fig. 14, the presentation control unit 193 displays the factors and levels for the test perspective: "display / hide" and the confirmation item: "the text is displayed according to the pattern" in the display area 1417.

[0141] 14 , the user selects the test perspective and check item for which the user desires factors and levels, and presses the operation object 1416. The terminal device 10 then transmits an instruction to the server 20 requesting a proposal of factors and levels. In response to the instruction, the creation support module 2033 causes the LLM system 30 to output factors and levels appropriate for the specification data specified by the user, and the test category, test perspective, and check item selected by the user. The presentation control unit 193 displays text information related to the factors and levels output from the LLM system 30 in the display area 1417.

[0142] In step S110, the terminal device 10 accepts an instruction to create a test pattern. Specifically, for example, the display 141 displays a screen for creating a pattern table based on factors and levels provided by the server 20 in response to an instruction from the user. The user selects a rule for creating a test pattern displayed on the creation screen. The rule for creating a test pattern is, for example, "two-factor coverage," "full coverage," or "constraint + full coverage." When a rule is selected, the transmitter / receiver 192 transmits information about the factors and levels, information about the selected rule, and an instruction to create a test pattern to the server 20.

[0143] In step S111, the server 20 creates a test pattern from factors and levels based on the selected rule. Specifically, for example, the creation support module 2033 applies the selected rule to the factors and levels to create a test pattern. At this time, the creation support module 2033, for example, expands predetermined marks in accordance with the selected rule in a matrix with multiple columns, each row representing a factor and a level. The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the created test pattern. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the created test pattern, to the terminal device 10.

[0144] In step S112, the terminal device 10 displays a screen for creating a test design document including a test pattern to the user.

[0145] 15 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0146] The operation object 1418 is an object for selecting a rule for generating a test pattern. In Fig. 15, "bifactorial coverage" is selected as the rule.

[0147] The created test pattern is displayed in a display area 1419. In Fig. 15, the presentation control unit 193 displays a matrix containing "circles" in the display area 1419 as a test pattern.

[0148] 15, the user presses an operation object 1418 for selecting a rule for creating a test pattern. The terminal device 10 then transmits an instruction to the server 20 requesting the creation of a test pattern. The creation assistance module 2033 creates a test pattern in accordance with the instruction and the selected rule. The presentation control unit 193 displays the created test pattern in the display area 1419.

[0149] In step S113, the terminal device 10 accepts an instruction to create an expected value. Specifically, for example, the display 141 displays factors and levels provided by the server 20 and a test pattern created by the server 20. The user inputs an instruction to the terminal device 10 to output an expected value corresponding to a case defined by the test pattern. The transmitter / receiver 192 transmits information about the case defined by the test pattern and an instruction to output the expected value to the server 20.

[0150] In step S114, the server 20 outputs an appropriate expected value for the selected case based on the specification data specified by the user and the test section, test perspective, check item, and case selected by the user. Specifically, for example, the creation support module 2033 reads, from the prompt table 2023, an instruction statement for outputting an expected value appropriate for the specification data, test section, test perspective, check item, and test pattern. The creation support module 2033 inputs the case defined by the specification data specified by the user, the test section, test perspective, check item, and test pattern selected by the user, and the read instruction statement into the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information. At this time, the second trained model 2026 outputs, for example, text information related to the expected value.

[0151] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the expected value output from the second trained model 2026. That is, for example, the transmission control module 2032 transmits information for displaying a test design document creation screen, including the expected value output from the second trained model 2026, to the terminal device 10.

[0152] In step S115, the terminal device 10 displays a test design document creation screen including the output expected values ​​to the user.

[0153] 16 is a schematic diagram showing another example of the test design document creation screen displayed on the display 141. The creation screen shown in FIG.

[0154] The operation object 14110 is an object for inputting an instruction to request a proposal of an expected value.

[0155] The display area 14111 is an area where the cases for which the expected values ​​are to be created are displayed. In Fig. 16, the presentation control unit 193 displays a matrix in which the factors and levels are rows and each case is a column in the display area 14111 as the target for which the expected values ​​are to be created. In Fig. 16, for example, the presentation control unit 193 displays the case where the factor is "product type" and the level is "regular products only" as the first case of the test pattern, with the factor being "total product price" and the level being "less than 2000 yen" in the matrix.

[0156] The display area 14112 is an area where the expected values ​​and the cases for which the expected values ​​were created are displayed. In Fig. 16, the presentation control unit 193 displays black circles representing the targets for which the expected values ​​are to be created and the created expected values ​​in the display area 14112. In Fig. 16, the presentation control unit 193 displays, for example, the expected values ​​created for the first case, but the expected values ​​for the other cases may also be created collectively.

[0157] 16 , the user presses the operation object 14110 in the display area 14112. In response to this, the terminal device 10 transmits an instruction to the server 20 requesting a proposal of an expected value. In response to the instruction, the creation assistance module 2033 causes the second trained model 2026 to output an expected value for a case defined by the test pattern. The presentation control unit 193 displays text information related to the output expected value in the display area 14112.

[0158] (Test design document creation support 2) This section explains the operation when at least one of multiple processes using a trained model when creating a test case based on a specification is adjusted for a customer. This operation will also be explained using Figure 11 to explain the operation of the terminal device 10 and the server 20.

[0159] In step S15 of FIG. 11, the server 20 outputs appropriate test viewpoints and check items for the test division based on the selected test division. Specifically, for example, the creation support module 2033 reads, from the prompt table 2023, an instruction statement for outputting test viewpoints and check items corresponding to the test division. The creation support module 2033 inputs the selected test division and the read instruction statement into the first trained model for customer 20251, and causes the first trained model for customer 20251 to output test viewpoints and check items appropriate for the test division. At this time, the first trained model for customer 20251 outputs, for example, text information representing the test viewpoints and text information representing the check items.

[0160] The presentation control module 2034 causes the terminal device 10 to display a screen for creating a test design document, including the test viewpoints and check items output from the first trained model for customer 20251. That is, for example, the transmission control module 2032 transmits information for displaying a creation screen for a test design document, including the test viewpoints and check items output from the first trained model for customer 20251, to the terminal device 10.

[0161] In step S16, the terminal device 10 displays a test design document creation screen including the outputted test viewpoints and check items to the user, for example, as shown in FIG.

[0162] (Adjusting processing using a trained model) 17 is a flowchart showing an example of the operation of the server 20 when adjusting a process using a trained model. The following describes an example of adjusting a process using a first trained model for customer 20251 that outputs test viewpoints (and check items) based on acquired test knowledge.

[0163] First, information provided by a customer and accumulated in debugging processes previously performed by the customer is stored in the client test information table 2027. The control unit 203 stores, for example, information provided by the customer in the client test information table 2027.

[0164] An administrator of a service that provides assistance in creating test design documents operates the server 20 and adjusts the processing using the trained model for the customer. The administrator selects, for example, an adjustment method. Examples of adjustment methods include the following. Note that the following methods may be adopted individually or in combination. Parameter tuning Fine tuning RAG

[0165] The method selected by the administrator is not limited to one. The administrator may select a combination of two or more of the above adjustments. The adjustment method may also be other methods than those described above, such as instruction tuning. When the administrator selects a tuning method, the adjustment module 2035 starts the adjustment.

[0166] In step S21, the adjustment module 2035 reads information from the client test information table 2027.

[0167] In step S22, the adjustment module 2035 adjusts the processing using the first trained model for customer 20251 for the customer. If parameter tuning is selected by the administrator, the adjustment module 2035 adjusts the threshold for the index value output from the first trained model for customer 20251 based on, for example, information read from the client test information table 2027.

[0168] 18 and 19 are conceptual diagrams for explaining parameter tuning by the adjustment module 2035. The adjustment module 2035 adjusts a threshold for an index value output from the first trained model for customer 20251.

[0169] In addition, when fine tuning is selected by the administrator, the adjustment module 2035 trains the first trained model 20251 for the customer, for example, using the test category stored in the client test information table 2027 as input data and the test perspectives and confirmation items stored in the client test information table 2027 corresponding to the input test category as correct output data.

[0170] Furthermore, when the administrator selects a RAG, the adjustment module 2035 constructs a searcher in the storage unit 202 that searches the client test information table 2027 for a test section similar to the input test section, for example.

[0171] Fig. 20 is a diagram showing another example of the functional configuration of the server 20 when RAG is adopted. As shown in Fig. 20, a searcher 2028 is constructed in the storage unit 202. The searcher 2028 is stored, for example, in a storage area set for the corresponding customer.

[0172] In step S23, the adjustment module 2035 provides processing using the first trained model for customer 20251 in a memory area for the customer.

[0173] The adjustment module 2035 may suggest to the user at a predetermined timing an adjustment of the processing using the first trained model for customer 20251. For example, the adjustment module 2035 suggests to the user an adjustment of the processing using the first trained model for customer 20251 at a predetermined period, for example, every six months. Also, for example, the adjustment module 2035 suggests to the user an adjustment of the processing using the first trained model for customer 20251 when a predetermined amount of data is accumulated in the client test information table 2027.

[0174] As described above, in the above embodiment, the adjustment module 2035 acquires test knowledge related to a customer. The adjustment module 2035 adjusts, based on the acquired test knowledge, a process using one of multiple trained models used when creating test cases based on specifications, the trained models being trained to output predetermined elements that constitute test cases in response to input data related to the specifications. The adjustment module 2035 provides the process using the adjusted trained model as a process for the customer to which the test knowledge was provided. In this way, the adjustment module 2035 can customize support for creating test design documents for the customer by adjusting a process using one of the multiple trained models used when creating test cases.

[0175] Therefore, according to the program, method, information processing device, and system of this embodiment, processing using a trained model used in debugging processing can be customized for each client while reducing effort.

[0176] In the above embodiment, the trained model is a large-scale language model, which enables the creation assistance module 2033 to provide the user with information corresponding to the input information with high accuracy.

[0177] Furthermore, in the above embodiment, the trained model to be adjusted is a trained model 2025 that has been trained to output a test perspective in response to input of a test section related to the specification. Because a test perspective is information that represents the content of a test, it is likely to reflect content specific to a customer and content specific to a business type / industry. Therefore, when processing using the trained model is customized for a customer, the effect of the customization is likely to be reflected in the output. Therefore, by adjusting processing using a trained model that has been trained to output a test perspective in response to input of a test section related to the specification, it is possible to more effectively customize support for creating test designs for customers.

[0178] In the above embodiment, the trained model is trained to output predetermined elements constituting a test case and index values ​​related to the elements in response to input data related to specifications. The adjustment module 2035 adjusts the threshold value of the index value used as a condition for presenting the output element based on the acquired test knowledge. This allows the adjustment module 2035 to more easily adjust processing using the trained model for the customer.

[0179] In the above embodiment, the trained model is trained to output predetermined elements constituting a test case and multiple types of index values ​​related to the elements in response to input data related to specifications. The adjustment module 2035 adjusts the threshold for each type of index value based on the acquired test knowledge. This enables the adjustment module 2035 to easily but highly accurately adjust processing using the trained model for a customer.

[0180] Furthermore, in the above embodiment, the adjustment module 2035 uses the acquired test knowledge as training data to additionally train the trained model. This allows the adjustment module 2035 to adjust the trained model so that customer-specific test items, test functions, test content, etc. are output from the trained model. Furthermore, the adjustment module 2035 allows the trained model to adjust so that business / industry-specific test items, test functions, test content, etc. are output from the trained model.

[0181] Furthermore, in the above embodiment, in processing using a trained model, the creation assistance module 2033 inputs data related to specifications to a retriever 2028, and causes the retriever 2028 to output data similar to the input data related to the specifications from information stored in a predetermined database 2027, and inputs the data to the trained model. The adjustment module 2035 stores the acquired test knowledge in the database 2027. This enables the adjustment module 2035 to adjust processing using a trained model for a customer with high accuracy while reducing processing man-hours.

[0182] In the above embodiment, the adjustment module 2035 acquires test results of test cases created using multiple trained models as test knowledge. This enables the adjustment module 2035 to adjust processing using trained models based on the test results of test cases created using test design document creation support. At least a portion of the results of debugging processing performed by a customer are not stored in the test information table 2024, but are stored in the client test information table 2027, thereby preventing information about the debugging processing performed by the customer from leaking.

[0183] <Modification> In the above embodiment, a case has been described in which a test case is created by a user selecting information output by a trained model. However, the information output by the trained model does not necessarily have to be adopted as is. For example, a user may modify the information output from the trained model. That is, a user may modify and adopt the test section output from the LLM system 30. A user may also add a new test section and adopt the added test section. A user may also modify and adopt the test perspectives and check items output from the first trained model 2025. A user may also add new test perspectives and check items and adopt the added test perspectives and check items. A user may also modify and adopt the factors and levels output from the LLM system 30. A user may also add new factors and levels and adopt the added factors and levels. A user may also modify a test pattern created based on the factors and levels. A user may also add a new test pattern and adopt the added test pattern. A user may also modify and adopt the expected value output from the second trained model 2026. The user may also create a new expected value and adopt the created expected value.

[0184] In addition, in the above embodiment, a case is described in which information of a type expected to be input is input to a trained model, and the trained model is caused to output predetermined information. However, the information input to the trained model may be part of the information expected to be output. In other words, a user may be responsible for part of creating the predetermined elements, and the trained model may perform the rest.

[0185] For example, when setting a test section, the user creates an upper-level test section, such as test section 1, and instructs the server 20 to output a lower-level test section. The creation support module 2033 outputs the specification data, the test section created by the user, and an instruction statement for outputting the lower-level test section to the LLM system 30. The LLM system 30 causes the LLM to output the lower-level test section and transmits the output information to the server 20.

[0186] Also, for example, when setting test perspectives and check items, the user creates at least one test perspective and check item and instructs the server 20 to output other test perspectives and check items. The creation assistance module 2033 outputs the test category, the test perspectives and check items created by the user, and an instruction statement for outputting the other test perspectives and check items to the first trained model 2025. The first trained model 2025 outputs the other test perspectives and check items.

[0187] Furthermore, for example, when setting factors and levels, the user creates at least one factor and level and instructs the server 20 to output other factors and levels. The creation support module 2033 outputs to the LLM system 30 instruction statements for outputting other appropriate factors and levels based on the specification data, test category, test viewpoint, check item, factors and levels created by the user, and this information. The LLM system 30 causes the LLM to output the factors and levels and transmits the output information to the server 20.

[0188] Also, for example, when setting an expected value, the user creates at least one expected value and instructs the server 20 to output other expected values. The creation assistance module 2033 outputs specification data, test category, test viewpoint, check item, test pattern, expected value created by the user, and an instruction statement for outputting other expected values ​​appropriate for this information to the second trained model 2026. The second trained model 2026 outputs the other test viewpoints and check items.

[0189] In the above embodiment, the LLM system 30 is configured to output the test divisions, factors, and levels. However, the test divisions, factors, and levels are not limited to being output from the LLM system 30, and may be output from trained models that have undergone additional training on the LLM, such as the first trained model 2025 and the second trained model 2026. Additional training, for example, involves fine-tuning, which teaches the correct answer for each task, to specialize in the creation of test design documents. Specifically, for example, by training a test division appropriate for a specification as the correct answer, a test division appropriate for the input specification data is output in response to input of a prompt for outputting the specification data and a test division appropriate for the specification data. Furthermore, for example, by training a factor and level corresponding to the specification data, test division, test perspective, and check item as the correct answer, a factor and level appropriate for the input information is output in response to input of a prompt for outputting the specification data, test division, test perspective, check item, and a factor and level corresponding to this information.

[0190] Furthermore, in the above embodiment, a case has been described in which various information is input to the trained model. However, the information input to the trained model is not necessarily limited to the above. For example, in the above embodiment, the check items are input to the trained model together with the test viewpoints. However, if the check items are not included in the training data, the check items do not necessarily have to be input to the trained model.

[0191] Furthermore, in the above embodiment, for example, the creation assistance module 2033 is configured to output test perspectives and check items that are plausibly set among the test perspectives and check items output from the first trained model 2025. However, the test perspectives and check items output from the first trained model 2025 are not limited to test perspectives and check items that are plausibly set. For example, the test information table 2024 stores the results of tests conducted for the set test perspectives and check items. Then, as learning data for training the first trained model 2025, for example, the test division stored in the test information table 2024 is used as input data, and the test perspectives, check items, and information on defects as test results stored in the test information table 2024 that correspond to the input test division are used as correct output data. As a result, the first trained model 2025 outputs a test perspective that is estimated to be appropriate for the input test division, at least one check item associated with the test perspective, and information on defects. This makes it possible to identify test perspectives and check items where many defects are likely to occur, thereby improving test efficiency. Note that the first trained model 2025 may be configured to preferentially output test perspectives and check items where many defects are likely to occur from among test perspectives estimated to be appropriate for the input test category and at least one check item associated with the test perspectives.

[0192] Furthermore, the first trained model 2025 may be trained to output a test perspective estimated to be appropriate for an input test category and at least one check item associated with the test perspective, as well as to output an index value representing the defect occurrence rate for the output test perspective. Furthermore, the first trained model 2025 may be trained to output a test perspective and check items, as well as to output an index value representing the defect occurrence rate for the output test perspective and check item.

[0193] Furthermore, in the above embodiment, the user can select, modify, or add to the suggestions output by the trained model. The creation assistance module 2033 stores, for example, in the test information table 2024, items set through modifications or additions by the user. The creation assistance module 2033 also stores, for example, in the test information table 2024, items proposed by the trained model and selected by the user. When fine-tuning the trained model, the server 20 includes in the training data the history of the adoption of the item (whether it was a selected item, a modified item, or an added item). This allows the trained model to prioritize output of content created by the user, making it possible to output information desired by the user with high accuracy.

[0194] Furthermore, in the above embodiment, an example has been described in which, when proposing test perspectives and check items, a test category is input to the first trained model 2025. However, if specification data is included in the learning data, the specification data may be input to the first trained model 2025 in addition to the test category.

[0195] In the above embodiment, the server 20 has the prompt table 2023. However, the server 20 does not necessarily have to have the prompt table 2023. For example, the server 20 may have a predetermined link module, and the link module may include a prompt template. The link module is, for example, an API (Application Programming Interface). Note that the link module may not be provided within the server 20, but may be provided on a server different from the server 20.

[0196] Specifically, for example, in step S12, the creation support module 2033 outputs information about the specification data specified by the user to the linking module. The linking module creates an instruction statement by combining information held as a template with information about the specification data. The linking module outputs the information about the specification data specified by the user and the created instruction statement to the LLM system 30.

[0197] Also, for example, in step S15, the creation assistance module 2033 outputs information about the test category selected by the user to the collaboration module. The collaboration module creates an instruction sentence by combining information held as a template with information about the test category. The collaboration module inputs the information about the test category selected by the user and the created instruction sentence into the first trained model 2025, and causes the first trained model 2025 to output test perspectives and check items appropriate for the test category.

[0198] Also, for example, in step S18, the creation support module 2033 outputs the specification data specified by the user and information regarding the test division, test perspective, and check items selected by the user to the collaboration module. The collaboration module creates an instruction statement by combining information held as a template with the specification data, test division, test perspective, and check items. The collaboration module outputs the specification data specified by the user, information regarding the test division, test perspective, and check items selected by the user, and the created instruction statement to the LLM system 30.

[0199] Also, for example, in step S114, the server 20 outputs an appropriate expected value for the selected case based on the specification data specified by the user, and the test division, test perspective, check items, and case selected by the user. Specifically, for example, the creation assistance module 2033 outputs the case defined by the specification data specified by the user, and the test division, test perspective, check items, and test pattern selected by the user to the linking module. The linking module creates an instruction statement by combining information held as a template with the specification data specified by the user, and the case defined by the test division, test perspective, check items, and test pattern selected by the user. The linking module inputs the specification data specified by the user, the test division, test perspective, check items, and test pattern selected by the user, and the created instruction statement to the second trained model 2026, and causes the second trained model 2026 to output an expected value appropriate for the input information.

[0200] In the above embodiment, the LLM of the LLM system 30 outputs the test section and also outputs the factors and levels, but this is not limited to this. The LLM that outputs the test section and the LLM that outputs the factors and levels may be different LLMs.

[0201] In addition, in the above embodiment, the case where the multiple trained models are LLMs has been described as an example. However, the multiple trained models according to this embodiment may be trained models that are not LLMs.

[0202] Furthermore, in the above embodiment, an example is described in which a process using the first trained model for customer 20251 that outputs test viewpoints (and confirmation items) is adjusted. However, the process to be adjusted is not limited to the process using the first trained model for customer 20251. For example, the adjustment module 2035 may customize a process using the second trained model 2026 for a customer.

[0203] Specifically, for example, when an expected value and an index value for the expected value are output from the second trained model 2026, the adjustment module 2035 adjusts the threshold for the index value output from the second trained model 2026 based on the information stored in the client test information table 2027.

[0204] In addition, the adjustment module 2035 fine-tunes the second trained model 2026 using, for example, specification data, test category, test perspective, confirmation items, and test pattern stored in the client test information table 2027 as input data, and the expected value stored in the client test information table 2027 corresponding to the input information as correct output data.

[0205] The adjustment module 2035 also employs, for example, RAG and constructs in the storage unit 202 a searcher that searches the client test information table 2027 for information similar to the input specification data, test section, test viewpoint, check item, and test pattern. The creation support module 2033 inputs, for example, the specification data, test section, test viewpoint, check item, and test pattern to the searcher. The searcher acquires information similar to the input information from the client test information table 2027. The creation support module 2033 inputs the specification data, test section, test viewpoint, check item, and test pattern, the information acquired from the client test information table 2027, and an instruction statement for outputting an expected value corresponding to the input information to the second trained model 2026. The second trained model 2026 outputs an expected value corresponding to the input information.

[0206] Also, for example, if the test section is output from a trained model that has undergone additional training in an LLM, the adjustment module 2035 may customize the processing using the trained model for the customer.

[0207] Specifically, for example, when a test section and an index value for the test section are output from the trained model, the adjustment module 2035 adjusts the threshold for the index value output from the trained model based on the information stored in the client test information table 2027.

[0208] In addition, the adjustment module 2035 fine-tunes the learned model, for example, using specification data stored in the client test information table 2027 as input data and the test category stored in the client test information table 2027 that corresponds to the input specification data as correct output data.

[0209] Furthermore, the adjustment module 2035 employs, for example, RAG and constructs in the storage unit 202 a searcher that searches the client test information table 2027 for specification data similar to input specification data. The creation support module 2033, for example, inputs specification data to the searcher. The searcher acquires specification data similar to the input specification data from the client test information table 2027. The creation support module 2033 inputs the specification data, the specification data acquired from the client test information table 2027, and an instruction statement for outputting a test category corresponding to the input specification data to the trained model. The trained model outputs a test category corresponding to the input information.

[0210] Also, for example, if the factors and levels are output from a trained model that has undergone additional training on an LLM, the adjustment module 2035 may customize the processing using the trained model for the customer.

[0211] Specifically, for example, when a trained model outputs factors and levels and index values ​​for the factors and levels, the adjustment module 2035 adjusts the threshold for the index values ​​output from the trained model based on the information stored in the client test information table 2027.

[0212] In addition, the adjustment module 2035 fine-tunes the trained model using, for example, specification data, test category, test viewpoint, and confirmation items stored in the client test information table 2027 as input data, and factors and levels stored in the client test information table 2027 corresponding to the input information as correct output data.

[0213] The adjustment module 2035 also employs, for example, RAG and constructs in the storage unit 202 a searcher that searches the client test information table 2027 for information similar to the input specification data, test section, test viewpoint, and check item. The creation support module 2033 inputs, for example, the specification data, test section, test viewpoint, and check item into the searcher. The searcher acquires information similar to the input information from the client test information table 2027. The creation support module 2033 inputs the specification data, test section, test viewpoint, and check item, the information acquired from the client test information table 2027, and an instruction statement for outputting appropriate factors and levels based on the input information into the trained model. The trained model outputs factors and levels corresponding to the input information.

[0214] In addition, in the above embodiment, parameter tuning, fine tuning, and RAG were described as adjustment methods for processing using a trained model. However, adjustment methods are not limited to these. Although this requires more labor than parameter tuning, fine tuning, and RAG, a trained model for a customer may be created by new learning. Specifically, for example, the first trained model for customer 20251 is created by new learning. In this embodiment, multiple trained models are used. Therefore, even if the first trained model for customer 20251 is created by new learning, the labor required can be significantly reduced compared to creating all trained models by new learning. Because test perspectives are information that represents the content of a test, they are likely to reflect content specific to the customer and content specific to the business type / industry. Therefore, by creating the first trained model for customer 20251 by new learning, it is possible to effectively customize support for creating test design documents for the customer.

[0215] Customization of a trained model for a customer may be applied to one of the trained models used in multiple steps, or may be applied to multiple trained models. In this case, the method used to customize the trained model may be, for example, any one of parameter tuning, fine tuning, and RAG, or a combination of these. Furthermore, when multiple trained models are customized, the method used to customize each trained model may be different.

[0216] In addition, in the above embodiment, an example has been described in which the first trained model for customer 20251 and the client test information table 2027 are constructed in an environment dedicated to the customer in the memory unit 202. However, the environment in which the first trained model for customer 20251 and the client test information table 2027 are constructed is not limited to within the server 20. The first trained model for customer 20251 and the client test information table 2027 may be stored on the customer's server.

[0217] Fig. 21 is a block diagram showing another example of the overall configuration of system 1. System 1 in Fig. 21 includes a terminal device 10, a server 20, a client server 40, and an LLM system 30. The terminal device 10, the server 20, the client server 40, and the LLM system 30 are connected for communication via a network 80, for example.

[0218] The client server 40 has a function as a storage unit 402. The storage unit 402 has, for example, a client test information table 4027, a first trained model for customer 40251, and the like.

[0219] In addition, in the above embodiment, the creation assistance module 2033 performs the support for creating a test design document, but this is not limited to being performed by the creation assistance module 2033. For example, if the client test information table 4027 and the first trained model for customer 40251 are stored in the client server 40, the function of the creation assistance module may be incorporated into the client server 40.

[0220] In the above embodiment, an example of the mode in which the creation support module 2033 is provided in the server 20 has been described, but the present invention is not limited to this mode, and the terminal device 10 may be configured to have the functions of the creation support module 2033. In this case, for example, some or all of the information stored in the storage unit 202 may be stored in the terminal device 10.

[0221] <4 Basic computer hardware configuration> 22 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main memory device 92, an auxiliary memory device 93, and a communication IF (interface) 99. These are electrically connected to each other by a bus.

[0222] The processor 91 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0223] The main storage device 92 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0224] The auxiliary storage device 93 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0225] The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0226] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0227] <Basic functional configuration of computer 90> A description will be given of the functional configuration of a computer realized by the basic hardware configuration of a computer 90 shown in Fig. 22. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0228] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0229] The control unit is realized by the processor 91 reading various programs stored in the auxiliary storage device 93, expanding them in the main storage device 92, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that processes information.

[0230] The storage unit is realized by a main storage device 92 and an auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Furthermore, the processor 91 can allocate a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 in accordance with the programs. Furthermore, the control unit can cause the processor 91 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0231] A database refers to a relational database, which manages data sets called tables, which are structured by rows and columns, by relating them to each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated. Typically, each table has a column set as a key for uniquely identifying a record, but setting a key to a column is not essential. The control unit can cause the processor 91 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

[0232] The communication unit is realized by the communication IF 99. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 91 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0233] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0234] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0235] <Additional Notes> The matters described in the above embodiments will be supplemented below. (Appendix 1) A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the following steps: acquiring test knowledge related to a customer; adjusting, based on the acquired test knowledge, processing using one of a plurality of trained models used when creating test cases based on specifications, the trained models being trained to output specified elements that constitute test cases in response to input data related to the specifications; and providing the processing using the adjusted trained model as processing for the customer who provided the test knowledge. (Appendix 2) The trained model is a large-scale language model (Appendix 1). (Appendix 3) In the adjustment step, the trained model to be adjusted is a trained model trained to output a test perspective in response to input of a test category related to the specification (Appendix 1) or (Appendix 2). (Appendix 4) In the adjustment step, the trained model is trained to output predetermined elements that constitute a test case and index values ​​related to those elements in response to input data related to specifications, and the program described in any one of (Appendix 1) to (Appendix 3) adjusts the threshold value of the index value used as a condition for presenting the output element based on the acquired test knowledge. (Appendix 5) A program described in Appendix 4, wherein in the adjusting step, the trained model is trained to output multiple types of index values, and the threshold is adjusted for each type of index value. (Appendix 6) A program described in any one of (Appendix 1) to (Appendix 5), wherein in the adjusting step, the acquired test knowledge is used as training data to additionally train a trained model. (Appendix 7) A program described in any one of (Appendix 1) to (Appendix 6), in which, in processing using a trained model, data related to specifications is input into a searcher, and data similar to the input data related to the specifications from information stored in a predetermined database is output from the searcher and input into the trained model, and in the adjustment step, the acquired test knowledge is stored in the database. (Appendix 8) A program described in any one of (Appendix 1) to (Appendix 7), in which, in the acquiring step, test results of test cases created using multiple trained models are acquired as test knowledge. (Appendix 9) A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 8). (Appendix 10) An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in the invention according to any one of (Appendix 1) to (Appendix 8). (Appendix 11) A system comprising means for executing all steps performed in any of the inventions according to (Appendix 1) to (Appendix 8). [Explanation of symbols]

[0236] 1. System 10...Terminal device 120…Communications Department 13...Input device 131...Touch-sensitive devices 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output interface 25…Memory 26…Storage 29...Processor 30...LLM System

Claims

1. A program to be executed by a computer including a processor and a memory, the program causing the processor to: acquiring test knowledge relating to a customer; A step of adjusting, based on the acquired test knowledge, a process using one of a plurality of trained models used when creating a test case based on a specification, the trained models being trained to output predetermined elements that constitute a test case in response to input data related to the specification; Providing a process using the adjusted trained model as a process for the customer who provided the test knowledge; A program that executes the following.

2. The program according to claim 1 , wherein the trained model is a large-scale language model.

3. 2. The program according to claim 1, wherein in the adjusting step, the trained model to be adjusted is a trained model trained to output a test viewpoint in response to input of a test section related to the specification.

4. 2. The program according to claim 1, wherein in the adjusting step, the trained model is trained to output predetermined elements that constitute a test case and index values ​​related to the elements in response to input of data related to the specification, and the program adjusts a threshold value of the index value used as a condition for presenting the output element based on the acquired test knowledge.

5. 5. The program according to claim 4, wherein in the adjusting step, the trained model is trained to output multiple types of index values, and the threshold is adjusted for each type of index value.

6. The program according to claim 1 , wherein in the adjusting step, the acquired test knowledge is used as learning data to additionally learn the trained model.

7. In the processing using the trained model, data related to the specification is input into a searcher, and data similar to the input data related to the specification among information stored in a predetermined database is output from the searcher and input into the trained model; 2. The program according to claim 1, wherein the acquired test knowledge is stored in the database in the adjusting step.

8. 2. The program according to claim 1, wherein in the acquiring step, test results of test cases created using a plurality of trained models are acquired as the test knowledge.

9. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 8.

10. 10. An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any one of the inventions according to claims 1 to 8.

11. A system comprising means for executing all steps performed in the invention according to any one of claims 1 to 8.

Images