Programs, information processing devices, methods, and systems

Abstract

Generate code that is reliably executable in a specific test environment from a test design document written in natural language. [Solution] A program that causes a processor to execute the following steps: a step of obtaining definition information including the names and argument information of multiple functions in a group of operation functions, a template that defines the structure of source code executable using the group of operation functions, and test design information including test conditions for a natural language target to be tested; a step of generating a code generation prompt that includes instruction information instructing a large-scale language model to generate source code by using the definition information as constraints to limit the functions that can be used for code writing, using the template as a framework for code writing, and embedding the test design information as the target of code writing in the prompt; and a step of inputting the code generation prompt into the large-scale language model to cause the large-scale language model to generate source code.

Description

Technical Field

[0001] This disclosure relates to programs, information processing apparatuses, methods, and systems.

Background Art

[0002] In order to improve the efficiency of the test process in software development, in recent years, technology development using generative AI models has been promoted. For example, Patent Document 1 discloses a technology for generating a schema from input data such as natural language or images, and generating source code or test code using the schema and a common module (template). This technology aims to ensure the quality and consistency of code corresponding to multiple platforms by means of a schema-driven development approach.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in a schema-driven approach such as Patent Document 1 and general AI code generation tools, it has been difficult to generate code that can surely be executed for operations specific to a particular test target application.

[0005] An object of this disclosure is to generate code that can surely be executed in a specific test environment from a natural language test design document.

Means for Solving the Problems

[0006] To solve the aforementioned problems, a program according to one aspect of the present disclosure is a program for operating a computer equipped with a processor, which involves the following steps: obtaining definition information including the names and argument information of a plurality of operation function groups, which consist of a plurality of functions for performing operations on a test object; a template defining the structure of source code executable using the operation function group; and test design information including test conditions for the test object written in natural language; embedding the obtained definition information as constraints to limit the functions that can be used for code writing, the obtained template as the framework for code writing, and the obtained test design information as the target of code writing into prompts, thereby generating a code generation prompt including instruction information that instructs a large-scale language model to write test logic using the functions included in the definition information within a predetermined area of ​​the template; and inputting the generated code generation prompt into the large-scale language model, thereby causing the large-scale language model to generate source code including variable definition code to be passed as arguments to the operation function group within the predetermined area. [Effects of the Invention]

[0007] According to this disclosure, it is possible to generate code that is reliably executable in a specific test environment from a test design document written in natural language. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the overall system configuration. [Figure 2] This is a block diagram showing an example of a user terminal's hardware configuration. [Figure 3] This is a block diagram showing the functional components implemented by the control unit of the user terminal. [Figure 4] Block diagram showing an example of a server hardware configuration. [Figure 5]This is a block diagram showing the functional parts implemented by the server's control unit. [Figure 6] This figure shows an example of the data structure of a test design document table. [Figure 7] This figure shows an example of the data structure of a function definition master table. [Figure 8] This figure shows an example of the data structure of the execution result log table. [Figure 9] This flowchart shows an example of server operation. [Figure 10] This is a schematic diagram illustrating an example of an execution dashboard. [Figure 11] This is a schematic diagram showing an example of the results details screen. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. A processor may be expressed, for example, as processing circuitry. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be other types of processors such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core. Also, at least one processor may be a general-purpose processor or a purpose-specific processor.

[0011] Also, at least one processor may be a processor in a broad sense, such as a hardware circuit (e.g., FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

[0012] In the following description, an expression such as "xxx table" may be used to describe information from which an output can be obtained for an input. However, this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, "xxx table" can be referred to as "xxx information".

[0013] In the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0014] The program may be pre-installed in the information processing device shown below. For example, it may be on a recording medium (e.g., non-transitory) readable by the information processing device, and this program may be installed in the information processing device. Also, the program may be transmitted from a program distribution server to the information processing device and installed. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0015] In the following description, identification information for various objects is used. However, the identification information may be any information indicating a predetermined object, and the specific data is not limited to the embodiments. The identification information may be an identification number or an identifier including letters and symbols.

[0016] 〔Overview〕 The system according to this embodiment is a system that supports test automation in software development (especially Web browser tests, etc.). The system according to this embodiment takes as input a test design document in natural language (e.g., an Excel file) created by a quality assurance engineer or the like, and realizes the automatic execution of tests by using a large language model.

[0017] The greatest feature of the system according to this embodiment is that it has a constraint mechanism that changes the "free generation" by the large language model into the "assembly (composition)" of accurate components. That is, the system according to this embodiment dynamically injects (context injection) the definition information (signatures, arguments, etc.) of a group of operation functions that have been implemented and verified in advance into the prompt, and further presents (template constraint) the code skeleton (template) conforming to a test framework such as Pytest in the prompt. As a result, the large language model is forced to describe test logic (variable definition and function call) by combining only permitted functions within the specified framework. Therefore, errors due to hallucinations such as calls to non-existent methods are eliminated, and test code that always operates immediately after generation is output.

[0018] 〔Embodiment〕 <Overall Configuration of the System> FIG. 1 is a diagram showing the overall configuration of the system 1 according to this embodiment. The system 1 is a system that realizes a series of processes for automatically executing a test on a test target application based on test design information described in natural language and determining the result. As shown in FIG. 1, the system 1 mainly includes a user terminal 10, a server 20, and a generation AI system 30. These devices are connected via a network 80 so that they can communicate with each other.

[0019] Network 80 is a communication network for transmitting data between devices. Network 80 consists of any type of wired or wireless communication network, such as the Internet, LAN (Local Area Network), WAN (Wide Area Network), or Virtual Private Network (VPN), or a combination thereof.

[0020] In this embodiment, a client / server system configuration is described as the primary example, in which the user terminal 10 functions as a client, the server 20 plays a central role in the automated test execution process, and the generation AI system 30 provides intelligent processing. However, the configuration of this disclosure is not limited to this. For example, in another embodiment, the server 20 may contain the functions of the generation AI system 30, namely a large-scale language model. In this case, the server 20 independently performs source code generation and pass / fail judgment processing.

[0021] In yet another embodiment, the system 1 may consist of a user terminal 10 and a generation AI system 30. In this case, the user terminal 10 is responsible for the functions of the server 20 described herein (analysis of test design information, prompt generation, test execution management, pass / fail determination, etc.). When performing necessary processing, such as source code generation or pass / fail determination, the user terminal 10 calls the external generation AI system 30 via an API (Application Programming Interface) through the network 80.

[0022] In yet another embodiment, the user terminal 10 may be configured as a standalone information processing device equipped with all the functions of the server 20 and the generation AI system 30. In this case, the user terminal 10 independently performs all the processing related to this disclosure, namely, from acquiring test design information to generating source code, executing tests, and determining pass / fail, without communicating with other devices via the network 80. Such a configuration is technically feasible with the emergence of large-scale language models that can operate locally on high-performance PCs or smartphones in recent years, and is particularly useful in environments where connection to external networks is restricted for security reasons, or when test execution in an offline environment is required.

[0023] <User terminal hardware configuration> Figure 2 is a block diagram showing an example of the hardware configuration of a user terminal 10. The user terminal 10 is an information processing device that provides an interface for a user to use this system. The user terminal 10 is, for example, a desktop or notebook personal computer (PC), a smartphone, a tablet terminal, or other electronic device having information processing functions. As shown in Figure 2, the user terminal 10 comprises a control unit 101, a storage unit 102, a communication unit 103, an input unit 104, and an output unit 105. Each of these components is electrically connected to one another via signal lines such as a system bus. The control unit 101 is a processor that comprehensively controls the operation of the entire user terminal 10. The control unit 101 is composed of, for example, a CPU, an MPU (Micro Processing Unit), a GPU, or an SoC (System on a Chip) that integrates these functions. The control unit 101 realizes the various functions of the user terminal 10, which will be described later, by reading and executing the operating system (OS) and various application programs stored in the memory unit 102.

[0024] The storage unit 102 is a storage device that stores programs executed by the control unit 101, data used temporarily during program execution, and files created by the user. The storage unit 102 includes a main memory and an auxiliary storage device. The main memory is composed of volatile memory such as RAM (Random Access Memory). The auxiliary storage device is composed of non-volatile memory such as ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory.

[0025] The communication unit 103 is a communication interface for performing data communication with other devices, specifically the server 20, via the network 80. The communication unit 103 is composed of, for example, a network interface card (NIC) for connecting to a wired LAN, a wireless communication module for connecting to a wireless LAN (Wi-Fi), or a communication module for connecting to a mobile communication network (e.g., 5G, LTE).

[0026] The input unit 104 is an input device for receiving various operation inputs from the user. The input unit 104 consists of one or more of the following: a keyboard, mouse, touchpad, trackball or other pointing device, a touch panel integrated with a display, or a microphone that accepts voice input. The user can perform actions such as uploading test design documents, starting test execution, or operating buttons on the display screen via the input unit 104.

[0027] The output unit 105 is an output device for presenting various information to the user. The output unit 105 consists of one or more display devices, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or a speaker that outputs audio information. The output unit 105 displays a GUI (Graphical User Interface), such as the execution dashboard screen 1000 or the results details screen 1100, which will be described later.

[0028] <User terminal functional configuration> Figure 3 is a block diagram showing an example of a functional configuration realized by the control unit 101 of the user terminal 10 executing a predetermined program. As shown in Figure 3, the control unit 101 mainly functions as an operation reception unit 131, a transmission / reception unit 132, and a presentation control unit 133.

[0029] The operation reception unit 131 has the function of receiving various operation instructions from the user via the input unit 104. Instructions received by the operation reception unit 131 include, for example, instructions to upload a test design file related to the application under test, instructions to execute a specific test case, instructions to display detailed test results, or instructions to change the settings of System 1. The operation reception unit 131 generates information according to the received instructions and passes it to the transmission / reception unit 132.

[0030] The transmitting / receiving unit 132 controls the communication unit 103 and has the function of sending and receiving various information with the server 20. For example, the transmitting / receiving unit 132 sends instruction information received by the operation reception unit 131 or test design document files stored in the storage unit 102 to the server 20. The transmitting / receiving unit 132 also receives information from the server 20, such as test execution results, progress status, and display screen data.

[0031] The display control unit 133 has the function of controlling the display of the output unit 105 and presenting various information to the user. Based on the information received by the transmitting / receiving unit 132, the display control unit 133 generates screens such as the execution dashboard screen 1000 shown in Figure 10 or the results detail screen 1100 shown in Figure 11, and displays them on the output unit 105. This allows the user to visually confirm the overall status of the test, the success or failure of individual test cases, and the detailed reasons for their success or failure.

[0032] <Server Hardware Configuration> Figure 4 is a block diagram showing an example of the hardware configuration of server 20. Server 20 is an information processing device that plays a central role in the automated test execution process according to this embodiment. Server 20 is, for example, a physical server machine, a virtual server built using virtualization technology, or a server instance provided on a cloud computing environment. As shown in Figure 4, server 20 comprises a control unit 201, a storage unit 202, a communication unit 203, and an input / output IF 204.

[0033] The control unit 201 is a processor that comprehensively controls the operation of the entire server 20, and is composed of, for example, one or more CPUs, MPUs, etc. The control unit 201 executes programs stored in the memory unit 202 to realize functions such as the reception unit 211, information acquisition unit 212, prompt generation unit 213, test execution unit 215, and pass / fail determination unit 216, which will be described later.

[0034] The memory unit 202 is a storage device that permanently or temporarily stores the program executed by the control unit 201 and various data necessary for processing. The memory unit 202 is composed of a storage medium such as ROM, RAM, HDD, or SSD. The memory unit 202 stores characteristic data used in the processing of this disclosure, such as the test design table 221, function definition master table 222, execution result log table 223, and code template group 224, which will be described later.

[0035] The communication unit 203 is a communication interface for data communication with other devices, specifically the user terminal 10 and the generation AI system 30, via the network 80. The communication unit 203 is composed of, for example, a NIC.

[0036] The I / O IF204 is an interface for inputting and outputting data between the server 20 and peripheral devices such as keyboards, mice, and displays. It is used when a system administrator directly operates the server 20, for example.

[0037] <Server Functional Configuration> Figure 5 is a block diagram showing an example of a functional configuration realized by the control unit 201 of the server 20 executing a predetermined program. As shown in Figure 5, the control unit 201 mainly functions as a reception unit 211, an information acquisition unit 212, a prompt generation unit 213, a source code generation control unit 214, a test execution unit 215, and a pass / fail determination unit 216.

[0038] The reception unit 211 has the function of receiving test design information sent from the user terminal 10. The test design information is data extracted from a test specification document created by a human, and includes the purpose of the test, the procedure, the expected value, and the preconditions (test conditions). The test design information is provided in a predetermined format, such as an Excel® file, a CSV (Comma Separated Values) file, or a JSON (JavaScript Object Notation) file. Specifically, this includes structured data such as "Category: Chat function," "Condition: Log in as administrator," and "Procedure: Press the send button," which is read from an Excel file. The reception unit 211 analyzes the received test design information and stores it in the test design document table 221, which will be described later.

[0039] The information acquisition unit 212 has the function of acquiring various information necessary for generating source code. The information acquisition unit 212 reads information necessary for generating prompts, executing tests, and determining pass / fail from the test design table 221, the function definition master table 222, and the code template group 224, etc. Specifically, the information acquisition unit 212 acquires definition information for operation function groups, source code templates, and test design information. Here, the set of operation functions is a collection of functions, methods, or APIs that have been pre-implemented in a programming language and whose operation has been verified in order to perform specific operations on the target of test, such as clicking, inputting, waiting, etc. For example, this would include actual code such as "click_button(target_name)" or "input_text(field_name, value)" implemented in Python.

[0040] Definition information is metadata necessary for large-scale language models to correctly invoke a set of operational functions. Specifically, definition information includes signature information such as the function name, list of arguments, type hints, and function description (docstring). For example, this includes data in JSON format that is automatically extracted from source code using Python's inspect module, etc.

[0041] A template is text data that forms the backbone of the generated source code. A template includes import statements for necessary libraries, class definitions, test setup processes (fixtures), and areas (placeholders) where a large-scale language model should write test logic. For example, a template might include a Python code template containing "def test_case(): pass".

[0042] Furthermore, the information acquisition unit 212 has the function to acquire, as needed, analysis examples for converting natural language descriptions into program variables, or knowledge data obtained from past test executions. Analysis examples are input / output pairs that show the thought process for converting natural language descriptions into program variables or values. For example, a conversion rule such as "Input: 'With administrator privileges' → Output: role="admin"" is an example. Knowledge data is structured data of solutions to problems that occurred during past test executions. For example, data that associates the issue of "delete button response delay" with the solution "wait 5 seconds after pressing" is an example.

[0043] The prompt generation unit 213 has the function of generating code generation prompts that serve as instructions for the large-scale language model of the generation AI system 30 by combining various pieces of information acquired by the information acquisition unit 212. The prompt generation unit 213 embeds the acquired definition information into the prompt as constraints to limit the functions that can be used for code writing, the acquired template as a framework for code writing, and the acquired test design information as the target of code writing. This restricts the free generation capability of the large-scale language model and constructs a prompt containing instruction information to ensure that executable source code is generated in a specific test environment. This instruction information is information for instructing the generation of source code and is usually composed of instruction statements in text format, but may also include image information or audio information that indicates the content of the instruction, as long as it is in a format that the large-scale language model can interpret.

[0044] Furthermore, the prompt generation unit 213 has the function of embedding knowledge data acquired by the information acquisition unit 212 into the prompt. Specifically, if knowledge data such as "[ADD after Step 2]: Wait 5 seconds" is written in the remarks column of the test design table 221, the prompt generation unit 213 interprets this and includes the instruction "Add a 5-second wait process after the execution of test procedure 2" in the code generation prompt. In this way, know-how gained from past failures is automatically reflected in future code generation.

[0045] The source code generation control unit 214 inputs the code generation prompt generated by the prompt generation unit 213 to the generation AI system 30 and has the function of causing the large-scale language model to generate source code. The source code generation control unit 214 controls the large-scale language model to write test logic, including variable definition code to be passed as arguments to the operation function group, within a predetermined area of ​​the template, in accordance with the constraints in the prompt. As a result, hallucinations such as syntax errors or reference errors are eliminated, and source code that can be executed immediately without manual correction is generated.

[0046] The test execution unit 215 has the function of executing the source code generated by the generation AI system 30 and automatically performing tests on the application under test. The test execution unit 215 operates the application under test using a test framework such as Playwright or Selenium in an isolated execution environment such as a sandbox environment. The application under test is not limited to a web application running on a PC, but may also be a mobile application running on a smartphone or an emulator / simulator.

[0047] Furthermore, the test execution unit 215 has a function to analyze screenshots of the screen under test that are displayed during source code execution. In this analysis, the test execution unit 215 extracts the visual characteristics of the elements on the screen. The elements on the screen are components that are displayed on the GUI of the application under test and that the user can perceive or interact with, such as buttons, text fields, checkboxes, icons, images, or links.

[0048] Visual features include at least text information, graphic information, or location information. Text information is information that can be recognized as characters that an element on the screen possesses, such as a button label ("Login"), placeholder text in an input field, or message text displayed on the screen. Graphic information is information about the visual shape, color, or pattern of an element on the screen that is not text, such as a trash can icon, a magnifying glass icon, or a warning mark with a specific color. Location information is information that indicates the relative position of an element on the screen to other elements, such as "the input field to the right of the label 'Username'" or "the navigation bar below the 'Header Logo'."

[0049] The test execution unit 215 has a function to identify the target of operation included in the test design information based on the extracted visual features. The target of operation is an element on the screen that the user should operate in the execution procedure included in the test design information, such as a "login button" or a "username input field".

[0050] Furthermore, the test execution unit 215 has a function to perform recovery processing when the target of operation cannot be identified. Recovery processing is, for example, a process of searching for an area on the screen that includes a specific area that is presumed to correspond to the target of operation, or a screen state change process to improve at least one of the visibility and operability of the target of operation. An area that includes a specific area is a part of the screen and its surroundings where the target of operation is presumed to exist when the target of operation cannot be directly identified. For example, this includes the area around the label text of the target of operation, or a specific cell and its adjacent cells when the entire screen is divided into a grid. Screen state change processing is, for example, changing the click position, adjusting the screen scroll, or closing a popup.

[0051] The test execution unit 215 has the function of acquiring and structuring information indicating a solution to a failure that occurs during the execution of the source code, and storing it as knowledge data. The acquisition of the solution is performed, for example, through an interactive interface in which the test execution unit 215 transmits and presents the details of the failure to the user terminal 10 and accepts input of the solution from the user.

[0052] The acquired solutions are structured into a reusable format. One structuring method is to format the data in JSON format with keys such as "what" for the failure description, "action" for the solution, and "related_cases" for related test cases. Other methods include adding the data as records to a database table with columns such as failure ID, category, event, and solution, or describing the data in XML format based on a specific schema. The test execution unit 215 accumulates the structured knowledge data by appending it to the remarks column of the test design document table 221 for the related test cases.

[0053] The test execution unit 215 has a function to output an operation log that records the processing content, results, and the state of the test target at each step of the execution process. The operation log is JSON formatted data that includes, for example, the name of the function executed, arguments, timestamp, success / failure status, error message, and the path to the screenshot taken at each step.

[0054] Furthermore, the test execution unit 215 has a function to perform the initial test execution in a learning mode, determining the next operation to be performed according to the state, and to record an operation log if successful. In addition, the test execution unit 215 also has a function to convert the recorded operation log into a predetermined test code based on predetermined conversion rules. The conversion rules are predetermined rules for converting the recorded operation log (e.g., in JSON format) into static test code (e.g., Playwright code). For example, a rule would be to convert the description "{"action": "click", "target": "login button"}" in the operation log into the code line "page.locator("text=login button").click()".

[0055] The pass / fail determination unit 216 has the function of determining whether a test has passed or failed by comparing the operation log output by the test execution unit 215 with the expected value included in the test design information. In this embodiment, the processing of the test execution unit 215, which executes the procedure, and the processing of the pass / fail determination unit 216, which determines whether the test has passed or failed, are executed independently of each other. The expected value is information included in the test design information that indicates the normal state that the test target should take after a specific operation procedure is completed. For example, this includes natural language descriptions such as "the message 'Saved' is displayed" or "the system transitions to the dashboard screen."

[0056] The pass / fail determination unit 216 has a function to output a pass / fail determination result along with a confidence score indicating the degree of certainty of the determination result. The confidence score is a numerical index that quantifies the degree of certainty of the determination model itself regarding the pass / fail determination result, and is expressed, for example, as a floating-point number between 0 and 1. Possible methods for calculating the confidence score include, for example, using the log probability when the determination model generates a specific token indicating the determination result, having the determination model itself self-evaluate its confidence level in the determination and output it as a numerical value, or scoring multiple evaluation items such as "expected keyword match rate" and "presence or absence of error messages," and using the sum or average of these scores as the confidence score. Furthermore, if this confidence score is lower than a predetermined threshold, the pass / fail determination unit 216 also has a function to determine pass or fail for each of several different large-scale language models and to determine the final pass or fail based on each determination result.

[0057] <Configuration of the Generative AI System> The generative AI system 30 is a system that performs intelligent processing such as source code generation or pass / fail judgment in response to requests from the server 20. The generative AI system 30 is equipped with one or more large-scale language models (LLMs). The large-scale language model is a neural network model pre-trained on a vast amount of text data and source code data, and has the ability to generate natural-sounding text or source code that resembles human-written text in response to input information called prompts.

[0058] In this embodiment, the generation AI system 30 can be configured to include two types of models functionally: an "execution control model" and a "decision model". The execution control model receives a code generation prompt generated by the prompt generation unit 213 of the server 20 as input and is responsible for generating executable source code according to the constraints and templates in the prompt.

[0059] On the other hand, the judgment model receives operation logs (especially visual information such as screenshots) output by the test execution unit 215 and expected values ​​(text information) included in the test design information as input, and is responsible for comparing and matching the two to determine whether the test passes or fails. For this reason, it is preferable that the judgment model is a multimodal large-scale language model (Vision LLM) that can handle both text and images. Note that the execution control model and the judgment model may be physically different models, or a single large-scale language model may perform both roles.

[0060] <Information stored in the server's memory> Next, the main data structures stored in the storage unit 202 of the server 20 will be described. As shown in Figure 4, the storage unit 202 has areas for storing the test design document table 221, the function definition master table 222, the execution result log table 223, and the code template group 224.

[0061] Figures 6 to 8 show the data structure of each table stored by the server 20. Note that Figures 6 to 8 are examples and do not exclude data not shown. Furthermore, even data listed in the same table may be stored in separate memory areas within the storage unit 202.

[0062] The databases shown in Figures 6 to 8 refer to relational databases, which are used to manage data sets called tables, which are structurally defined by rows and columns, by associating them with each other. In a database, tables are called tables, the columns of a table are called columns, and the rows of a table are called records. In a relational database, relationships can be established and linked between tables.

[0063] Typically, each table has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit 201 can add, delete, or update records in specific tables stored in the storage unit 202 according to various programs.

[0064] Figure 6 shows an example of the data structure of the test design document table 221. The test design document table 221 is a table that stores structured data extracted from test design documents (Excel files, etc.) uploaded by users in natural language. The test design document table 221 includes columns such as test ID, test category, conditions, execution procedure, expected value, and remarks.

[0065] The "Test ID" column stores an identifier (e.g., "001", "002") to uniquely identify each test case on System 1. The "Test Category" column stores the type of test or the target function (e.g., "Functional Test", "Home Screen"). The "Conditions" column stores the prerequisites and input patterns for executing the test (e.g., "Account Type: Operations Team", "Publication Status: Public"). The "Execution Procedure" column stores information describing the operation procedure for the test target in natural language (e.g., "1. Log in to Urakata-san as an operations member, 2. Transition to the Genius-kun editing screen"). The "Expected Value" column stores the conditions for determining that the test was successful (e.g., "Genius-kun is displayed on the home screen"). The "Notes" column stores notes from the design phase, correction instructions based on knowledge data obtained during test execution, etc. For example, as shown in Figure 6, it describes what additional processing should be performed after each step, such as "[ADD after Step 2]: Wait 5 seconds [REF: TEN-003]", and also includes the ID of the knowledge data that serves as the basis for that processing.

[0066] Figure 7 shows an example of the data structure of the function definition master table 222. The function definition master table 222 is a table that stores definition information for a group of operation functions that have been pre-implemented in a programming language and whose operation has been verified in order to perform specific operations on the system under test. The function definition master table 222 includes columns such as function ID, function name, argument information, and function description.

[0067] The "Function ID" column stores an identifier that uniquely identifies each operation function within the system. The "Function Name" column stores the name of the function to be called in the source code (e.g., "click", "login", "input_text"). The "Argument Information" column stores a list of arguments that the function should accept, along with type hints (e.g., "target_name: str", "role"). The "Function Description" column stores descriptive text, i.e., docstring information, that indicates what operation the function performs (e.g., "Click the element with the specified name", "Log in with the specified privileges"). This definition information is embedded in the prompt as constraints when the large-scale language model generates code.

[0068] Figure 8 shows an example of the data structure of the execution result log table 223. The execution result log table 223 is a table that stores evidence of the results of source code execution in the test execution environment. The execution result log table 223 includes columns such as test case ID, execution date and time, code used, operation procedure result, and judgment result.

[0069] The "Test Case ID" column stores an identifier associated with the test ID in the test design table 221. The "Execution Date and Time" column stores the date and time when the test case was executed. The "Code Used" column stores the text data of the source code that was actually executed, with the code embedded in the placeholders of the code template group 224 by the large-scale language model. The "Operation Procedure Result" column stores information indicating the result when each step of the source code was executed (e.g., success or failure of each step, the path of the acquired screenshot, and error logs such as "Timeout exceeded") in JSON format or similar. The "Judgment Result" column stores the pass / fail result (e.g., "OK", "NG") and the confidence level of the judgment (e.g., a confidence score such as "0.95") that were ultimately determined by comparison with the expected value.

[0070] In addition, the server 20's memory also stores a set of code templates 224 in file or text data format, separate from the aforementioned tables. The code template set 224 is a collection of text data that forms the backbone of the generated source code. Specifically, it is managed as a template file containing import statements for libraries essential for running the tests, class definitions required by the test framework, fixture code, and placeholders that specify the areas where the large-scale language model should actually write the test logic (for example, the location where "pass" is written in the Python code). This ensures that the generated source code always has an executable structure.

[0071] <Operation> Figure 9 is a flowchart illustrating an example of server 20 operation. The premise is that a user operates user terminal 10 to upload test design information to server 20. This test design information is, for example, a spreadsheet file containing the test objective, test conditions, execution procedure described in natural language, and expected results (expected values).

[0072] First, in step S11, the reception unit 211 of the server 20 receives test design information transmitted from the user terminal 10. The reception unit 211 analyzes the received test design information and stores it as structured data for each test case in the test design document table 221 provided in the storage unit 202.

[0073] Next, in step S12, the information acquisition unit 212 of the server 20 executes a process to acquire various information necessary for source code generation from the storage unit 202. Specifically, the information acquisition unit 212 reads the definition information of the operation function group from the function definition master table 222 provided in the storage unit 202.

[0074] Next, in step S13, the information acquisition unit 212 of the server 20 reads a code template that defines the structure of the source code from the code template group 224 provided in the storage unit 202. This code template ensures that the generated code conforms to the conventions of a specific test framework and is in an immediately executable format.

[0075] Note that the process of reading definition information in step S12 and the process of reading code templates in step S13 are not limited to this order and may be executed in the reverse order or in parallel. In addition to these processes, the information acquisition unit 212 acquires test design information corresponding to the test case from which source code is to be generated, from the test design table 221 stored in step S11. This test design information includes at least test conditions and execution procedures written in natural language.

[0076] Next, in step S14, the prompt generation unit 213 of the server 20 generates a code generation prompt, which is instruction information for the large-scale language model, using the various information acquired in steps S12 and S13. The generation of this code generation prompt is one of the core processes of this embodiment.

[0077] Specifically, the prompt generation unit 213 embeds the definition information of the acquired set of operation functions into the prompt as "constraints" to limit the functions that can be used for code writing. This suppresses so-called hallucination, where the large-scale language model calls non-existent functions or uses unintended functions, and ensures that the generated code consists only of executable functions. The prompt generation unit 213 also embeds the acquired code template into the prompt as a "framework" for code writing. This means instructing the large-scale language model not to write code from scratch, but to fill in only specific predetermined areas of the provided template (for example, the parts marked "pass"). This ensures the structural consistency and executableness of the entire generated code. Furthermore, the prompt generation unit 213 embeds the acquired test design information into the prompt as the "target" for code writing. This specifically instructs the large-scale language model on what test logic should be implemented.

[0078] Here, there are several variations in how the prompt generation unit 213 generates code generation prompts. For example, the prompt generation unit 213 may generate a single prompt by simply concatenating definition information, templates, and test design information as text. Alternatively, the prompt generation unit 213 may embed this information within the prompt as structured data such as JSON format. This method allows the large-scale language model to more clearly recognize the role of each piece of information (constraints, frameworks, targets) and accurately interpret the intent of the instructions. Another method is for the prompt generation unit 213 to clearly separate each section of the prompt. For example, by providing headings such as "System Instruction," "Context: Constraints," "Input Data: Target for Code Description," and "Template: Framework for Code Description," the meaning and role of each piece of information can be presented hierarchically to the large-scale language model.

[0079] In this way, the prompt generation unit 213 constructs a structured code generation prompt that integrates three elements: "constraints (available tools)," "framework (templates to be filled in)," and "target (specifications to be implemented)." This approach transforms the free generation capabilities of large language models into the assembly of precise components, solving reliability problems in test automation.

[0080] Finally, in step S15, the source code generation control unit 214 of the server 20 inputs the generated code generation prompt to the generation AI system 30, causing it to generate source code for the large-scale language model. The generation AI system 30 executes the process according to the precise instructions of the input code generation prompt.

[0081] Specifically, the generating AI system 30 interprets the test design information (especially the test conditions) embedded in the prompt. For example, it semantically understands the natural language description "Account type: Operation team" and generates variable definition code for handling this in the program, such as "role="operation_team"". Next, the generating AI system 30 assembles the test logic based on the execution procedure of the test design information and the interpreted test conditions. At this time, it generates code that calls only permitted operation functions (e.g., the "login" function or the "click" function) derived from the function definition master table 222, in accordance with the constraints embedded in the prompt. Then, it generates code that passes the variables from the previously generated variable definition code as arguments to these functions (e.g., "login(role)"). The generating AI system 30 completes the final source code by inserting the variable definition code and test logic code generated in this way into the designated area of ​​the code template embedded in the prompt.

[0082] The completed source code is returned from the generation AI system 30 to the server 20. The source code generation control unit 214 stores the returned source code in the storage unit 202 and uses it in the subsequent test execution process.

[0083] Through the above flow, it is possible to automatically generate source code from a test design document written in natural language that is reliably executable in a specific test environment and accurately reflects the intent of the test conditions.

[0084] <Summary> As described above, according to this embodiment, it is possible to generate source code that can be reliably executed in a specific test environment from a test design document written in natural language. In particular, this embodiment is characterized by not simply giving a free instruction to a large-scale language model to "generate test code," but by imposing a dual constraint of "context injection" and "template constraint."

[0085] Context injection, that is, embedding definition information of pre-verified operational function sets into prompts, limits the functions available to large-scale language models and effectively suppresses hallucination, such as calling non-existent functions. Furthermore, template constraint, that is, requiring only a predetermined area of ​​the template that forms the backbone of the source code to be written, ensures that the generated code always conforms to the conventions of a specific test framework and is structurally correct and free of syntax errors.

[0086] This dual constraint allows large-scale language models to focus their advanced natural language comprehension and reasoning capabilities on the specific task of generating highly reliable code. As a result, the generated source code can be executed immediately without requiring manual modification, significantly reducing the effort required for implementing and operating test automation. Furthermore, since test code that always follows the latest specifications is regenerated simply by modifying the test design document, maintenance of the test code itself becomes unnecessary, greatly contributing to an accelerated development cycle.

[0087] [Variation] <First variation> This modification relates to a process for improving the accuracy of interpreting test conditions written in natural language. In this modification, the information acquisition unit 212 of the server 20 acquires, in addition to the basic processing, "analysis examples" from the storage unit 202 that show specific input / output pairs for converting natural language descriptions into program variables. These analysis examples are example data for few-shot prompting that include the thought process, such as "Input: 'Logged in as administrator' → Thought: 'Administrator' corresponds to role="admin". → Output code: role = "admin"".

[0088] When generating a code generation prompt, the prompt generation unit 213 of the server 20 further embeds the acquired analysis example as "conversion logic" into the prompt, in addition to the three basic pieces of information. By referring to this analysis example, the generation AI system 30 can more accurately interpret the diverse natural language expressions contained in the test design information. For example, even if the test conditions contain expressions that are different from the analysis example but semantically similar, such as "use Admin account" or "operate with operational privileges," the large-scale language model can understand the context and infer and generate appropriate variable definition code such as "role = "admin"" or "role = "operation"" respectively.

[0089] This modified approach improves the ability to flexibly handle inconsistencies or ambiguities in test design documents—which were difficult to address with fixed rule-based methods such as regular expressions—and to accurately map test conditions to program variables. This further enhances the accuracy of source code generation.

[0090] <Second variation> This modified example relates to the process of identifying the target of operation when executing the generated source code. In this modified example, the test execution unit 215 of the server 20 executes the generated source code. In this execution process, the test execution unit 215 not only relies on conventional DOM (Document Object Model) selectors (such as IDs or class names), but also performs a process of identifying the target of operation based on visual information.

[0091] Specifically, the test execution unit 215 takes a screenshot of the screen displaying the application under test immediately before executing a specific operation in the source code (for example, "clicking the login button"). This screenshot is sent to the generating AI system 30 (in particular, a multimodal model with image recognition capabilities). The generating AI system 30 analyzes the received screenshot and extracts the visual features of each element on the screen. These visual features include at least the following three pieces of information: First, the text information of the element (for example, the string "Login" displayed on the button). Second, the image information of the element (for example, the floppy disk icon indicating the save button). Third, the positional information indicating the relative position of the element to other elements (for example, "the element to the right of the username input field").

[0092] The generation AI system 30 compares the extracted visual features with the description of the target to be operated on (e.g., "login button") included in the test design information. This comparison identifies which area on the screenshot is the target to be operated on. After identification, the test execution unit 215 performs operations such as clicking or inputting on the identified area. According to this modified example, even if the HTML structure or element IDs change due to updates to the application under test, the target to be operated on can be flexibly identified, just as a human would judge from the appearance of the screen. This significantly improves the robustness of the test and reduces maintenance costs.

[0093] <Third variation> This modification relates to autonomous recovery processing in the case where the target of operation cannot be identified in the processing of the second modification. When the test execution unit 215 of the server 20 fails to identify the target of operation after attempting the visual-based identification process described in the second modification, it does not immediately fail the test but instead performs autonomous recovery processing. This recovery processing consists of two types: a search process and a screen state change process.

[0094] One example of a search process is "hover search." This process involves stepwise searching for areas where the target object is most likely to exist. First, the test execution unit 215 identifies a "reference point," such as text or a label related to the target object, from within the screen. Next, it divides a narrow area around the reference point (e.g., 20 pixels around) into a virtual grid, moves the mouse cursor to the midpoint of each grid (hover), and observes the element's reaction (e.g., a change in color or the display of a tooltip). If the target is not found, the search range is expanded to a wider area (e.g., 50 pixels around) and the same hover search is performed again. In this way, the system persistently attempts to find the target object by gradually expanding the search range.

[0095] The screen state change process is a process to improve the visibility (whether it is displayed on the screen) or operability (whether it is clickable) of the target object. For example, if the target object is at the bottom of the screen and may not be currently displayed, the test execution unit 215 will execute a process to scroll the screen downwards. Also, if the target object is hidden by other elements (e.g., a header or a popup window) and may not be clickable, the test execution unit 215 will adjust the scroll or attempt an operation to close the popup (e.g., pressing the Esc key or clicking the "Close" button). Furthermore, if the first click does not produce a response, the process will also include shifting the click position slightly from the center of the element and clicking again.

[0096] The test execution unit 215 repeats these recovery processes up to a predetermined number of times (for example, three times). Only if the target of the operation cannot be identified or operated after these attempts, the test execution unit 215 considers that step a failure and interrupts the process.

[0097] According to this modification, the system can autonomously address minor issues that may frequently occur during test execution, such as temporary screen display glitches or the appearance of unexpected pop-ups, thereby improving the test continuation rate.

[0098] <Fourth variation> This modification relates to a process that improves the objectivity and accuracy of the judgment by separating the test execution process from the pass / fail judgment process. In this modification, the test execution phase is clearly separated into an "output step" as the first process and a "judgment step" as the second process.

[0099] These steps are performed using at least one of independent prompts and / or independent AI agents. Specifically, they are performed by sending separate, context-decoupled, independent prompts to a single large-scale language model for runtime and decision-making, by calling separate, logically or physically separated, independent AI agents (execution-only agents and decision-only agents), or by a combination thereof.

[0100] First, the test execution unit 215 of the server 20 executes the output step. In the configuration using an independent prompt, the test execution unit 215 sends a first prompt (execution prompt) to the generating AI system 30 that contains only a role definition such as "You are the screen operation executor" and execution instructions. This first prompt does not contain any information regarding expected values. On the other hand, in the configuration using an independent AI agent, the test execution unit 215 calls an "execution AI agent" that has been fine-tuned or configured specifically for screen operation tasks and delegates the processing. The test execution unit 215 (or execution AI agent) executes the generated source code and performs a series of operations on the application under test.

[0101] In this process, the test execution unit 215 (or the execution AI agent) does not make any determination as to whether the results of each operation meet the expected values ​​of the test design document. Instead, it records the details of the operations performed, screenshots of the screen after the operations, and objective facts observed on the screen (for example, "the message 'Saved' was displayed") as an operation log in the execution result log table 223 of the storage unit 202, in chronological order.

[0102] Next, the pass / fail determination unit 216 of the server 20 executes the determination step. In the configuration using an independent prompt, the pass / fail determination unit 216 resets the output step history (context) and sends a second prompt (determination prompt) to the generating AI system 30, which includes a determination instruction such as, "You are a quality assurance auditor. Determine whether the following logs meet the expected values." On the other hand, in the configuration using an independent AI agent, the pass / fail determination unit 216 calls a "determination AI agent" that operates as a separate model instance from the aforementioned execution AI agent.

[0103] The pass / fail determination unit 216 inputs the operation log output by the test execution unit 215 and the "expected value" of the test case obtained from the test design document table 221 to the generation AI system 30 (or judgment AI agent). The generation AI system 30 (or judgment AI agent) compares the input operation log (especially the screenshot and observed facts) with the text of the expected value and objectively evaluates whether the expected value has been met. For example, if the expected value is "the dashboard screen is displayed correctly and all menu items can be confirmed," the generation AI system 30 (or judgment AI agent) comprehensively verifies whether the screen shown in the screenshot is the dashboard screen, whether all expected menu items are displayed, whether there are any display errors, etc., and determines whether the test is pass or fail (OK / NG / needs verification). The judgment result is recorded in the execution result log table 223 by the pass / fail determination unit 216.

[0104] In this way, execution bias in large-scale language models is eliminated by separating prompts for execution tasks and decision tasks, or by separating the AI ​​agents themselves that run on them (or by combining both).

[0105] This modified approach eliminates the "execution bias" that can occur when large-scale language models become so focused on "executing procedures" that they neglect to compare with expected values, overlooking bugs and making inaccurate pass / fail judgments. This results in rigorous and reliable test judgments.

[0106] <Fifth variation> This modification further enhances the judgment step of the fourth modification and relates to a process for further improving the reliability of the judgment. In this modification, when the pass / fail judgment unit 216 of the server 20 requests a judgment from the generation AI system 30, the generation AI system 30 outputs a "confidence level" for the judgment as a numerical value (for example, a value between 0.0 and 1.0) along with the pass / fail judgment result (OK / NG). This confidence level indicates the degree of certainty of the judgment result, and the higher the value, the more confident the judgment is. Figure 8 shows an example of an execution result log table 223 including the confidence level.

[0107] The pass / fail determination unit 216 determines whether the confidence level returned from the generating AI system 30 is lower than a predetermined threshold (for example, 0.80). If the confidence level is equal to or greater than the threshold, the determination result is adopted as the final result. On the other hand, if the confidence level is lower than the threshold, the pass / fail determination unit 216 determines that the determination is ambiguous and that the determination by a single model may be incorrect, and starts the "multi-model consensus" process.

[0108] In the multi-model consensus process, the pass / fail determination unit 216 prepares multiple large-scale language models (e.g., Model A, Model B, Model C) that have different architectures or training data from each other. Then, it independently determines the pass / fail status of each of these multiple large-scale language models, using the same operation log and expected value as input. The pass / fail determination unit 216 aggregates the determination results obtained from each model (e.g., "OK" from Model A, "NG" from Model B, and "OK" from Model C). Then, based on predetermined consensus rules such as majority voting, it determines the final pass / fail status of the test. In the above example, since "OK" is the majority, the final determination result is "OK".

[0109] This modified method reduces the risk of hallucination or oversight due to the characteristics of a single model in gray areas where judgment is difficult, such as when the description of expected values ​​is ambiguous or the display on the screen is subtle. This makes it possible to obtain highly reliable judgment results that are close to double-checked by a human.

[0110] <Sixth variation> This modified version relates to a self-evolving process in which the system learns from problems discovered during test execution and their solutions, and autonomously improves itself. This modified version specifies the processing that occurs when a failure (e.g., a timeout error or element mismatch) occurs during source code execution by the test execution unit 215 of the server 20. When a failure occurs, system 1 obtains information indicating a solution to that failure. This information may be obtained in an interactive format, for example, by querying the user for a solution via the user terminal 10 and accepting their input, or, if the autonomous recovery process described in the third modified version is successful, the successful procedure may be adopted as the solution.

[0111] When information indicating a solution is obtained, the information acquisition unit 212 of the server 20 stores that information in the storage unit 202 as structured "knowledge data". The knowledge data is written in JSON format, for example, and includes information such as a unique ID, the circumstances under which the problem occurred, a specific solution (action), and the ID of the related test case. For example, if the problem is "a timeout occurs after pressing the delete button" and the solution obtained is "wait 5 seconds after pressing the button", this is recorded as knowledge data.

[0112] After this knowledge data is accumulated, special processing is performed when the prompt generation unit 213 generates a code generation prompt for a subsequent test case or the next test execution. In addition to the three basic pieces of information (definition information, template, and test design information), the prompt generation unit 213 reads relevant knowledge data from the storage unit 202 and embeds it in the prompt. The generation AI system 30 generates source code taking this knowledge data into consideration. In the previous example, if the test procedure contains the description "press the delete button," the generation AI system 30 refers to the knowledge data and generates source code that automatically inserts a 5-second waiting process such as "wait(5)" immediately afterward.

[0113] According to this modified version, with each test run, know-how regarding the specific quirks or points of caution of the application under test is automatically accumulated within System 1, and this is utilized in future test code generation. This enables self-evolving test automation, where System 1 becomes smarter on its own without human intervention.

[0114] <7th variation> This modification relates to a process that converts the knowledge gained from the initial test run into reusable assets, thereby improving the efficiency of the test. In this modification, there are two test execution modes: "learning mode" and "playback mode." The initial test run is performed in "learning mode." In this mode, the test execution unit 215 of the server 20 does not execute a predetermined series of procedures, but rather proceeds with the test by dynamically determining the next operation to be performed according to the state of the test target. This is an exploratory execution process involving trial and error, making full use of the visual-based identification in the second modification and the autonomous recovery process in the third modification.

[0115] If, as a result of execution in this learning mode, the test is judged to pass in the judgment step of the fourth modification, the pass / fail judgment unit 216 of the server 20 records in detail the series of successful operation logs that led to the pass in a replayable format. This operation log includes the entire history of what operations (clicks, inputs, etc.) were performed on which elements, in what order. Next, the source code generation control unit 214 of the server 20 performs a process to convert the recorded operation log into static test code based on predetermined conversion rules. These conversion rules are defined, for example, as "the 'browser_click' action in the operation log is converted to the 'page.locator(...).click()' method of Playwright." This conversion automatically generates highly reproducible test code (e.g., a Python script) with a predetermined series of operation procedures. Subsequent executions of the same test case are performed in "playback mode." In this mode, the test execution unit 215 simply executes the static test code generated by the conversion, rather than performing dynamic inference as in learning mode.

[0116] This modified approach allows for a reduction in the high inference cost and time required for the initial execution of a large-scale language model, which can be achieved in subsequent executions. This enables fast, inexpensive, and stable execution of regression tests, especially for features that change infrequently, significantly improving the operational efficiency of test automation.

[0117] <Screen example> An example of a screen displayed on the display, which is the output unit 105 of the user terminal 10, will be described.

[0118] Figure 10 is a schematic diagram showing an example of the execution dashboard screen 1000. The execution dashboard screen 1000 is a screen for managing the execution status of tests based on uploaded test design documents and displaying the overall progress and quality status in an overview. For example, the execution dashboard screen 1000 has a design document list area 1001 on the left side of the screen and a test case list area 1002 on the right side.

[0119] The design document list area 1001 displays a list of previously uploaded test design documents (for example, "XX Test 3") and their respective upload dates. When an administrator selects a specific design document from this list, the test case list area 1002 displays a grid-formatted list of the test cases included in that design document.

[0120] Each row in the test case list area 1002 displays a test ID (e.g., "1", "2") and a test category (e.g., "Functional Test", "Login Screen", "Dashboard"). Additionally, a result icon 1003 (e.g., a green checkmark indicating success or failure, a red cross mark indicating failure) is displayed in real-time at the left end of each row, allowing administrators to quickly grasp the test results. A screen transition button 1004 is located at the right end of each row; pressing this button transitions to the detailed results screen 1100, described later. This allows users to view detailed results for a specific test case.

[0121] Figure 11 is a schematic diagram showing an example of the results details screen 1100. The results details screen 1100 is displayed when the screen transition button 1004 in Figure 10 is pressed, and is a screen for checking in detail the execution procedure, expected value, and execution result of the generated source code for individual test cases. The results details screen 1100 includes, for example, a metadata display area 1101, an extracted information display area 1102, and an execution result section 1103.

[0122] The metadata display area 1101 (top of the results details screen 1100) displays the metadata of the test case (test ID, category path, etc.). The extracted information display area 1102 (below it) displays information extracted from the test design document, such as "verification items" (e.g., "confirm that the dashboard screen is displayed correctly"), "expected results" (e.g., "the dashboard screen is displayed correctly and all menu items can be viewed"), "prerequisites," and "execution steps," organized into blocks.

[0123] Furthermore, the execution results section 1103 (lower half of the screen) displays the results obtained from executing the source code generated by the large-scale language model in a comparative format. Specifically, the reason presentation area 1104 of the execution results section 1103 presents the reasons for pass or fail based on the results of comparing the expected value with the operation log (for example, "The expression 'displayed correctly' in the expected value and verification items is abstract, and there is a lack of objective verification criteria," etc.) in natural language. The post-execution status presentation area 1105 contains a summary of the final screen transition destination or displayed content.

[0124] Furthermore, in the execution history area 1106 of the execution results section 1103, including "Execution History of Prerequisites" and "Test Execution History," individual operation steps executed by the source code (for example, "Enter an account in the email address input field") are displayed in chronological order. Each step is accompanied by an icon indicating the success or failure of the execution, and text indicating the results of the analysis of the screen state at that time. Locations where errors occurred and locations where the confidence level of the large-scale language model was low are highlighted, allowing users to quickly identify the cause of the problem. In addition, a link to an "evidence video" recording the browser's behavior during test execution is provided at the bottom of the screen to support visual verification. This allows users to compare the contents of the test design document with the actual execution results and analyze in detail whether the test was performed correctly and why a particular judgment result was obtained.

[0125] 〔supplement〕 In the embodiments described above, the cases in which the units and means are implemented by a processor were explained, but the invention is not limited to this. The units and means may be any hardware known to perform the operation.

[0126] Furthermore, although the above-described embodiment explained an example in which the server 20 provides each function, the configuration is not limited to this. Some or all of the functions provided by the server 20 in this embodiment may be provided by the user terminal 10 instead of the server 20.

[0127] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. For example, a configuration or process in one embodiment may be combined with a configuration or process in another embodiment, or a modification of one embodiment may be applied to another embodiment. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0128] [Note] The details described in each of the above embodiments are noted below.

[0129] <Note 1> A program for operating a computer equipped with a processor, the program causing the processor to execute: definition information including the names and argument information of a plurality of operation function groups, which consist of a plurality of functions for performing operations on a target under test; a template that defines the structure of source code executable using the plurality of operation function groups; and test design information including test conditions for the target under test written in natural language; a step of generating a code generation prompt including instruction information that instructs a large-scale language model to generate source code by embedding the obtained definition information as constraints to limit the functions that can be used for code writing, the obtained template as the framework for code writing, and the obtained test design information as the target of code writing into prompts, thereby causing a large-scale language model to write test logic using the functions included in the definition information within a predetermined area of ​​the template; and a step of inputting the generated code generation prompt into the large-scale language model, thereby causing the large-scale language model to generate source code including variable definition code to be passed as arguments to the plurality of operation function groups within the predetermined area.

[0130] <Note 2> The program as described in Appendix 1, wherein in the acquisition step, parsing examples for converting the natural language description into program variables are acquired; in the generation step, the acquired parsing examples are further embedded in the prompt; and in the generation step, the source code is generated by causing the large-scale language model to interpret the test conditions included in the code generation prompt based on the embedded parsing examples.

[0131] <Note 3> The program according to Appendix 1 or Appendix 2, which causes the processor to further perform a step of executing the source code generated by the generation step, wherein in the execution step, visual features including at least one of text information, graphic information, or positional information indicating the relative position of an element on the screen by analyzing a screenshot of the screen under test displayed during the execution of the source code, and identifies the target of operation included in the test design information based on the visual features.

[0132] <Note 4> The program as described in Appendix 3, wherein, in the step described above, if the target of the operation cannot be identified, the program performs a process to search for an area on the screen that includes a specific area presumed to correspond to the target of the operation, or a screen state change process to improve at least one of the visibility and operability of the target of the operation.

[0133] <Note 5> The program according to any one of Appendix 1 to Appendix 4, further comprising: causing the processor to execute the source code generated by the generation step and output an operation log including the result of executing the source code; and determining whether the test passed or failed by comparing the output operation log with an expected value indicating the pass condition for the test by executing the source code, which is included in the test design information, and the output step and the determination step are each executed using at least one of an independent prompt and an independent AI agent.

[0134] <Note 6> The program described in Appendix 5, wherein in the determination step, along with the pass / fail determination result, a confidence level indicating the degree of certainty of the determination result is output, and if the confidence level is lower than a predetermined threshold, the program is made to determine the pass / fail status of the test for each of several different large-scale language models, taking the operation log and the expected value as input, and the final pass / fail status of the test is determined based on the determination results from each of the several different large-scale language models.

[0135] <Note 7> The processor performs an initial test execution in a learning mode, separate from the execution of the test, determining the next operation to be performed on the test target according to the state of the test target obtained during the test execution, and records the operation log when the test is determined to be successful in the determination step, and converts the recorded operation log into a test code in which a series of operation procedures are predetermined based on predetermined conversion rules. The program described in Appendix 5 further executes the above.

[0136] <Note 8> The program according to any one of the appendices 1 to 7, wherein the processor is further instructed to perform the step of obtaining and structuring information indicating a solution to the failure when a failure occurs during the execution of the source code, and storing it as knowledge data, and in the generation step, after the knowledge data has been stored, the stored knowledge data is embedded in the prompt in addition to the definition information, the template, and the test design information to generate the code generation prompt.

[0137] <Note 9> An information processing device comprising a processor, wherein the processor executes all steps in any of the programs described in Appendix 1 to Appendix 8.

[0138] <Note 10> A method to be performed on a computer equipped with a processor, wherein the processor performs all steps in any of the programs described in Appendix 1 to Appendix 8.

[0139] <Note 11> A system comprising one or more of the aforementioned processors that execute all the steps in the program described in any of the appendices 1 to 8. [Explanation of symbols]

[0140] 1... System 10…User terminal 20... Server 30…Generating AI system 80…Network 101, 201… Control Units 102, 202...Storage section 103, 203... Communications Department 104...Input section 105...Output section 131...Operation reception unit 132... Transmitter / Receiver 133…Display Control Unit 204… Input / Output Interface 211... Reception Department 212…Information acquisition department 213... Prompt generation unit 214...Source code generation control unit 215...Test execution unit 216... Pass / Fail Judgment Department 221... Test Design Document Table 222... Function Definition Master Table 223...Execution result log table 224... Code template group 1000...Execution Dashboard Screen 1001...Design Document List Area 1002…Test Case List Area 1003... Result icon 1004... Screen transition button 1100…Result details screen 1101... Metadata display area 1102...Extracted information display area 1103...Execution Results Section 1104…Reason presentation area 1105... Post-execution state display area 1106... Execution history area

Claims

1. A program that operates a computer equipped with a processor, The aforementioned processor, A step of obtaining definition information including the names and argument information of multiple operation functions in a group of operation functions that perform operations on the target under test, a template that defines the structure of source code that can be executed using the group of operation functions, and test design information including test conditions for the target under test written in natural language. The steps include: generating a code generation prompt that includes instruction information to generate the source code by embedding the acquired definition information as constraints to limit the functions that can be used for code writing, the acquired template as the framework for code writing, and the acquired test design information as the target of code writing, instructing the large-scale language model to write test logic using the functions included in the definition information within a predetermined area of ​​the template; and using the acquired definition information as constraints to limit the functions that can be used for code writing, the acquired template as the framework for code writing, and the acquired test design information as the target of code writing, into the prompt, thereby instructing the large-scale language model to write test logic using the functions included in the definition information within a predetermined area of ​​the template. The steps include: inputting the generated code generation prompt into the large-scale language model, thereby causing the large-scale language model to generate source code including variable definition code to be passed as arguments to the group of operation functions within the predetermined domain; A program that executes something.

2. In the aforementioned acquisition step, further examples of parsing to convert the natural language description into program variables are acquired. In the generation step, the acquired analysis example is further embedded in the prompt. In the generation step, the source code is generated by causing the large-scale language model to interpret the test conditions included in the code generation prompt based on the embedded analysis example. The program according to claim 1.

3. The processor is further instructed to perform the step of executing the source code generated by the generation step, The program according to claim 1, wherein in the step of execution, the program extracts visual features including at least one of text information, graphic information, or positional information indicating the relative position of an element on the screen by analyzing a screenshot of the screen under test displayed during the execution of the source code, and identifies the target of operation included in the test design information based on the visual features.

4. The program according to claim 3, wherein in the step of execution, if the target of operation cannot be identified, the program performs a process of searching for a region on the screen that includes a specific region that is presumed to correspond to the target of operation, or a screen state change process to improve at least one of the visibility and operability of the target of operation.

5. The aforementioned processor, The steps include executing the source code generated by the above generation step and outputting an operation log including the execution result of the source code, The steps include determining whether the test passed or failed by comparing the output operation log with the expected value indicating the pass condition for the test by executing the source code, which is included in the test design information, Let's execute this further, The program according to claim 1, wherein the output step and the determination step are performed using at least one of an independent prompt and an independent AI agent.

6. The program according to claim 5, wherein in the step of making a determination, the program outputs a confidence level indicating the degree of certainty of the determination result along with the pass / fail determination result, and if the confidence level is lower than a predetermined threshold, it causes each of the multiple different large-scale language models to determine whether the test has passed or failed using the operation log and the expected value as input, and determines the final pass or fail of the test based on the determination results from the multiple different large-scale language models.

7. The aforementioned processor, In addition to the execution of the aforementioned test, the initial test is performed in a learning mode, determining the next operation to be performed on the test subject according to the state of the test subject obtained during the test execution, and recording the operation log if the test is determined to be passed in the determination step, The recorded operation log is converted into a predetermined test code based on a set of conversion rules, and the series of operation procedures are converted into a predetermined test code. The program according to claim 5, which further executes the following.

8. The processor is then instructed to perform a further step of acquiring and structuring information indicating a solution to a failure that occurs during the execution of the source code, and storing it as knowledge data. The program according to claim 1, wherein in the generation step, after the knowledge data has been accumulated, the accumulated knowledge data is embedded in the prompt in addition to the definition information, the template, and the test design information to generate the code generation prompt.

9. An information processing device comprising a processor, wherein the processor executes all steps in any one of claims 1 to 8.

10. A method to be performed on a computer having a processor, wherein the processor performs all steps of a program according to any one of claims 1 to 8.

11. A system comprising one or more processors that perform all steps in the program according to any one of claims 1 to 8.

Images