Program, method, information processing device and system

Abstract

To provide a program for reducing a burden of creating reports relating to results of software test.SOLUTION: There is provided a program to be executed by a computer including a processor and a memory. The program causes the processor to execute the steps of: acquiring test results; aggregating information contained in the test results on the basis of a perspective being summarized in a report; inputting the aggregated results into a trained model that has been so trained, when the aggregation results based on the perspective is input, as to output the information contained in the aggregated results in a format conforming to specifications of the report, thereby causing the model to output the information contained in the aggregation results in the format conforming to the specifications of the report; and presenting the information output from the trained model to a user.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a program, a method, an information processing device, and a system. [Background technology]

[0002] To ensure the quality of software, its functions are tested before it is released. Testing may be performed by the software development company itself or by an external company separate from the software development company. When testing is outsourced to an external company, the company that contracted the testing must report the results of the tests to the software development company that commissioned the testing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-197393 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating a report that reports the results of the tests that have been performed is a laborious process, so a technology that can reduce the burden of creating the report is desired.

[0005] In Patent Document 1, the system learns in advance the correspondence between the defect contents of software and their defect classifications, and upon receiving a request for a natural language string requesting a document to be used to improve the defect classification, it estimates the defect classification specified by the natural language string through linguistic analysis and responds with the most accurate estimation result. In Patent Document 1, the client terminal tallys up the defect classification results and displays the tally results to the user. However, Patent Document 1 does not mention supporting the creation of reports on the results of software testing.

[0006] An object of the present disclosure is to reduce the burden of producing reports on the results of software testing. [Means for solving the problem]

[0007] A program to be executed by a computer having a processor and a memory, the program instructs the processor to execute the following steps: acquiring test results, aggregating information included in the test results based on viewpoints to be summarized in a report, inputting the aggregated results based on the viewpoints into a trained model that has been trained to output information included in the aggregated results in a manner conforming to the report specifications when the aggregated results are input, thereby outputting the information included in the aggregated results in a manner conforming to the report specifications, and presenting the information output from the trained model to a user. [Effects of the Invention]

[0008] According to the present disclosure, the burden of creating reports on the results of software testing can be reduced. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a system 1. FIG. [Figure 2] 2 is a block diagram illustrating an example of the configuration of a terminal device 10 shown in FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of a server 20. [Figure 4] FIG. 2 is a diagram showing the data structure of a user information table 2021. [Figure 5] FIG. 2 is a diagram showing the data structure of a test information table 2022. [Figure 6] FIG. 20 is a diagram showing the data structure of a test case information table 20221 for information about test cases included in a test suite. [Figure 7] FIG. 10 is a diagram showing the data structure of a prompt table 2023. [Figure 8]10 is a flowchart showing an example of the operation of the server 20 when assisting in the creation of a final report. [Figure 9] FIG. 10 is a schematic diagram illustrating an example of a screen for selecting data to be used in creating a final report. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of a screen that prompts a user to confirm data to be used in creating a final report. [Figure 11] FIG. 10 is a schematic diagram illustrating an example of a screen for inputting a creation purpose. [Figure 12] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 13] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 15] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 16] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 17] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 18] FIG. 10 is a schematic diagram illustrating an example of materials included in a final report. [Figure 19] FIG. 2 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same components are designated by the same reference numerals.

[0011] <Summary> The system according to this embodiment aggregates the results of debugging software by a predetermined classification and inputs the aggregated results into a trained model to generate a debugging completion report. The trained model may be, for example, a large language model (LLM).

[0012] <1 Overall system configuration> Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

[0013] 1 shows an example in which the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two. The number of terminal devices 10 included in the system 1 may be one, or three or more.

[0014] 1 shows an example in which the system 1 includes one LLM system 30, but the number of LLM systems 30 included in the system 1 is not limited to one. The number of LLM systems 30 included in the system 1 may be two or more.

[0015] In this embodiment, a collection of multiple devices may be considered as one server. The allocation of multiple functions required to realize the server 20 according to this embodiment to one or more pieces of hardware may be determined appropriately in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20. Specifically, for example, a server having a function for creating test cases based on specifications and a server on which a trained model is built may be different devices.

[0016] The terminal device 10 shown in Fig. 1 is an information processing device used by a user who uses a service provided by the server 20. For example, the terminal device 10 is an information processing device operated by a user who uses the completion report creation support provided by the server 20 to create a completion report of a software debugging process. The terminal device 10 is realized, for example, by a desktop personal computer (PC), a laptop PC, a head-mounted display, or the like. The terminal device 10 may also be a portable computer such as a smartphone or a tablet terminal.

[0017] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving input operations from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.).

[0018] The server 20 is an information processing device that provides a service to support the creation of a final report. For example, the server 20 uses a trained model to support the creation of a final report of a debugging process.

[0019] The server 20 is, for example, an information processing device realized by a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an input device for receiving input operations from a user and as an interface with an output device for outputting information to the user.

[0020] The LLM system 30 is a system built with a large-scale artificial intelligence model (LLM) used in the field of natural language processing (NLP), for example. By learning from large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns in human language and effectively perform natural language generation (NLG) tasks.

[0021] LLMs are used in many NLP tasks, such as generating answers to specific questions, automatic sentence generation, text summarization, translation, and sentiment analysis. They can also be used in a variety of applications, including education, entertainment, customer service, and product development. Examples of LLMs include: GPT-4 (registered trademark) (OpenAI) PaLM2 (registered trademark) (Google Inc.) StableLM (StableAI) Llama 2 (Meta)

[0022] The LLM system 30 inputs test-related information and prompts output from the server 20 into the LLM, and causes the LLM to output answers based on the input text data and prompts. The LLM system 30 transmits the answers output from the LLM to the server 20.

[0023] Each information processing device is configured by a computer equipped with an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer will be omitted.

[0024] <1.1 Terminal device configuration> Fig. 2 is a block diagram showing an example configuration of the terminal device 10 shown in Fig. 1. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected by, for example, a bus or the like.

[0025] The communication unit 120 performs processing such as modulation and demodulation for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0026] The input device 13 is a device for inputting instructions or information by a user operating the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like, which inputs instructions by touching an operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, keyboard, mouse, or the like. The input device 13 converts instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may include, for example, a receiving port that receives electrical signals input from an external input device.

[0027] The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0028] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0029] The camera 160 is a device that receives light with a light receiving element and outputs the light as an image capturing signal.

[0030] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0031] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0032] The user information 181 includes, for example, information about the user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, contact information, and the like.

[0033] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating in accordance with the program.

[0034] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131 or the like.

[0035] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal that is input from the microphone 171 and converted into a digital signal by the voice processing unit 17. For example, the operation reception unit 191 analyzes the received voice signal and extracts a predetermined noun, thereby acquiring an instruction from the user.

[0036] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits information input by a user or instructions from a user to the server 20. In addition, the transmitting / receiving unit 192 receives information provided by the server 20.

[0037] The presentation control unit 193 controls the output device 14 to present information provided from the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the information transmitted from the server 20 to be displayed on the display 141. In addition, the presentation control unit 193 causes the information transmitted from the server 20 to be output from the speaker 172.

[0038] <1.2 Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0039] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0040] The storage unit 202 is realized by, for example, the memory 25 and the storage 26, and stores data and programs used by the server 20. The storage unit 202 has, for example, a user information table 2021, a test information table 2022, and a prompt table 2023. Note that the information stored in the storage unit 202 is not limited to these.

[0041] The user information table 2021 is a table that stores information about users, and will be described in detail later.

[0042] The test information table 2022 is a table that stores information about test results. Details will be described later. In this embodiment, the information about test results includes, for example, the results of tests performed on multiple test suites. In this embodiment, a test suite includes, for example, information about a set of test cases performed by a tester. The information about a set of test cases performed by a tester includes, for example, the results of test execution of multiple test cases in a specific process. The process may be, for example, a specific process in a specific service, or a specific process in a specific project of a specific service. The service, project, and process may be referred to as, for example, attributes. In addition to these, the attributes may also include business type / industry, system type, etc.

[0043] Test cases are defined, for example, by a test design document. Information about test cases is, for example, information in which test cases are converted into a format that allows test implementation management. Information about test cases includes, for example, "test target items / target functions." "Test target items / target functions" represent, for example, the items / functions that are the target of testing. "Test target items / target functions" may also be referred to as test sections. Test sections may include sub-concepts such as major functions (section 1), medium functions (section 2), and minor functions (section 3). Information about test cases may include, for example, test perspectives. Test perspectives represent, for example, the viewpoints (philosophies) that form the basis of testing. Test perspectives are associated with test sections. Check items may be associated with test perspectives. Check items represent information for identifying the tests to be performed.

[0044] The information about the test cases may include, for example, a schedule until the product is launched. The schedule until the product is launched may include, for example, a product development schedule, and may be expressed in predetermined units of time. The information about the test cases may include, for example, information about the progress of the tests that have been conducted. The information about the progress of the tests includes, for example, information that represents the status of the test cases. The information that represents the status of the test cases includes, for example, "OK" indicating that the test has passed, "FIXED" indicating that the defect has been corrected, "NG" indicating that a defect has occurred, "PENDING" indicating that the test is pending, and "OUT OF COVERAGE" indicating that the test is not included in the test.

[0045] The information about the test case may include, for example, information about the filing of a report. The filing of a report is, for example, a communication to a developer that is issued based on an executed test. The information about the filing of a report may include the purpose of the report, such as a "defect," "inquiry," or "request." The information about the test case may include, for example, the reason for closure. The reason for closure may be, for example, that the issue cannot be reproduced, that the issue has been resolved after reproduction, and that the issue has been released.

[0046] The information about the test case may include, for example, information about the test environment, which indicates the environment in which the test is executed, and includes, for example, the operating system (OS) used on the terminal on which the test is executed, the browser used on the terminal on which the test is executed, the model of the terminal on which the test is executed, etc.

[0047] The information about the test case may include, for example, information about the process to be detected and information about the embedding process. The information about the process to be detected and information about the embedding process are, for example, information about items that are included in the format of the final report but not included in the data.

[0048] The prompt table 2023 is a table that stores information about prompts (instructions), and will be described in detail later.

[0049] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that runs on the web browser application stored in the terminal device 10. The control unit 203 operates in accordance with the program to perform functions shown as a reception control module 2031, a transmission control module 2032, a counting module 2033, a support module 2034, and a presentation control module 2035.

[0050] The reception control module 2031 controls the process by which the server 20 receives signals from external devices in accordance with a communication protocol.

[0051] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0052] The tallying module 2033 tally information about test results specified by the user from a predetermined perspective. Specifically, for example, the tallying module 2033 tally information about multiple test cases included in the information about test results from a predetermined perspective. The predetermined perspectives include, for example, a schedule perspective, a test progress perspective, a form classification perspective, a function perspective, a test perspective perspective, a test environment perspective, a perspective of processes to be detected, and a perspective of embedded processes. The predetermined perspectives are based on, for example, the items for which test results are reported in the final report.

[0053] The final report is, for example, a document summarizing the report contents regarding the test execution results. In the final report, for example, the report contents are summarized by chapter. Each chapter includes multiple items corresponding to the chapter. Each chapter includes materials summarized by item. The materials for each item are created using, for example, one or multiple slides. Materials for multiple items may be created using one slide.

[0054] The final report may include, for example, the following chapters: Test results Fault analysis · Report a problem ·Improvement suggestions

[0055] "Test Results" is a chapter that reports the results of tests. Test results include, for example, items that report the execution status of tests according to the schedule, and items that report the progress of tests according to the schedule. Test execution status, for example, indicates the trend in the number of tests executed. Test progress, for example, indicates the trend in the status changes of test cases. Test results include, for example, items that report the execution status of tests over the number of days that have passed, and the progress of tests. Test results include, for example, items that report the defect rate that occurred in the executed tests. Test results include, for example, items that report the classification of defects that occurred in the executed tests. Defects can be classified based on, for example, a predetermined perspective. For example, defects can be classified based on filing, status, or reason for closure.

[0056] "Failure Analysis" is a chapter that reports the results of an analysis of failures (defects) that occurred during testing. Failure analysis includes, for example, an item that reports failure trends by function based on the number and rate of failures for each function (test category). Failure analysis includes, for example, an item that reports failure trends by test perspective based on the number and rate of failures for each test perspective. Failure analysis includes, for example, an item that reports failure trends by test environment based on the number and rate of failures for each test environment. Failure analysis includes, for example, an item that reports failure trends by detection process based on the number and rate of failures for each process to be detected. Failure analysis includes, for example, an item that reports failure trends by embedding process based on the number and rate of failures for each embedding process. Failure analysis includes, for example, an analysis result that summarizes failure occurrence trends, issues, and anticipated risks for each item.

[0057] "Problem Report" is a chapter that reports details of problems that occurred during testing. The problem report includes information such as the time when the test in which the problem occurred was executed, the environment, the tester, and the developer of the test target.

[0058] "Improvement Proposals" is a chapter that proposes improvements to the testing system or development system based on the test results. The improvement proposals include, for example, items that report issues with the testing system or development system that are identified based on the test results. The improvement proposals include, for example, proposals for the state that the testing system or development system should aim to achieve in order to improve the issues identified based on the test results. The improvement proposals include, for example, proposals for measures to achieve the state that is aimed for.

[0059] The assistance module 2034 assists the user in creating a final report. Specifically, for example, the assistance module 2034 creates report content based on a predetermined perspective by inputting, for example, information obtained by summarizing test results into the trained model. More specifically, the assistance module 2034 inputs, for example, information summarized based on a predetermined perspective into the trained model and causes the trained model to output report content for items corresponding to the perspective. The assistance module 2034 inputs information summarized for each perspective into the trained model and causes the trained model to output report content for multiple items. In other words, the assistance module 2034 inputs the aggregation results into the trained model and outputs the information included in the aggregation results in a format conforming to the specifications of the final report. The trained model compiles multiple report contents and outputs them as a first draft of the final report.

[0060] Specifically, for example, the support module 2034 uses the LLM system 30 when creating a final report. For example, the support module 2034 transmits to the LLM system 30 information that compiles test results selected by the user and instructions for outputting report content for items corresponding to the compiled information. The LLM system 30 inputs the compilation results and instructions transmitted from the server 20 into the LLM, and causes the LLM to output a final report.

[0061] The support module 2034 issues an instruction statement for outputting a final report, for example, in response to a user pressing a predetermined button. The instruction statement may include, for example, content for explaining the configuration of the final report. The instruction statement may include, for example, content for specifying an output format for the report content. The instruction statement may include, for example, content for specifying aggregated data to be used when creating the report content. The instruction statement may include, for example, content for outputting a report on an aspect of the aggregated data that should be noted. The support module 2034 may convert the aggregated data into data in a predetermined format and transmit it to the LLM system 30. The predetermined format may include, for example, Markdown format.

[0062] The support module 2034 may receive a creation purpose of the final report from the user and reflect the received creation purpose in the final report. For example, the support module 2034 receives input of a creation purpose of the final report from the user. The support module 2034 reflects the input creation purpose in an instruction statement. The support module 2034 transmits the instruction statement reflecting the creation purpose to the LLM system 30. The support module 2034, for example, stores instruction statements in which wording corresponding to the creation purpose can be rewritten. The support module 2034, for example, reflects the input creation purpose in a corresponding area of ​​the instruction statement and transmits the reflected instruction statement to the LLM system 30. The support module 2034 may, for example, have an instruction statement for each creation purpose. At this time, the support module 2034 reads out an instruction statement corresponding to the input creation purpose and transmits the read instruction statement to the LLM system 30.

[0063] The presentation control module 2035 controls the process of presenting information to the user. For example, the presentation control module 2035 presents information output from the trained model to the user. The presentation control module 2035 may present the information output from the trained model to the user in the format in which it was output (e.g., Markdown format), or may convert the information into a predetermined format (e.g., PPT file format) and present it to the user. Specifically, for example, the presentation control module 2035 presents a final report output from the LLM system 30 to the user.

[0064] <2 Data Structure> 4 to 7 are diagrams showing the data structures of tables stored in the server 20. Note that Figures 4 to 7 are merely examples and do not exclude data that is not listed. Furthermore, even data that is listed in the same table may be stored in separate storage areas in the storage unit 202.

[0065] Fig. 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with the user ID as a key.

[0066] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The gender is an item that stores the gender of the user. The date of birth is an item that stores the date of birth of the user. The contact information is an item that stores the contact information (e.g., telephone number, email address, etc.) of the terminal device 10 that the user has.

[0067] Fig. 5 is a diagram showing the data structure of the test information table 2022. The test information table 2022 shown in Fig. 5 is a table having columns such as service, project, process, test case, test executor, and execution date and time, with the test result ID as a key. Note that the information stored in Fig. 5 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0068] The test result ID is an item that stores an identifier for uniquely identifying a test suite. The service is an item that stores, for example, information for identifying a service to which a test suite is related. The "service" item may store a service ID or a service name. The project is an item that stores, for example, information for identifying a project to which a test suite is related. The "project" item may store a project ID or a project name.

[0069] The step is an item that stores information for identifying the step that the test suite is involved in, for example. The "step" item may store a step ID or a step name. The test case is an item that stores the number of test cases included in the test suite. The test executor is an item that stores the person who executed the test. The "test executor" item may store a user ID or a user name. The execution date and time is an item that stores the date and time when the test was executed.

[0070] Figure 6 is a diagram showing the data structure of a test case information table 20221 for information about test cases included in a test suite. The test case information table 20221 shown in Figure 6 is a table with a test case ID as a key and columns such as test result ID, major function, medium function, minor function, test perspective, execution schedule, status, initiation, environment, test executor, and execution date and time. Note that the information stored in Figure 6 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0071] The test case ID is an item that stores an identifier for uniquely identifying a test case. The major function is an item that stores the first classification as a test target item / target function. The medium function is an item that stores the second classification as a test target item / target function. The minor function is an item that stores the third classification as a test target item / target function. The test perspective is an item that stores the test perspective set for the test case. The execution schedule is an item that stores the schedule when the test case was executed. The status is an item that stores the current status of the test case. The "status" item may store the date and time when the status was set along with the status. The issue is an item that stores the purpose for which the test case was issued. The issue may also be called a ticket. The environment is an item that stores the environment in which the test case was executed. The "environment" item stores, for example, the model of the terminal on which the test case was executed, the OS used on the terminal, the browser, etc.

[0072] FIG. 7 is a diagram showing the data structure of the prompt table 2023. The prompt table 2023 shown in FIG. 7 is a table having columns such as prompt data, with a prompt ID as a key. Note that the information stored in FIG. 7 is not limited to this. For example, further information may be stored. For example, a column for storing the creation purpose may be included. This allows the assistance module 2034 to select a command statement for each creation purpose, for example.

[0073] The prompt ID is an item that stores an identifier for uniquely identifying a prompt (instruction). The prompt data is an item that stores data related to a prompt that serves as an instruction to be input to the LLM. The data related to the prompt is, for example, text information related to the prompt.

[0074] The text information related to the prompt is, for example, an instruction statement for outputting a final report based on the aggregated data, and includes, for example, the following content: User's responsibility to create final report - The structure of chapters and items in the final report (i.e., explanation of the specifications of the final report) The purpose of the explanation to be given in the materials for each item (for example, arguments from aggregated data may be included) -Format of explanation to be written in the materials for each item - The format of graphs to be used in the materials for each item (no instructions are given for items that do not require graphs) -Table format to be used in each item (no instructions are given for items that do not require tables) - Purpose of explanation taking into account the purpose of creation

[0075] <3 operations> The operation of the server 20 when assisting in the creation of the final report will now be described. FIG. 8 is a flowchart showing an example of the operation of the server 20 when assisting in the creation of a final report.

[0076] First, a user (test engineer) who wants to create a final report accesses the server 20. The user logs in to a service that assists in creating a final report, provided by the server 20, for example, by entering their own user ID and password.

[0077] The user operates the terminal device 10 to select test results for which a final report is to be created. Specifically, for example, the user selects a predetermined attribute to select test results that belong to the attribute. More specifically, for example, the user selects a service to which a test suite is related to select a test suite to which the selected service is associated. Also, for example, the user selects a project to which a test suite is related to select a test suite to which the selected project is associated. Also, for example, the user selects a process to which a test suite is related to select a test suite to which the selected process is associated.

[0078] 9 is a schematic diagram showing an example of a screen for selecting data to be used in creating a final report. The selection screen shown in FIG.

[0079] The first area 1411 is an area for selecting test results to be used when creating a final report. In the example shown in Fig. 9, the presentation control unit 193 displays services, projects, and processes in a selectable manner. This allows the user to select a test suite by selecting a service, project, or process. The operation object 1412 is an object for proceeding with the process of creating the final report.

[0080] In step S11, the server 20 receives from the user a specification of information related to the test results. Specifically, for example, the terminal device 10 receives from the user a selection of test results to be used when creating a final report, and transmits information that can identify the selected test results to the server 20. The reception control module 2031 receives the information transmitted from the terminal device 10.

[0081] In step S12, the server 20 aggregates the information related to the selected test results from a predetermined perspective. Specifically, for example, the aggregation module 2033 obtains information related to multiple test cases included in the information related to the selected test results from the test information table 2022, and aggregates the information from, for example, a schedule perspective, a test progress perspective, a form classification perspective, a function perspective, a test perspective perspective, a test environment perspective, a process to be detected perspective, an embedding process perspective, etc. The aggregation module 2033 associates the aggregation results obtained by aggregation with items in the final report.

[0082] In step S13, the server 20 presents the data used to create the final report to the user. Specifically, for example, the presentation control module 2035 presents the user with information obtained by aggregating information related to the test results. Also, for example, the presentation control module 2035 presents the user with items in the final report and perspectives related to the aggregation, which are associated with each other. Also, the presentation control module 2035 presents the user with an overview of the aggregation results. For example, the transmission control module 2032 transmits information for displaying the information obtained by the aggregation to the terminal device 10. The presentation control unit 193 displays the information transmitted from the server 20 on the display 141.

[0083] 10 is a schematic diagram showing an example of a screen for prompting the user to confirm the data to be used in creating the final report. The confirmation screen shown in FIG. 10 includes a second area 1413 and a third area 1414.

[0084] The second area 1413 is an area for displaying information obtained by aggregating information related to test results. The second area 1413 indicates which perspective of the aggregation results has been obtained by the aggregation process. For example, if no information is stored in the item related to aggregation, no aggregation results are obtained. In the example shown in FIG. 10, aggregation results aggregated from the perspective of schedule, from the perspective of form classification, from the perspective of function, from the perspective of test perspective, and from the perspective of test environment have been obtained. Furthermore, in the example shown in FIG. 10, aggregation results aggregated from the perspective of test progress, from the perspective of processes to be detected, and from the perspective of embedding processes have not been obtained.

[0085] The third area 1414 is an area for displaying the items in the final report in association with a summary of the tabulation results from the perspective of tabulation. The third area 1414 displays the item names in the final report and a summary of the tabulation results referenced in the items. The referenced tabulation results are marked with an instruction object 14131 in the second area 1413. The user can check whether there are any problems with the data used to create materials for specific items in the final report by referring to the second area 1413 and the third area 1414. If there is a problem with the data used to create the materials, the user can delete or edit the data, for example, by pressing an edit icon displayed in the third area 1414.

[0086] 10 shows the case where information for creating the document "4.3 Failure Trends (by Test Viewpoint)" is displayed, but a confirmation screen may exist for each item in the final report. After the user has finished checking the data to be used in creating the final report, the user inputs an instruction to the terminal device 10 to proceed to the next process.

[0087] In step S14, the server 20 accepts input of a creation purpose. Specifically, for example, the presentation control module 2035 causes the terminal device 10 to display an input screen for the creation purpose. For example, the presentation control module 2035 causes the terminal device 10 to display a plurality of creation purposes so that the user can select one. The user selects a creation purpose displayed on the display 141.

[0088] FIG. 11 is a schematic diagram illustrating an example of a screen used when inputting a creation purpose. The input screen illustrated in FIG. 11 includes a fourth area 1415. The fourth area 1415 is an area for displaying multiple creation purposes in a selectable manner. In the example illustrated in FIG. 11, the presentation control unit 193 displays "regular report," "additional proposal," and "explanatory report" as creation purposes in a selectable manner. The regular report is, for example, a periodic report of test results. The additional proposal is, for example, a report that identifies improvements based on test results and proposes specific content. The explanatory report is, for example, a report that provides a detailed explanation of the test results. The user selects one of the displayed creation purposes. The terminal device 10 transmits information for specifying the creation purpose to the server 20. The displayed creation purposes are not limited to "regular report," "additional proposal," and "explanatory report," but may be a combination of at least two of "regular report," "additional proposal," and "explanatory report." Furthermore, the creation purposes are not limited to "regular report," "additional proposal," and "explanatory report."

[0089] In step S15, the server 20 creates a final report. Specifically, for example, the support module 2034 reads from the prompt table 2023 an instruction statement for outputting a final report based on the aggregated data. If an instruction statement is prepared for each creation purpose, the support module 2034 reads from the prompt table 2023 an instruction statement corresponding to the creation purpose. If an instruction statement with a blank creation purpose is prepared, the support module 2034 reads from the prompt table 2023 an instruction statement with a blank creation purpose and inserts information based on the creation purpose into the blank to create an instruction statement. The support module 2034 outputs the aggregated data and the instruction statement to the LLM system 30.

[0090] The LLM system 30 inputs the aggregated data and instructions sent from the server 20 into the LLM, and causes the LLM to output a final report. The LLM system 30 transmits the information output from the LLM to the server 20.

[0091] The presentation control module 2035 displays the final report transmitted from the LLM system 30 on the terminal device 10. That is, for example, the transmission control module 2032 transmits information for displaying the final report transmitted from the LLM system 30 to the terminal device 10. The terminal device 10 displays the information transmitted from the server 20 to the user. The presentation control module 2035 may present the final report transmitted from the LLM system 30 to the user in a format output from the LLM system 30, for example, Markdown format. The presentation control module 2035 may also convert the final report transmitted from the LLM system 30 into a predetermined file format for presentations (for example, PPT file format) and present the converted information to the user.

[0092] 12 to 18 are schematic diagrams showing examples of materials included in a final report. Fig. 12 shows an example of a material reporting the progress of a test, for example, in the "Test Results" chapter. When creating the material shown in Fig. 12, the instructions may include, for example, the following: -Create a title for the test progress based on the aggregated data on test progress / failure rate. - Create a table in the following format based on the aggregated data related to test progress / defect rate. Based on the aggregated data on test progress / defect rate, please create a pie chart in the format of... Based on the aggregated data on test progress / defect rate, please create a line graph in the format of... -Please explain the important information in speech bubbles. 13 shows an example of a document for reporting a defect classification based on a form in the chapter "Fault Analysis." When creating the document shown in FIG. 13, the instruction text may include, for example, the following content: - Create a title for the defect based on the aggregated data related to the reporting classification. - Based on the aggregated data related to the classification of documents, please create a table in the following format. Based on the aggregated data related to the classification of documents, please create a pie chart in the format of...

[0093] 14 shows an example of a document that reports on failure trends by function in the chapter "Failure Analysis." When creating the document shown in FIG. 14, the instruction text may include, for example, the following content: - Create a title based on the aggregated data by function, taking into account the number of failures and failure rate. - Based on the aggregated data by function, please comment on the number of failures and failure rate by function. - Create a table in the following format based on the aggregated data by function. Based on the aggregated data pertaining to each function, please create a line graph in the format of...

[0094] 15 shows an example of a document that reports failure trends by test perspective in the chapter "Failure Analysis." When creating the document shown in FIG. 15, the instruction text may include, for example, the following content: - Create a title based on the aggregated data pertaining to each test aspect, taking into account the number of failures and failure rate. - Based on the aggregated data pertaining to each test aspect, please provide comments on the number of defects and defect rates per function. Create a table in the following format based on the aggregated data for each test perspective. - Create a line graph in the following format based on the aggregated data for each test aspect.

[0095] 16 shows an example of a document that reports on failure trends by test environment in the chapter "Failure Analysis." When creating the document shown in FIG. 16, the instructions may include, for example, the following content: Based on the aggregated data pertaining to each test environment, please write a title for the problem that occurred from the perspective of the test environment. - Please comment on the number of failures per environment based on the aggregate data pertaining to each test environment. Create a table in the following format based on the aggregated data for each test environment:

[0096] 17 shows an example of a document for reporting the analysis results (summary) in the chapter "Fault Analysis." When creating the document shown in FIG. 17, the instruction may include, for example, the following content: - Please summarize the trends by perspective based on the aggregated data related to failures (by function, test perspective, test environment, process to be detected, embedding process, etc.) - Based on the aggregated data related to the failure, please present the issues and anticipated risks from each perspective.

[0097] Fig. 18 shows an example of a document that reports issues and improvement proposals for the issues in the "Improvement Proposal" chapter. When creating the document shown in Fig. 18, the instructions may include, for example, the following content: -Create a task based on all the collected data -Create improvement proposals based on all collected data and the purpose of creation {...}

[0098] Although FIG. 18 shows a document containing the issues and improvement proposals, the issues and improvement proposals may be compiled in separate documents.

[0099] The user refers to the contents of the final report displayed on the terminal device 10 and edits the contents as necessary. For example, if there are no problems with the contents of the final report, the user downloads the final report in a predetermined format.

[0100] As described above, in the above embodiment, the reception control module 2031 acquires test results. The aggregation module 2033 aggregates information included in the test results based on the perspective of being summarized in a report. When the assistance module 2034 receives the classification-based aggregation results, it inputs the aggregation results into a trained model that has been trained to output the information included in the aggregation results in a format conforming to the report specifications, thereby outputting the information included in the aggregation results in a format conforming to the report specifications. The presentation control module 2035 presents the information output from the trained model to the user. This enables the server 20 to automatically create a report based on the input test results.

[0101] Therefore, the program, method, information processing device, and system according to this embodiment can reduce the burden of creating a report on the results of software testing.

[0102] In the above embodiment, the trained model is a large-scale language model. The assistance module 2034 inputs the aggregation results and instructions for outputting information included in the aggregation results in a format conforming to the report specifications into the trained model, thereby causing the trained model to output the information included in the aggregation results in a format conforming to the report specifications. This enables the server 20 to create a report using the large-scale language model.

[0103] In the above embodiment, the reception control module 2031 also receives the purpose of creating the report from the user. The trained model is trained to output information included in the aggregation results in a format that matches the report specifications and with content that is consistent with the purpose of creation when the aggregated results based on the viewpoint and the purpose of creation are input. The assistance module 2034 inputs the aggregated results and the purpose of creation into the trained model, causing the trained model to output information included in the aggregation results in a format that matches the report specifications and with content that is consistent with the purpose of creation. This enables the server 20 to create a report that matches the user's purpose. The content of the report changes depending on the purpose of creation. For example, the purpose of creation can be "periodic report," "additional proposal," "explanatory report," etc., and depending on the purpose of creation, for example, the analytical considerations in the report change. In this embodiment, the purpose of creation is also input into the trained model, making it possible to output a report that matches the purpose of creation.

[0104] In the above embodiment, the reception control module 2031 receives the purpose of creating the report from the user. The trained model is a large-scale language model. The assistance module 2034 inputs the aggregation results and instructions for outputting the information included in the aggregation results in a format that conforms to the specifications of the report and with content that conforms to the creation purpose into the trained model, thereby causing the trained model to output the information included in the aggregation results in a format that conforms to the specifications of the report and with content that conforms to the creation purpose. This enables the server 20 to use the large-scale language model to create a report that conforms to the creation purpose.

[0105] In the above embodiment, the support module 2034 inputs, into the trained model, the aggregated results of the perspectives to be referenced when analyzing failures occurring in the test results and instructions for outputting a report on the perspectives to focus on in the analysis based on the information included in the aggregated results, thereby causing the trained model to output a report on the perspectives to focus on in the analysis based on the information included in the aggregated results. As a result, the report content analyzing the aggregated results can be automatically included in the item of the report for analyzing failures from a predetermined perspective in the failure analysis.

[0106] Furthermore, in the above embodiment, the assistance module 2034 inputs, into the trained model, the aggregation results of the perspectives to be referenced when analyzing failures occurring in the test results, and instructions for outputting a table related to the information included in the aggregation results, a drawing related to the information included in the aggregation results, or a combination thereof, and causes the trained model to output, based on the information included in the aggregation results, a table related to the information included in the aggregation results, a drawing related to the information included in the aggregation results, or a combination thereof. As a result, a table showing the aggregation results, a drawing related to the aggregation results, or a combination thereof can be automatically included in the item of the report analyzing failures from a predetermined perspective in the failure analysis.

[0107] In the above embodiment, the support module 2034 inputs the aggregation results of multiple perspectives referenced when analyzing failures occurring in the test results and an instruction for outputting a summary of the failures into the trained model, thereby causing the trained model to output a summary of the failures based on the information included in the aggregation results. As a result, in the item in the report reporting the analysis results (summary) of the failure analysis, the failure trends and issues / risks are automatically included as a summary of the analysis from the aggregation results of multiple perspectives.

[0108] In the above embodiment, the support module 2034 inputs the aggregation results for all perspectives and instructions for outputting test problems, test improvement suggestions, or a combination thereof, recognized from the aggregation results, into the trained model, and causes the trained model to output test problems, test improvement suggestions, or a combination thereof, based on the information included in the aggregation results. This makes it possible to automatically include improvement suggestions in a report based on all aggregation results.

[0109] Furthermore, in the above embodiment, the support module 2034 inputs the aggregation results for all perspectives and instructions for outputting test assignments, test improvement suggestions, or a combination thereof recognized from the aggregation results, in a manner that is consistent with the creation purpose, into the trained model, thereby causing the trained model to output test assignments, test improvement suggestions, or a combination thereof, in a manner that is consistent with the creation purpose, based on the information contained in the aggregation results. As a result, improvement suggestions that take the creation purpose into consideration and are based on all the aggregation results, are automatically included in the report.

[0110] <Modification> In the above embodiment, the LLM system 30 is configured to output a final report. However, the final report may be output from a trained model stored in the server 20, rather than from the LLM system 30. The trained model is a model for outputting a final report based on aggregated data. Specifically, the trained model is a large-scale artificial intelligence model used in the field of natural language processing (NLP) that has undergone additional training. The additional training involves teaching the correct answer for each task, or so-called fine tuning, to specialize the model in creating a final report. Specifically, for example, the trained model is trained to output information contained in input aggregated data in a manner conforming to the specifications of the final report. In this case, the learning data for training the trained model is, for example, aggregated data of multiple items as input data, and the output format of the final report based on the input aggregated data is the correct output data. For example, by inputting aggregated data and an instruction statement for outputting the information contained in the aggregated data in a manner conforming to the specifications of the final report into the trained model, the trained model outputs the information contained in the aggregated data in a manner conforming to the specifications of the final report.

[0111] Furthermore, context tuning may be performed on the trained model, for example, by including example answers, answer formats, and the like in the instruction text so that the trained model outputs information in a format desired by the user.

[0112] The trained model may be retrained at any time in accordance with changes in the output format of the final report, etc.

[0113] In the above embodiment, an example has been described in which a final report including report contents of multiple items is output from the trained model. However, what is output from the trained model is not limited to the final report. The trained model may output report contents related to at least one item that may be included in the final report. For example, the assistance module 2034 inputs information aggregated for a predetermined perspective into the trained model. The trained model outputs report contents related to items corresponding to the input aggregate information.

[0114] In the above embodiment, the server 20 has the prompt table 2023. However, the server 20 does not necessarily have to have the prompt table 2023. For example, the server 20 may have a predetermined link module, and the link module may include a prompt template. The link module is, for example, an API (Application Programming Interface). Note that the link module may not be provided within the server 20, but may be provided on a server different from the server 20.

[0115] Specifically, for example, in step S15, the support module 2034 outputs the aggregated data and the purpose of creation to the linking module. The linking module creates an instruction statement by combining the information held as a template with the aggregated data and the purpose of creation. The linking module outputs the aggregated data and the created instruction statement to the LLM system 30.

[0116] In addition, in the above embodiment, the case where the trained model is an LLM has been described as an example. However, the trained model according to this embodiment may be a trained model other than an LLM.

[0117] In the above embodiment, an example of the configuration in which the tallying module 2033 and the support module 2034 are provided in the server 20 has been described, but the present invention is not limited to this configuration, and the functions of the tallying module 2033 and the support module 2034 may be provided in the terminal device 10. In this case, for example, some or all of the information stored in the storage unit 202 may be stored in the terminal device 10.

[0118] <4 Basic computer hardware configuration> 19 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main memory device 92, an auxiliary memory device 93, and a communication IF (interface) 99. These are electrically connected to one another by a bus.

[0119] The processor 91 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

[0120] The main storage device 92 is used to temporarily store programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0121] The auxiliary storage device 93 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0122] The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using wired or wireless communication standards. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can connect to the Internet via a predetermined access point. In the case of a wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), and Bluetooth (registered trademark). In the case of a wired connection, the network also includes a direct connection using a USB (Universal Serial Bus) cable, etc.

[0123] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration across multiple computers 90 and interconnecting them via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0124] <Basic functional configuration of computer 90> A description will be given of the functional configuration of a computer realized by the basic hardware configuration of a computer 90 shown in Fig. 19. The computer includes at least the functional units of a control unit, a storage unit, and a communication unit.

[0125] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 interconnected via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0126] The control unit is realized by the processor 91 reading various programs stored in the auxiliary storage device 93, expanding them in the main storage device 92, and executing processing in accordance with the programs. The control unit can realize functional units that perform various types of information processing depending on the type of program. In this way, the computer is realized as an information processing device that processes information.

[0127] The storage unit is realized by a main storage device 92 and an auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Furthermore, the processor 91 can allocate a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 in accordance with the programs. Furthermore, the control unit can cause the processor 91 to execute processes for adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0128] A database refers to a relational database, which manages data sets called tables, which are structured by rows and columns, by relating them to each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated. Typically, each table has a column set as a key for uniquely identifying a record, but setting a key to a column is not essential. The control unit can cause the processor 91 to add, delete, or update records in a specific table stored in the storage unit according to various programs.

[0129] The communication unit is realized by the communication IF 99. The communication unit realizes the function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 91 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0130] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0131] Although several embodiments of the present disclosure have been described above, these embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are intended to be included in the scope of the inventions and their equivalents as defined in the claims, as well as in the scope and spirit of the inventions.

[0132] <Additional Notes> The matters described in the above embodiments will be supplemented below. (Appendix 1) A program to be executed by a computer having a processor and a memory, the program causing the processor to execute the steps of: acquiring test results; aggregating information contained in the test results based on perspectives to be summarized in a report; inputting the aggregated results based on the perspectives into a trained model that has been trained to output the information contained in the aggregated results in a manner conforming to the specifications of the report, thereby outputting the information contained in the aggregated results in a manner conforming to the specifications of the report; and presenting the information output from the trained model to a user. (Appendix 2) The trained model is a large-scale language model, and in the step of outputting information, the program described in Appendix 1 outputs the information contained in the aggregated results from the trained model in a manner that conforms to the report specifications by inputting the aggregated results and instructions for outputting the information contained in the aggregated results in a manner that conforms to the report specifications. (Appendix 3) A program described in Appendix 1, which causes a processor to execute a step of receiving the purpose of creating a report from a user and outputting information, wherein the trained model is trained to output the information contained in the aggregation results in a manner that matches the report specifications and with content that is in line with the purpose of creating when the aggregated results based on a viewpoint and the purpose of creating are input, and by inputting the aggregated results and the purpose of creating into the trained model, the information contained in the aggregation results is output in a manner that matches the report specifications and with content that is in line with the purpose of creating. (Appendix 4) A program described in Appendix 1, which causes a processor to execute a step of receiving the purpose of creating a report from a user, wherein the trained model is a large-scale language model, and in a step of outputting information, inputs into the trained model the aggregated results and instructions for outputting the information contained in the aggregated results in a manner that conforms to the specifications of the report and with content that conforms to the purpose of creation, thereby causing the trained model to output the information contained in the aggregated results in a manner that conforms to the specifications of the report and with content that conforms to the purpose of creation. (Appendix 5) A program described in (Appendix 3) or (Appendix 4) whose purpose is to produce a periodic report, a further proposal, an explanatory report, or a combination of at least two of these. (Appendix 6) A program described in (Appendix 2) or (Appendix 4), in which, in the step of outputting information, the trained model is input with the aggregated results of the aspects to be referenced when analyzing failures that occur in the test results and an instruction statement for outputting a report on the aspects that should be focused on in the analysis based on the information contained in the aggregated results, thereby causing the trained model to output a report on the aspects that should be focused on in the analysis based on the information contained in the aggregated results. (Appendix 7) A program described in any one of (Appendix 2), (Appendix 4), and (Appendix 6), in which, in the step of outputting information, the trained model is input with the aggregated results of the viewpoints to be referenced when analyzing failures that occur in the test results and an instruction statement for outputting a table relating to the information included in the aggregated results, a drawing relating to the information included in the aggregated results, or a combination thereof, thereby causing the trained model to output a table relating to the information included in the aggregated results, a drawing relating to the information included in the aggregated results, or a combination thereof, based on the information included in the aggregated results. (Appendix 8) A program described in any one of (Appendix 2), (Appendix 4), (Appendix 6), and (Appendix 7), in which, in the step of outputting information, the aggregated results of multiple perspectives that are referenced when analyzing failures that occur in the test results and an instruction statement for outputting a summary of the failures are input to the trained model, thereby causing the trained model to output a summary of the failures based on the information contained in the aggregated results. (Appendix 9) A program described in any one of (Appendix 2), (Appendix 6), (Appendix 7), and (Appendix 8), in which, in the step of outputting information, the aggregated results for all perspectives and instructions for outputting test tasks, test improvement suggestions, or a combination thereof recognized from the aggregated results are input to the trained model, thereby causing the trained model to output test tasks, test improvement suggestions, or a combination thereof based on the information contained in the aggregated results. (Appendix 10) In the step of outputting information, the program described in Appendix 4 outputs test tasks, test improvement suggestions, or a combination thereof, based on the information contained in the aggregation results, from the trained model, by inputting the aggregation results for all perspectives and instructions for outputting test tasks, test improvement suggestions, or a combination thereof, recognized from the aggregation results, in a manner that is in line with the purpose of creation, along with the training model. (Appendix 11) A method executed by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 10). (Appendix 12) An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any of the inventions according to (Appendix 1) to (Appendix 10). (Appendix 13) A system comprising means for executing all steps performed in any of the inventions according to (Appendix 1) to (Appendix 10). [Explanation of symbols]

[0133] 1. System 10...Terminal device 120…Communications Department 13...Input device 131...Touch-sensitive devices 14...Output device 15...Memory 16…Storage 19...Processor 20...Server 22...Communication IF 23...Input / output interface 25…Memory 26…Storage 29...Processor 30...LLM System

Claims

1. A program to be executed by a computer having a processor and a memory, the program causing the processor to: obtaining test results; aggregating the information contained in the test results based on aspects that can be summarized in a report; a step of inputting an aggregated result based on a viewpoint into a trained model that has been trained to output information included in the aggregated result in a manner conforming to a report specification, thereby outputting the information included in the aggregated result in a manner conforming to the report specification; presenting information output from the trained model to a user; A program that executes the following.

2. the trained model is a large-scale language model, 2. The program of claim 1, wherein in the step of outputting the information, the aggregated results and an instruction statement for outputting the information contained in the aggregated results in a manner conforming to the specifications of the report are input to the trained model, thereby causing the trained model to output the information contained in the aggregated results in a manner conforming to the specifications of the report.

3. having the processor execute a step of accepting from the user a purpose for creating the report; 2. The program of claim 1, wherein in the step of outputting the information, the trained model is trained to output the information contained in the aggregation results in a manner that matches the specifications of the report and with content that is in line with the purpose of creation when the aggregated results based on a viewpoint and the purpose of creation are input, and by inputting the aggregated results and the purpose of creation into the trained model, the information contained in the aggregation results is output in a manner that matches the specifications of the report and with content that is in line with the purpose of creation.

4. having the processor execute a step of accepting from the user a purpose for creating the report; the trained model is a large-scale language model, 2. The program of claim 1, wherein in the step of outputting the information, the aggregated results and an instruction statement for outputting the information contained in the aggregated results in a manner that conforms to the specifications of the report and with content that conforms to the purpose of creation are input to the trained model, thereby causing the trained model to output the information contained in the aggregated results in a manner that conforms to the specifications of the report and with content that conforms to the purpose of creation.

5. 4. The program according to claim 3, wherein the purpose of creation is a periodic report, an additional proposal, an explanatory report, or a combination of at least two of these.

6. 3. The program of claim 2, wherein in the step of outputting the information, the trained model is input with an aggregated result of the viewpoints to be referenced when analyzing the failures that occurred in the test results and an instruction statement for outputting a report on the viewpoints that should be focused on in the analysis based on the information contained in the aggregated result, thereby causing the trained model to output a report on the viewpoints that should be focused on in the analysis based on the information contained in the aggregated result.

7. 3. The program of claim 2, wherein in the step of outputting the information, the trained model is input with an instruction for outputting the aggregated results of the viewpoints to be referenced when analyzing the failures that occurred in the test results, and a table relating to the information contained in the aggregated results, a drawing relating to the information contained in the aggregated results, or a combination thereof, thereby causing the trained model to output a table relating to the information contained in the aggregated results, a drawing relating to the information contained in the aggregated results, or a combination thereof, based on the information contained in the aggregated results.

8. 3. The program of claim 2, wherein in the step of outputting the information, the trained model is input with the aggregated results of multiple perspectives to be referenced when analyzing the faults that occurred in the test results and an instruction statement for outputting a summary of the faults, thereby causing the trained model to output a summary of the faults based on the information contained in the aggregated results.

9. The program of claim 2, wherein in the step of outputting the information, the aggregated results for all perspectives and instructions for outputting test issues, test improvement suggestions, or a combination thereof recognized from the aggregated results are input to the trained model, thereby causing the trained model to output test issues, test improvement suggestions, or a combination thereof based on the information contained in the aggregated results.

10. The program of claim 4, wherein in the step of outputting the information, the trained model is input with the aggregated results for all perspectives and instructions for outputting test tasks, test improvement suggestions, or combinations thereof recognized from the aggregated results, with content that is in line with the purpose of creation, thereby causing the trained model to output test tasks, test improvement suggestions, or combinations thereof, with content that is in line with the purpose of creation, based on the information contained in the aggregated results.

11. A method implemented on a computer having a processor and a memory, wherein the processor performs all of the steps performed in the invention according to any one of claims 1 to 10.

12. 11. An information processing device comprising a processor and a memory, wherein the processor executes all of the steps executed in any one of the inventions according to claims 1 to 10.

13. A system comprising means for executing all steps performed in any one of the inventions according to claims 1 to 10.

Images