Program, method, information processing apparatus, system

Abstract

Reduce the burden of creating a report on the results of software testing. 【Solution means】 A program for causing a computer including a processor and a memory to execute. The program causes the processor to perform steps of obtaining test results, aggregating information included in the test results based on viewpoints that can be summarized in a report, and inputting the aggregation result based on the viewpoints to a learned model that has been learned to output information included in the aggregation result in a manner conforming to the specifications of the report, thereby causing the information included in the aggregation result to be output in a manner conforming to the specifications of the report, and presenting the information output from the learned model to the user.

Description

Technical Field

[0001] The present disclosure relates to a program, a method, an information processing apparatus, and a system.

Background Art

[0002] In order to ensure its quality, software is tested for functionality before launch. The testing may be performed by a software development company or by an external company different from the software development company. When outsourcing the testing to an external company, the company undertaking the testing needs to report the results of the performed testing to the software development company that commissioned the testing.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is laborious to create a report for reporting the results of the performed testing. Therefore, a technology capable of reducing the burden of creating the report is desired.

[0005] In Patent Document 1, the correspondence relationship between the defect content of software and its defect classification is learned in advance. When receiving a request for a natural language string that requires a document used to improve the defect classification, after estimating the defect classification specified by the natural language string by language analysis, the response with the highest accuracy is given. In Patent Document 1, the client terminal aggregates the classification results of the defect classification and displays the aggregation result to the user. However, Patent Document 1 does not describe supporting the creation of a report on the results of software testing.

[0006] The object of the present disclosure is to reduce the burden of creating a report on the results of software testing.

Means for Solving the Problem

[0007] A program for causing a computer including a processor and a memory to execute. The program causes the processor to execute steps of: obtaining test results; aggregating information included in the test results based on viewpoints that can be summarized in a report; inputting the aggregation result based on the viewpoints into a learned model that has been learned to output information included in the aggregation result in a manner conforming to the specifications of the report, thereby causing the information included in the aggregation result to be output in a manner conforming to the specifications of the report; and presenting the information output from the learned model to the user.

Advantages of the Invention

[0008] According to the present disclosure, the burden of creating a report on the results of software testing can be reduced.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals.

[0011] <Overview> The system according to this embodiment creates a final report on debugging by aggregating the results of debugging software in a predetermined classification and inputting the aggregation results into a learned model. For example, a large language model (LLM) may be used as the learned model.

[0012] <Overall Configuration Diagram of the Entire System> FIG. 1 is a block diagram showing an example of the overall configuration of System 1. The System 1 shown in FIG. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

[0013] In FIG. 1, an example is shown in which System 1 includes two terminal devices 10, but the number of terminal devices 10 included in System 1 is not limited to two. The number of terminal devices 10 included in System 1 may be one, or may be three or more.

[0014] In FIG. 1, an example is shown in which System 1 includes one LLM system 30, but the number of LLM systems 30 included in System 1 is not limited to one. The number of LLM systems 30 included in System 1 may be two or more.

[0015] In this embodiment, an aggregate of a plurality of devices may be regarded as one server. The way of distributing the plurality of functions required to implement the server 20 according to this embodiment for one or more pieces of hardware can be appropriately determined in view of the processing capabilities of each piece of hardware and / or the specifications required for the server 20. Specifically, for example, a server having a function of creating a test case based on a specification document and a server in which a learned model is built may be different devices.

[0016] The terminal device 10 shown in FIG. 1 is an information processing device used by a user who utilizes the services provided by the server 20. For example, the terminal device 10 is an information processing device operated by a user who utilizes the support for creating an end report provided by the server 20 to create an end report for software debugging processing. The terminal device 10 is realized, for example, by a stationary PC (Personal Computer), a laptop PC, a head-mounted display, or the like. Further, the terminal device 10 may be a computer having portability such as a smartphone or a tablet terminal.

[0017] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving an input operation from the user (for example, a pointing device such as a touch panel, a touch pad, a mouse, etc., a keyboard, etc.). The output device 14 is a device for presenting information to the user (a display, a speaker, etc.).

[0018] The server 20 is an information processing device that provides a service for supporting the creation of an end report. For example, the server 20 uses a learned model to support the creation of an end report for debugging processing.

[0019] The server 20 is an information processing device realized, for example, by a computer connected to the network 80. As shown in FIG. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an interface with an input device for receiving an input operation from the user and an output device for outputting information to the user.

[0020] The LLM system 30 is a system in which a large-scale artificial intelligence model (LLM) used in the field of natural language processing (NLP), for example, is constructed. By learning a large amount of text data (web pages, books, articles, etc.), the LLM can understand the patterns of the language used by humans and effectively execute natural language generation (NLG) tasks.

[0021] The LLM is used in many NLP tasks such as generating responses to specific questions, automatically generating articles, summarizing text, translation, sentiment analysis, etc. It can also be utilized in various applications such as education, entertainment, customer service, product development, etc. There are, for example, the following types of LLM. · GPT-4 (registered trademark) (OpenAI) · PaLM2 (registered trademark) (Google) · StableLM (StableAI) · Llama2 (Meta)

[0022] The LLM system 30 inputs information about the test output from the server 20 and the prompt into the LLM, and causes the LLM to output a response based on the input text data and the prompt. The LLM system 30 transmits the response output from the LLM to the server 20.

[0023] Each information processing device is composed of a computer equipped with an arithmetic device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. Regarding each of the terminal device 10 and the server 20, descriptions overlapping with the basic hardware configuration of the computer and the basic functional configuration of the computer to be described later are omitted.

[0024] <1.1 Configuration of Terminal Device> FIG. 2 is a block diagram showing a configuration example of the terminal device 10 shown in FIG. 1. As shown in FIG. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected by, for example, a bus or the like.

[0025] The communication unit 120 performs processes such as modulation / demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0026] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 is realized by, for example, a touch-sensitive device 131 where an instruction is input by touching the operation surface. When the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, a keyboard, a mouse, or the like. The input device 13 converts the instruction input from the user into an electrical signal and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a reception port for receiving an electrical signal input from an external input device.

[0027] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is realized by, for example, a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is realized by, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like.

[0028] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts the signal given from the microphone 171 into a digital signal and gives the converted signal to the control unit 190. Also, the audio processing unit 17 gives the audio signal to the speaker 172. The audio processing unit 17 is realized by, for example, a processor for audio processing. The microphone 171 receives an audio input and gives an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal given from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0029] The camera 160 is a device that receives light by a light receiving element and outputs it as a shooting signal.

[0030] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals from at least three or four satellites are received, and based on the received signals, the current position of the terminal device 10 on which the GPS module is mounted is detected. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0031] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, the user information 181.

[0032] The user information 181 includes, for example, information about the user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0033] The control unit 190 is realized by the processor 19 reading the program stored in the storage unit 180 and executing the instructions included in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 functions as an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0034] The operation reception unit 191 performs processing for receiving an instruction or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives an instruction or information input from the touch-sensitive device 131 or the like.

[0035] Also, the operation reception unit 191 receives a voice instruction input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal input from the microphone 171 and converted into a digital signal by the voice processing unit 17. The operation reception unit 191 obtains an instruction from the user, for example, by analyzing the received voice signal and extracting a predetermined noun.

[0036] The transmission / reception unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 according to a communication protocol. Specifically, for example, the transmission / reception unit 192 transmits information input by the user or an instruction from the user to the server 20. Also, the transmission / reception unit 192 receives information provided from the server 20.

[0037] The presentation control unit 193 controls the output device 14 in order to present the information provided from the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the information transmitted from the server 20 to be displayed on the display 141. Also, the presentation control unit 193 causes the information transmitted from the server 20 to be output from the speaker 172.

[0038] <1.2 Functional Configuration of the Server> FIG. 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG. 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0039] The communication unit 201 performs processes for the server 20 to communicate with external devices.

[0040] The storage unit 202 is realized by, for example, a memory 25, a storage 26, etc., and stores data and programs used by the server 20. The storage unit 202 has, for example, a user information table 2021, a test information table 2022, and a prompt table 2023, etc. Note that the information stored in the storage unit 202 is not limited to these.

[0041] The user information table 2021 is a table that stores information about users. Details will be described later.

[0042] The test information table 2022 is a table that stores information regarding test results. Details will be described later. In the present embodiment, the information regarding test results includes, for example, the execution results of tests for a plurality of test suites. In the present embodiment, a test suite includes, for example, information regarding a set of test cases executed by a test performer. The information regarding a set of test cases executed by a test performer includes, for example, the execution results of tests for a plurality of test cases in a predetermined process. The process may be, for example, a predetermined process in a predetermined service, or a predetermined process in a predetermined project of a predetermined service. The service, project, and process may be referred to as attributes, for example. In addition to these, the attributes may include industry / sector, system type, etc.

[0043] Test cases are defined, for example, by a test design document. Information about a test case is, for example, information obtained by converting the test case into a form that enables test execution management. Information about a test case includes, for example, "test target item / target function". "Test target item / target function" represents, for example, the item / function to be tested. "Test target item / target function" may also be referred to as a test category. The test category may include major functions (category 1), medium functions (category 2), and minor functions (category 3) as subordinate concepts. Information about a test case may include, for example, a test perspective. A test perspective represents, for example, the basis (way of thinking) for a test. The test perspective is associated with the test category. Confirmation items may be associated with the test perspective. Confirmation items represent information for specifying the tests to be performed.

[0044] Information about a test case may include, for example, a schedule until the product is launched. The schedule until product launch includes, for example, the product development schedule and may be represented in units of a predetermined period. Information about a test case may include, for example, information regarding the progress of the executed test. Information regarding the progress of the test includes, for example, information representing the status of the test case. Information representing the status of the test case includes, for example, "OK" indicating that the test has passed, "Fixed" indicating that defects have been fixed, "NG" indicating that a defect has occurred, "Pending" indicating that the execution of the test is on hold, "Excluded" indicating that the test is not applicable, etc.

[0045] Information about a test case may include, for example, information regarding an issue ticket. An issue ticket is, for example, a communication to the developer issued based on the executed test. Information regarding the issue ticket includes the purpose of the issue ticket, for example, "defect", "inquiry", or "request", etc. Information about a test case may include, for example, the reason for closing. Reasons for closing include, for example, non-reproducible, resolved and released after reproduction, etc.

[0046] Information about test cases may include, for example, information about the test environment. Information about the test environment represents the environment in which the test is executed and includes, for example, the OS (Operating System) used on the terminal where the test target is executed, the browser used on the terminal where the test target is executed, the model of the terminal where the test target is executed, and the like.

[0047] Information about test cases may include, for example, information about the process to be detected and information about the embedding process. Information about the process to be detected and information about the embedding process are, for example, information about items included in the format of the end report but not included on the data side.

[0048] The prompt table 2023 is a table that stores information about prompts (instructions). Details will be described later.

[0049] The control unit 203 is realized by the processor 29 reading the program stored in the storage unit 202 and executing the instructions included in the program. The program includes applications such as a web browser application. The program includes programming languages such as JavaScript (registered trademark) that are executed on the web browser application stored in the terminal device 10. By operating according to the program, the control unit 203 exhibits the functions shown as the reception control module 2031, the transmission control module 2032, the aggregation module 2033, the support module 2034, and the presentation control module 2035.

[0050] The reception control module 2031 controls the process of the server 20 receiving a signal from an external device according to a communication protocol.

[0051] The transmission control module 2032 controls the process of the server 20 transmitting a signal to an external device according to a communication protocol.

[0052] The aggregation module 2033 aggregates information on test results specified by the user from a predetermined perspective. Specifically, for example, the aggregation module 2033 aggregates information on a plurality of test cases included in the information on the test results from a predetermined perspective. The predetermined perspective includes, for example, perspectives such as schedule, test progress, ticket classification, function, test perspective, test environment, process to be detected, embedded process, etc. The predetermined perspective is based on, for example, items in which test results are reported in the final report.

[0053] The final report is, for example, a document that summarizes the content of the report on the execution results of the test. In the final report, for example, the content of the report is summarized by chapter. Each chapter includes a plurality of items corresponding to the chapter. Each chapter includes a document summarized by item. The document for each item is created by, for example, one or more slides. The documents for a plurality of items may be created by one slide.

[0054] The final report includes, for example, the following chapters. · Test Results · Defect Analysis · Defect Report · Improvement Proposals

[0055] "Test Results" is a chapter that reports the results of the test. The test results include, for example, items that report the execution status of the test according to the schedule and items that report the progress status of the test according to the schedule. The execution status of the test represents, for example, the transition of the number of test executions. The progress status of the test represents, for example, the transition of the status change of the test case. The test results include, for example, items that report the execution status of the test and the progress status of the test with respect to the number of days elapsed. The test results include, for example, items that report the defect rate that occurred in the executed test. The test results include, for example, items that report the classification of defects that occurred in the executed test. Defects can be classified, for example, based on a predetermined perspective. For example, the classification of defects can be classified based on tickets, status, and close reasons.

[0056] "Failure Analysis" is a chapter that reports the content of the analysis of failures (defects) that occurred during testing. Failure analysis includes items that report the tendency of failures by function, for example, based on the number of occurrences and occurrence rate of failures for each function (test category). Failure analysis includes items that report the tendency of failures from each test perspective, for example, based on the number of occurrences and occurrence rate of failures for each test perspective. Failure analysis includes items that report the tendency of failures for each test environment, for example, based on the number of occurrences and occurrence rate of failures for each test environment. Failure analysis includes items that report the tendency of failures for each process to be detected, for example, based on the number of occurrences and occurrence rate of failures for each process to be detected. Failure analysis includes items that report the tendency of failures for each embedding process, for example, based on the number of occurrences and occurrence rate of failures for each embedding process. Failure analysis includes, for example, the analysis results that summarize the tendency of failures, issues, and assumed risks for each item.

[0057] "Failure Report" is a chapter that reports the details of failures that occurred during testing. The failure report includes information such as the time, environment, test executor, and developers related to the test target when the test in which the failure occurred was executed.

[0058] "Improvement Proposals" is a chapter that proposes improvement measures for the test system or development system based on the test results. Improvement proposals include items that report issues with the test system or development system recognized based on the test results, for example. Improvement proposals include proposals for the ideal state that the test system or development system aims for in order to improve the issues recognized based on the test results, for example. Improvement proposals include proposals for measures to reach the ideal state, for example.

[0059] The support module 2034 supports the user in creating an end report. Specifically, for example, the support module 2034 creates report content based on a predetermined perspective by, for example, inputting information such as aggregated test results into a learned model. More specifically, the support module 2034, for example, inputs information aggregated based on a predetermined perspective into a learned model and causes the learned model to output report content for items corresponding to the perspective. The support module 2034 inputs information aggregated for each perspective into a learned model and causes the learned model to output report content for a plurality of items. This can be equivalently stated as the support module 2034 causing the learned model to input the aggregation result and output the information contained in the aggregation result in a manner conforming to the specifications of the end report. The learned model outputs a plurality of report contents together as a first draft of the end report.

[0060] Specifically, for example, the support module 2034 uses the LLM system 30 when creating the end report. For example, the support module 2034 sends information aggregated from the test results selected by the user and an instruction text for outputting report content for items corresponding to the aggregated information to the LLM system 30. The LLM system 30 inputs the aggregation result and the instruction text sent from the server 20 into the LLM and causes the LLM to output the end report.

[0061] The support module 2034 issues an instruction text for outputting the end report, for example, in response to a predetermined button being pressed by the user. The instruction text may include, for example, content for explaining the composition of the end report. The instruction text may include, for example, content for specifying the output format of the report content. The instruction text may include, for example, content for specifying the aggregated data to be used when creating the report content. The instruction text may include, for example, content for outputting a report on a perspective to be noted in the aggregated data. The support module 2034 may convert the aggregated data into data in a predetermined format and send it to the LLM system 30. The predetermined format includes, for example, the Markdown format.

[0062] The support module 2034 may receive the purpose of creating the end report from the user and reflect the received purpose of creation in the end report. For example, the support module 2034 receives an input of the purpose of creating the end report from the user. The support module 2034 reflects the input purpose of creation in the instruction text. The support module 2034 transmits the instruction text reflecting the purpose of creation to the LLM system 30. For example, the support module 2034 stores an instruction text in which the words corresponding to the purpose of creation are rewritable. For example, the support module 2034 reflects the input purpose of creation in the corresponding area of the instruction text and transmits the reflected instruction text to the LLM system 30. For example, the support module 2034 may have an instruction text for each purpose of creation. At this time, the support module 2034 reads out the instruction text corresponding to the input purpose of creation and transmits the read instruction text to the LLM system 30.

[0063] The presentation control module 2035 controls the process of presenting information to the user. For example, the presentation control module 2035 presents the information output from the learned model to the user. The presentation control module 2035 may present the information output from the learned model to the user in the output format (for example, Markdown format), or may convert it into a predetermined format (for example, PPT file format) and present it to the user. Specifically, for example, the presentation control module 2035 presents the end report output from the LLM system 30 to the user.

[0064] <2 Data Structure> Figures 4 to 7 are diagrams showing the data structure of the tables stored in the server 20. Note that Figures 4 to 7 are examples and do not exclude data not described. Also, even data described in the same table may be stored in separate storage areas in the storage unit 202.

[0065] Figure 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Figure 4 is a table having columns of name, age, gender, date of birth, and contact information with the user ID as the key.

[0066] The user ID is an item for storing an identifier for uniquely identifying a user. The name is an item for storing the name of the user. The age is an item for storing the age of the user. The gender is an item for storing the gender of the user. The date of birth is an item for storing the date of birth of the user. The contact information is an item for storing the contact information (e.g., phone number, email address, etc.) of the terminal device 10 that the user has.

[0067] Figure 5 is a diagram showing the data structure of the test information table 2022. The test information table 2022 shown in Figure 5 is a table having columns such as service, project, process, test case, test executor, execution date and time, etc. with the test result ID as the key. Note that the information stored in Figure 5 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0068] The test result ID is an item for storing an identifier for uniquely identifying a test suite. The service is an item for storing information for identifying, for example, the service related to the test suite. In the item "service", a service ID may be stored, or a service name may be stored. The project is an item for storing information for identifying, for example, the project related to the test suite. In the item "project", a project ID may be stored, or a project name may be stored.

[0069] The process is an item that stores information for identifying, for example, a process involving a test suite. In the item "process", the process ID may be stored, or the process name may be stored. The test case is an item that stores the number of test cases included in the test suite. The test executor is an item that stores the executor of the test. In the item "test executor", the user ID may be stored, or the user name may be stored. The execution date and time is an item that stores the date and time when the test was executed.

[0070] FIG. 6 is a diagram showing the data structure of the test case information table 20221 regarding information about test cases included in a test suite. The test case information table 20221 shown in FIG. 6 is a table having columns such as a test result ID, major function, medium function, minor function, test perspective, execution schedule, status, ticket issuance, environment, test executor, execution date and time, etc. with the test case ID as the key. Note that the information stored in FIG. 6 is not limited to these. For example, any of these may not be stored, or additional information may be stored.

[0071] The test case ID is an item that stores an identifier for uniquely identifying a test case. The major function is an item that stores the first classification as the test target item / target function. The medium function is an item that stores the second classification as the test target item / target function. The minor function is an item that stores the third classification as the test target item / target function. The test perspective is an item that stores the test perspective set for the test case. The execution schedule is an item that stores the schedule when the test case was executed. The status is an item that stores the current status of the test case. In the item "status", the date and time when the status was set may be stored together with the status. The ticket issuance is an item that stores the purpose for which a ticket was issued for the test case. The ticket issuance may be referred to as a ticket. The environment is an item that stores the environment in which the test case was executed. In the item "environment", for example, the model of the terminal on which the test case was executed, the OS used on the terminal, the browser, etc. are stored.

[0072] FIG. 7 is a diagram showing the data structure of the prompt table 2023. The prompt table 2023 shown in FIG. 7 is a table having columns such as prompt data with the prompt ID as a key. Note that the information stored in FIG. 7 is not limited to these. For example, further information may be stored. For example, it may have a column for storing the purpose of creation. Thereby, for example, the support module 2034 can select an instruction sentence for each purpose of creation.

[0073] The prompt ID is an item for storing an identifier for uniquely identifying a prompt (instruction sentence). The prompt data is an item for storing data related to a prompt that is an instruction sentence to be input to the LLM. The data related to the prompt is, for example, text information related to the prompt.

[0074] The text information related to the prompt is, for example, an instruction sentence for outputting a final report based on aggregated data, and includes, for example, the following contents. · The responsibilities of the user who creates the final report · The composition of chapters and items in the final report (that is, an explanation of the specifications of the final report) · The gist of the explanations described in the materials for each item (for example, it may include arguments from the aggregated data) · The format of the explanatory text described in the materials for each item · The format of the graph used in the materials for each item (no instruction is given for items where a graph is not required) · The format of the table used in the materials for each item (no instruction is given for items where a table is not required) · The gist of the explanation considering the purpose of creation

[0075] <3 Operations> The operation of the server 20 when assisting in creating the final report will be described. FIG. 8 is a flowchart showing an example of the operation of the server 20 when assisting in creating the final report.

[0076] First, the user (test engineer) who wants to create an end report accesses server 20. The user logs in to the service provided by server 20 to assist in creating the end report by, for example, entering their user ID and password.

[0077] The user operates terminal device 10 and selects the test results for creating the end report. Specifically, for example, the user selects the test results belonging to an attribute by selecting a predetermined attribute. More specifically, for example, the user selects the test suite associated with the selected service by selecting the service involving the test suite. Also, for example, the user selects the test suite associated with the selected project by selecting the project involving the test suite. Also, for example, the user selects the test suite associated with the selected process by selecting the process involving the test suite.

[0078] FIG. 9 is a schematic diagram showing an example of a screen when selecting data used for creating an end report. The selection screen shown in FIG. 9 includes a first area 1411 and an operation object 1412.

[0079] The first area 1411 is an area for selecting the test results used when creating the end report. In the example shown in FIG. 9, the presentation control unit 193 displays the service, project, and process so that they can be selected. Thereby, the user can select the test suite by selecting the service, project, and process. The operation object 1412 is an object for proceeding with the creation process of the end report to the next step.

[0080] In step S11, server 20 receives from the user the specification of information regarding the test results. Specifically, for example, terminal device 10 receives from the user the selection of the test results used when creating the end report and transmits to server 20 the information that can identify the selected test results. The reception control module 2031 receives the information transmitted from terminal device 10.

[0081] In step S12, the server 20 aggregates information regarding the selected test results from a predetermined perspective. Specifically, for example, the aggregation module 2033 acquires information regarding a plurality of test cases included in the information regarding the selected test results from the test information table 2022, and aggregates from perspectives such as a schedule perspective, a test progress perspective, a ticket classification perspective, a function-by-function perspective, a test perspective-by-test perspective, a test environment-by-test environment perspective, a process-to-be-detected perspective, an embedded process perspective, etc. The aggregation module 2033 associates the aggregation result obtained by the aggregation with the items in the end report.

[0082] In step S13, the server 20 presents data to be used for creating the end report to the user. Specifically, for example, the presentation control module 2035 presents the information obtained by aggregating the information regarding the test results to the user. Also, for example, the presentation control module 2035 presents to the user the items in the end report and the perspectives related to the aggregation that are associated with each other. Further, the presentation control module 2035 presents an overview of the aggregation result to the user. The transmission control module 2032 transmits, for example, information for displaying the information obtained by the aggregation to the terminal device 10. The presentation control unit 193 displays the information transmitted from the server 20 on the display 141.

[0083] FIG. 10 is a schematic diagram showing an example of a screen when allowing the user to confirm data to be used for creating the end report. The confirmation screen shown in FIG. 10 includes a second area 1413 and a third area 1414.

[0084] The second area 1413 is an area for displaying information obtained by aggregating information regarding test results. In the second area 1413, it is indicated which aggregation result has been obtained from which perspective by the aggregation process. For example, if there is no information stored in the items related to the aggregation, the aggregation result cannot be obtained. In the example shown in FIG. 10, the aggregation result aggregated from the perspective of schedule, the aggregation result aggregated from the perspective of ticket classification, the aggregation result aggregated from the perspective of function type, the aggregation result aggregated from the perspective of test viewpoints, and the aggregation result aggregated from the perspective of test environments have been obtained. Also, in the example shown in FIG. 10, the aggregation result aggregated from the perspective of test progress, the aggregation result aggregated from the perspective of processes to be detected, and the aggregation result aggregated from the perspective of embedded processes have not been obtained.

[0085] The third area 1414 is an area for associatively displaying the items in the end report and the summary of the aggregation results from the perspectives related to the aggregation. In the third area 1414, the item names of the end report and the summary of the aggregation results referred to in the items are displayed. The aggregation results to be referred to are attached with an instruction object 14131 in the second area 1413. By referring to the second area 1413 and the third area 1414, the user checks whether there are any problems with the data for creating the materials for the predetermined items in the end report. If there is a problem with the data used for creating the materials, for example, the user can delete or edit the data by pressing the edit icon displayed in the third area 1414.

[0086] In the example shown in FIG. 10, it shows the case where information for creating the materials of "4.3 Failure Tendency (by Test Viewpoint)" is displayed, but the confirmation screen may exist for each item of the end report. When the user finishes checking the data used for creating the end report, the user inputs an instruction to proceed to the next process to the terminal device 10.

[0087] In step S14, the server 20 receives an input for the creation purpose. Specifically, for example, the presentation control module 2035 causes the terminal device 10 to display an input screen for the creation purpose. For example, the presentation control module 2035 causes the terminal device 10 to display a plurality of creation purposes in a selectable manner. The user selects a creation purpose displayed on the display 141.

[0088] FIG. 11 is a schematic diagram showing an example of a screen when inputting a creation purpose. The input screen shown in FIG. 11 includes a fourth area 1415. The fourth area 1415 is an area for displaying a plurality of creation purposes in a selectable manner. In the example shown in FIG. 11, the presentation control unit 193 displays "Regular Report", "Additional Proposal", and "Explanation Report" in a selectable manner as creation purposes. The regular report is, for example, a regular report of test results. The additional proposal is, for example, a report that proposes predetermined content after identifying improvement points based on the test results. The explanation report is, for example, a report that details the content of the test results. The user selects one from the displayed creation purposes. The terminal device 10 transmits information for specifying the creation purpose to the server 20. The displayed creation purposes are not limited to "Regular Report", "Additional Proposal", and "Explanation Report", and may be at least any combination of two of "Regular Report", "Additional Proposal", and "Explanation Report". Also, the creation purpose is not limited to "Regular Report", "Additional Proposal", and "Explanation Report".

[0089] In step S15, the server 20 creates a final report. Specifically, for example, the support module 2034 reads an instruction statement for outputting a final report based on the aggregated data from the prompt table 2023. If an instruction statement is prepared for each creation purpose, the support module 2034 reads the instruction statement corresponding to the creation purpose from the prompt table 2023. If an instruction statement with a blank creation purpose is prepared, the support module 2034 reads the instruction statement with a blank creation purpose from the prompt table 2023 and inserts information based on the creation purpose into the blank to create an instruction statement. The support module 2034 outputs the aggregated data and the instruction statement to the LLM system 30.

[0090] The LLM system 30 inputs the aggregated data and instructions sent from the server 20 into the LLM and causes the LLM to output a final report. The LLM system 30 transmits the information output from the LLM to the server 20.

[0091] The presentation control module 2035 causes the terminal device 10 to display the final report sent from the LLM system 30. That is, for example, the transmission control module 2032 transmits information for displaying the final report sent from the LLM system 30 to the terminal device 10. The terminal device 10 displays the information sent from the server 20 to the user. The presentation control module 2035 may present the final report sent from the LLM system 30 to the user in the format output from the LLM system 30, for example, in Markdown format. Further, the presentation control module 2035 may convert the final report sent from the LLM system 30 into a predetermined file format for presentation (for example, PPT file format) and present the converted information to the user.

[0092] Figures 12 to 18 are schematic diagrams showing an example of the materials included in the final report. Figure 12 shows, for example, an example of the material for reporting the progress of the test in the chapter of "Test Results". When creating the material shown in Figure 12, the instructions may include, for example, the following contents. · Based on the aggregated data related to the test progress / defect rate, please create a title regarding the test progress. · Based on the aggregated data related to the test progress / defect rate, please create a table in the form of... · Based on the aggregated data related to the test progress / defect rate, please create a pie chart in the form of... · Based on the aggregated data related to the test progress / defect rate, please create a line graph in the form of... · Please explain the information to be noted in a balloon. Figure 13 shows, for example, an example of a document reporting defect classification based on ticket creation in the chapter "Fault Analysis". When creating the document shown in Figure 13, the instruction text may include, for example, the following content. · Based on the aggregated data related to the ticket creation classification, please create a title for the defects based on ticket creation. · Based on the aggregated data related to the ticket creation classification, please create a table in the form of... · Based on the aggregated data related to the ticket creation classification, please create a pie chart in the form of...

[0093] Figure 14 shows, for example, an example of a document reporting the failure tendency by function in the chapter "Fault Analysis". When creating the document shown in Figure 14, the instruction text may include, for example, the following content. · Based on the aggregated data related to each function, please create a title from the perspectives of the number of failures and the failure rate. · Based on the aggregated data related to each function, please create comments on the number of failures and the failure rate by function. · Based on the aggregated data related to each function, please create a table in the form of... · Based on the aggregated data related to each function, please create a line graph in the form of...

[0094] Figure 15 shows, for example, an example of a document reporting the failure tendency from different test perspectives in the chapter "Fault Analysis". When creating the document shown in Figure 15, the instruction text may include, for example, the following content. · Based on the aggregated data related to different test perspectives, please create a title from the perspectives of the number of failures and the failure rate. · Based on the aggregated data related to different test perspectives, please create comments on the number of failures and the failure rate by function. · Based on the aggregated data related to different test perspectives, please create a table in the form of... · Based on the aggregated data related to different test perspectives, please create a line graph in the form of...

[0095] FIG. 16 shows, for example, an example of a document reporting failure trends by test environment in the chapter of "Failure Analysis". When creating the document shown in FIG. 16, the instructions may include, for example, the following content. · Based on the aggregated data for each test environment, create a title for the occurrence of failures from the perspective of the test environment. · Based on the aggregated data for each test environment, create comments on the number of failures for each environment. · Based on the aggregated data for each test environment, create a table in the form of...

[0096] FIG. 17 shows, for example, an example of a document reporting the analysis results (summary) in the chapter of "Failure Analysis". When creating the document shown in FIG. 17, the instructions may include, for example, the following content. · Based on the aggregated data related to failures (by function, test perspective, test environment, process to be detected, embedded process, etc.), summarize the trends by perspective. · Based on the aggregated data related to failures, raise issues and assumed risks by perspective.

[0097] FIG. 18 shows, for example, an example of a document reporting issues and improvement plans for the issues in the chapter of "Improvement Proposals". When creating the document shown in FIG. 18, the instructions may include, for example, the following content. · Based on all the aggregated data, create issues. · Based on all the aggregated data and the purpose of creation {...}, create improvement proposals.

[0098] Note that in FIG. 18, a document with issues and improvement proposals is shown, but the issues and improvement proposals may be summarized in separate documents.

[0099] The user refers to the content of the end report displayed on the terminal device 10 and edits the content as necessary. For example, if there are no problems with the content of the end report, the user downloads the end report in a predetermined format.

[0100] As described above, in the above embodiment, the reception control module 2031 acquires the test results. The aggregation module 2033 aggregates the information included in the test results based on the viewpoints that can be summarized in a report. When the support module 2034 inputs the aggregation result based on classification, the support module 2034 inputs the aggregation result to a learned model that has been learned to output the information included in the aggregation result in a manner conforming to the specifications of the report, so as to cause the information included in the aggregation result to be output in a manner conforming to the specifications of the report. The presentation control module 2035 presents the information output from the learned model to the user. Thereby, the server 20 can automatically create a report based on the input test results.

[0101] Therefore, according to the program, method, information processing apparatus, and system according to the present embodiment, the burden of creating a report regarding the results of software testing can be reduced.

[0102] Also, in the above embodiment, the learned model is a large language model. The support module 2034 inputs the aggregation result and an instruction statement for outputting the information included in the aggregation result in a manner conforming to the specifications of the report to the learned model, so as to cause the learned model to output the information included in the aggregation result in a manner conforming to the specifications of the report. Thereby, the server 20 can create a report using the large language model.

[0103] In addition, in the above-described embodiment, the reception control module 2031 receives the purpose of creating a report from the user. The learned model is trained to output the information included in the aggregation result in a manner that conforms to the specification of the report and in content that conforms to the purpose of creation when the aggregation result based on the perspective and the purpose of creation are input. The support module 2034 inputs the aggregation result and the purpose of creation to the learned model, thereby causing the learned model to output the information included in the aggregation result in a manner that conforms to the specification of the report and in content that conforms to the purpose of creation. As a result, the server 20 can create a report that conforms to the user's purpose. The content of the report changes according to the purpose of creation. For example, the purpose of creation includes "regular report", "additional proposal", "explanation report", etc., and depending on the purpose of creation, for example, the analysis and consideration in the report change. In this embodiment, by also inputting the purpose of creation to the learned model, it is possible to output a report that conforms to the purpose of creation.

[0104] In addition, in the above-described embodiment, the reception control module 2031 receives the purpose of creating a report from the user. The learned model is a large language model. The support module 2034 inputs the aggregation result and an instruction statement for causing the learned model to output the information included in the aggregation result in a manner that conforms to the specification of the report and in content that conforms to the purpose of creation to the learned model, thereby causing the learned model to output the information included in the aggregation result in a manner that conforms to the specification of the report and in content that conforms to the purpose of creation. As a result, the server 20 can use the large language model to create a report that conforms to the purpose of creation.

[0105] Also, in the above embodiment, the support module 2034 inputs the aggregation result of viewpoints referred to when analyzing the failures occurring in the test results and an instruction statement for outputting a report on the viewpoints to be focused on in the analysis based on the information included in the aggregation result into the learned model, and causes the learned model to output a report on the viewpoints to be focused on in the analysis based on the information included in the aggregation result. As a result, in the item of analyzing the failures from a predetermined viewpoint regarding the failure analysis in the report, the report content analyzing the aggregation result is automatically loaded.

[0106] Also, in the above embodiment, the support module 2034 inputs the aggregation result of viewpoints referred to when analyzing the failures occurring in the test results and an instruction statement for outputting a table related to the information included in the aggregation result, a drawing related to the information included in the aggregation result, or a combination thereof into the learned model, and causes the learned model to output a table related to the information included in the aggregation result, a drawing related to the information included in the aggregation result, or a combination thereof based on the information included in the aggregation result. As a result, in the item of analyzing the failures from a predetermined viewpoint regarding the failure analysis in the report, a table representing the aggregation result, a drawing related to the aggregation result, or a combination thereof is automatically loaded.

[0107] Also, in the above embodiment, the support module 2034 inputs the aggregation results of a plurality of viewpoints referred to when analyzing the failures occurring in the test results and an instruction statement for outputting a summary of the failures into the learned model, and causes the learned model to output a summary of the failures based on the information included in the aggregation results. As a result, in the item of reporting the analysis result (summary) of the failure analysis in the report, the tendency of the failures and the issues / risks are automatically loaded as the summary of the analysis from the aggregation results for a plurality of viewpoints.

[0108] In the above-described embodiment, the support module 2034 inputs the aggregation results for all viewpoints and an instruction statement for causing the learned model to output test issues recognized from the aggregation results, test improvement proposals, or a combination thereof to the learned model. Based on the information included in the aggregation results, the learned model outputs test issues, test improvement proposals, or a combination thereof. As a result, improvement proposals can be automatically included in the report based on all the aggregation results.

[0109] In the above-described embodiment, the support module 2034 inputs the aggregation results for all viewpoints, test issues recognized from the aggregation results, test improvement proposals, or a combination thereof, and an instruction statement for causing the learned model to output them in content conforming to the purpose of creation to the learned model. Based on the information included in the aggregation results, the learned model outputs test issues, test improvement proposals, or a combination thereof in content conforming to the purpose of creation. As a result, improvement proposals considering the purpose of creation can be automatically included in the report based on all the aggregation results.

[0110] <Modification Example> In the above-described embodiment, the LLM system 30 is made to output a termination report. However, the termination report may be output not from the LLM system 30 but from the trained model stored in the server 20. The trained model is a model for outputting a termination report based on aggregated data. Specifically, the trained model is obtained by further training a large artificial intelligence model used in the field of natural language processing (NLP). In the further training, so-called fine-tuning is performed to teach the correct answer for each task, so as to specialize in creating a termination report. Specifically, for example, the trained model is trained to output the information contained in the input aggregated data in a manner conforming to the specifications of the termination report. At this time, the training data for training the trained model uses, for example, aggregated data of a plurality of items as input data, and the output form to the termination report based on the input aggregated data as correct answer output data. For example, by inputting the aggregated data and an instruction for outputting the information contained in the aggregated data in a manner conforming to the specifications of the termination report to the trained model, the trained model outputs the information contained in the aggregated data in a manner conforming to the specifications of the termination report.

[0111] Also, context tuning may be performed on the trained model. In context tuning, for example, answer examples, answer formats, etc. are included in the instruction so that information is output from the trained model in the format desired by the user.

[0112] The trained model may be retrained at any time in accordance with changes in the output mode of the termination report, etc.

[0113] Also, in the above embodiment, an example in which an end report including the report contents of a plurality of items is output from the learned model has been described. However, the output from the learned model is not limited to the end report. From the learned model, the report contents related to at least one item that can be included in the end report may be output. For example, the support module 2034 inputs the information aggregated from a predetermined perspective into the learned model. The learned model outputs the report contents related to the item corresponding to the input aggregated information.

[0114] Also, in the above embodiment, the case where the server 20 has the prompt table 2023 has been described as an example. However, the server 20 does not necessarily have to have the prompt table 2023. For example, the server 20 may have a predetermined cooperation module, and the cooperation module may include a prompt template. The cooperation module is, for example, an API (Application Programming Interface). Note that the cooperation module may not be provided in the server 20 and may be provided in a server different from the server 20.

[0115] Specifically, for example, in step S15, the support module 2034 outputs the aggregated data and the creation purpose to the cooperation module. The cooperation module creates an instruction sentence by combining the information it has as a template with the aggregated data and the creation purpose. The cooperation module outputs the aggregated data and the created instruction sentence to the LLM system 30.

[0116] Also, in the above embodiment, the case where the learned model is an LLM has been described as an example. However, the learned model according to this embodiment may be a learned model that is not an LLM.

[0117] In the above-described embodiment, an example of an aspect in which the server 20 includes the aggregation module 2033 and the support module 2034 has been described. However, the present invention is not limited to this aspect, and the functions of the aggregation module 2033 and the support module 2034 may be provided in the terminal device 10. At this time, for example, part or all of the information stored in the storage unit 202 may be stored in the terminal device 10.

[0118] <4 Basic Hardware Configuration of Computer> FIG. 19 is a block diagram showing a basic hardware configuration of a computer 90. The computer 90 includes at least a processor 91, a main memory device 92, an auxiliary storage device 93, and a communication IF 99 (interface). These are electrically connected to each other by a bus.

[0119] The processor 91 is hardware for executing an instruction set described in a program. The processor 91 is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0120] The main memory device 92 is for temporarily storing a program and data processed by the program and the like. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0121] The auxiliary storage device 93 is a storage device for storing data and programs. For example, it is a flash memory, an HDD (Hard Disc Drive), a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like.

[0122] The communication IF 99 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of various mobile communication systems constructed by the Internet, LAN, wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi (registered trademark)) that can be connected to the Internet through a predetermined access point. When connecting wirelessly, communication protocols such as Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. are included. When connecting by wire, the network also includes those directly connected by a USB (Universal Serial Bus) cable, etc.

[0123] Note that all or part of each hardware configuration can be distributed and provided in a plurality of computers 90, and the computers 90 can be virtually realized by connecting them to each other via a network. In this way, the computer 90 is a concept that includes not only a single housing or a computer 90 housed in a case but also a virtualized computer system.

[0124] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of the computer 90 shown in FIG. 19 will be described. The computer includes at least functional units of a control unit, a storage unit, and a communication unit.

[0125] Note that the functional units included in the computer 90 can also be realized by distributing all or part of each functional unit to a plurality of computers 90 interconnected by a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0126] The control unit is realized by the processor 91 reading out various programs stored in the auxiliary storage device 93 and expanding them in the main storage device 92, and executing processing according to the programs. The control unit can realize functional units that perform various information processes according to the types of programs. As a result, the computer is realized as an information processing device that performs information processing.

[0127] The storage unit is realized by the main storage device 92 and the auxiliary storage device 93. The storage unit stores data, various programs, and various databases. Also, the processor 91 can secure a storage area corresponding to the storage unit in the main storage device 92 or the auxiliary storage device 93 according to a program. Further, the control unit can cause the processor 91 to execute processes of adding, updating, and deleting data stored in the storage unit according to various programs.

[0128] The database refers to a relational database and is for managing a set of data called a table, which is structurally defined by rows and columns, in association with each other. In a database, a table is called a table, a column of a table is called a column, and a row of a table is called a record. In a relational database, relationships between tables can be set and associated. Normally, each table is set with a column that serves as a key for uniquely identifying a record, but setting a key for a column is not essential. The control unit can cause the processor 91 to execute adding, deleting, and updating records in a specific table stored in the storage unit according to various programs.

[0129] The communication unit is realized by the communication IF 99. The communication unit realizes the function of communicating with another computer 90 via a network. The communication unit can receive information transmitted from another computer 90 and input it to the control unit. The control unit can cause the processor 91 to execute information processing on the received information according to various programs. Also, the communication unit can transmit information output from the control unit to another computer 90.

[0130] The functions implemented by the components described in this specification may be implemented in circuitry or processing circuitry including a general-purpose processor, a special-purpose processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuitry, and / or combinations thereof, programmed to implement the described functions. The processor includes transistors and other circuitry and is considered to be circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory. In this specification, circuitry, unit, and means are hardware programmed to implement the described functions or hardware that executes the functions. The hardware may be any hardware disclosed in this specification or any hardware known to be programmed or execute to implement the described functions. When the hardware is considered to be a processor of the circuitry type, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or the processor.

[0131] Although some embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are intended to be included in the scope and gist of the invention, as well as in the invention described in the claims and the scope of equivalents thereof.

[0132] <Supplementary Note> The matters described in each of the above embodiments are appended below. (Supplementary Note 1) A program for causing a computer including a processor and a memory to execute. The program causes the processor to perform steps of: obtaining test results; aggregating information included in the test results based on a perspective in which the information can be summarized in a report; inputting the aggregation result based on the perspective into a learned model that has been learned to output information included in the aggregation result in a manner conforming to the specifications of the report, thereby causing the information included in the aggregation result to be output in a manner conforming to the specifications of the report; and presenting the information output from the learned model to a user. (Appendix 2) The learned model is a large language model. In the step of outputting information, by inputting the aggregation result and an instruction text for causing the information included in the aggregation result to be output in a manner conforming to the specifications of the report into the learned model, the learned model outputs the information included in the aggregation result in a manner conforming to the specifications of the report (described in Appendix 1). (Appendix 3) The processor is caused to perform a step of receiving from a user the purpose of creating the report. In the step of outputting information, when the learned model is input with the aggregation result based on the perspective and the purpose of creation, the learned model has been learned to output the information included in the aggregation result in a manner conforming to the specifications of the report and with content along the lines of the purpose of creation. By inputting the aggregation result and the purpose of creation into the learned model, the information included in the aggregation result is output in a manner conforming to the specifications of the report and with content along the lines of the purpose of creation (described in Appendix 1). (Appendix 4) The processor is caused to perform a step of receiving from a user the purpose of creating the report. The learned model is a large language model. In the step of outputting information, by inputting the aggregation result, an instruction text for causing the information included in the aggregation result to be output in a manner conforming to the specifications of the report, and content along the lines of the purpose of creation into the learned model, the learned model outputs the information included in the aggregation result in a manner conforming to the specifications of the report and with content along the lines of the purpose of creation (described in Appendix 1). (Appendix 5) The purpose of creation is a program described in the periodic report, additional proposal, explanatory report, or a combination of at least two of these (Appendix 3) or (Appendix 4). (Appendix 6) In the step of outputting information, by inputting the aggregation result of viewpoints referred to when analyzing the failures occurred in the test results and the instruction text for outputting a report on the viewpoints to be focused on in the analysis based on the information included in the aggregation result into the learned model, from the learned model, based on the information included in the aggregation result, a program described in (Appendix 2) or (Appendix 4) for outputting a report on the viewpoints to be focused on in the analysis is obtained. (Appendix 7) In the step of outputting information, by inputting the aggregation result of viewpoints referred to when analyzing the failures occurred in the test results and the instruction text for outputting a table related to the information included in the aggregation result, a drawing related to the information included in the aggregation result, or a combination of these into the learned model, from the learned model, based on the information included in the aggregation result, a program described in any of (Appendix 2), (Appendix 4), (Appendix 6) for outputting a table related to the information included in the aggregation result, a drawing related to the information included in the aggregation result, or a combination of these is obtained. (Appendix 8) In the step of outputting information, by inputting the aggregation results of a plurality of viewpoints referred to when analyzing the failures occurred in the test results and the instruction text for outputting a summary of the failures into the learned model, from the learned model, based on the information included in the aggregation result, a program described in any of (Appendix 2), (Appendix 4), (Appendix 6), (Appendix 7) for outputting a summary of the failures is obtained. (Appendix 9) In the step of outputting information, by inputting the aggregation result for all viewpoints and the instruction text for outputting the issues of the test, improvement proposals for the test, or a combination of these recognized from the aggregation result into the learned model, from the learned model, based on the information included in the aggregation result, a program described in any of (Appendix 2), (Appendix 6), (Appendix 7), (Appendix 8) for outputting the issues of the test, improvement proposals for the test, or a combination of these is obtained. (Appendix 10) In the step of outputting information, by inputting the aggregation result for all viewpoints and the instructions for outputting, in content conforming to the creation purpose, the issues of the test, the improvement proposals for the test, or a combination of these, recognized from the aggregation result, into the learned model, from the learned model, based on the information included in the aggregation result, output the issues of the test, the improvement proposals for the test, or a combination of these, in content conforming to the creation purpose (Appendix 4) program. (Appendix 11) A method executed on a computer comprising a processor and a memory, wherein the processor executes all the steps executed in the invention according to any one of (Appendix 1) to (Appendix 10). (Appendix 12) An information processing apparatus comprising a processor and a memory, wherein the processor executes all the steps executed in the invention according to any one of (Appendix 1) to (Appendix 10). (Appendix 13) A system comprising means for executing all the steps executed in the invention according to any one of (Appendix 1) to (Appendix 10).

Explanation of Signs

[0133] 1…System 10…Terminal device 120…Communication unit 13…Input device 131…Touch - sensitive device 14…Output device 15…Memory 16…Storage 19…Processor 20…Server 22…Communication IF 23…Input / output IF 25…Memory 26…Storage 29…Processor 30…LLM system

Claims

1. A program for causing a computer having a processor and a memory to execute the program, the program causing the processor to: obtaining test results; aggregating information contained in the test results based on aspects that can be summarized in a report; accepting a purpose for creating the report from a user; acquiring an instruction statement for outputting information included in the tabulation result based on the viewpoint in a manner conforming to the report specifications and with content conforming to the report purpose by reflecting the purpose of creation received from the user in the instruction statement, the instruction statement being a rewritable statement corresponding to the purpose of creation, and a step of inputting the tabulation result based on a viewpoint and the instruction sentence into a large-scale language model as a trained model trained to output information included in the tabulation result in a manner conforming to a report specification when the tabulation result based on the viewpoint is input, thereby outputting information included in the tabulation result from the trained model in a manner conforming to the report specification and with content conforming to the purpose of the report; presenting information output from the trained model to the user; A program that executes the following.

2. A program for execution by a computer having a processor and a memory, the program causing the processor to: obtaining test results; aggregating information contained in the test results based on aspects that can be summarized in a report; accepting a purpose for creating the report from a user; a step of pre-storing an instruction statement for outputting information included in the tabulation result based on the viewpoint in a manner conforming to the specifications of the report for each purpose of creation, and, upon receiving a purpose of creation from the user, reading out an instruction statement corresponding to the purpose of creation received from the user, thereby acquiring an instruction statement for outputting information included in the tabulation result based on the viewpoint in a manner conforming to the specifications of the report and with content conforming to the purpose of creation; a step of inputting the tabulation result based on a viewpoint and the instruction sentence into a large-scale language model as a trained model trained to output information included in the tabulation result in a manner conforming to a report specification when the tabulation result based on the viewpoint is input, thereby outputting information included in the tabulation result from the trained model in a manner conforming to the report specification and with content conforming to the purpose of the report; presenting information output from the trained model to the user; A program that executes the following.

3. 2. The program according to claim 1, wherein the purpose of the creation is a periodic report, an additional proposal, an explanatory report, or a combination of at least two of them.

4. 2. The program of claim 1, wherein in the step of outputting the information, the trained model is input with an instruction for outputting a report on aspects that should be focused on in the analysis based on the information contained in the aggregated results, and the trained model is caused to output a report on aspects that should be focused on in the analysis based on the information contained in the aggregated results.

5. 2. The program of claim 1, wherein in the step of outputting the information, an instruction for outputting, into the trained model, the results of an aggregation of viewpoints to be referenced when analyzing a fault occurring in the test results, and a table relating to the information included in the aggregation results, a drawing relating to the information included in the aggregation results, or a combination thereof, is input to the trained model, thereby causing the trained model to output, based on the information included in the aggregation results, a table relating to the information included in the aggregation results, a drawing relating to the information included in the aggregation results, or a combination thereof.

6. 2. The program of claim 1, wherein in the step of outputting the information, the trained model is input with an aggregated result of multiple perspectives to be referenced when analyzing a fault occurring in the test results and an instruction statement for outputting a summary of the fault, thereby causing the trained model to output a summary of the fault based on the information contained in the aggregated result.

7. The program of claim 1, wherein in the step of outputting the information, the aggregated results for all perspectives and an instruction for outputting test issues, test improvement suggestions, or a combination thereof recognized from the aggregated results are input to the trained model, thereby causing the trained model to output test issues, test improvement suggestions, or a combination thereof based on the information contained in the aggregated results.

8. The program of claim 1, wherein in the step of outputting the information, the aggregated results for all perspectives and an instruction for outputting test tasks, test improvement suggestions, or a combination thereof recognized from the aggregated results, with content in line with the creation purpose, are input to the trained model, thereby causing the trained model to output test tasks, test improvement suggestions, or a combination thereof, with content in line with the creation purpose, based on the information contained in the aggregated results.

9. A method implemented on a computer having a processor and a memory, the processor performing all of the steps performed in any one of claims 1 to 8.

10. 9. An information processing device comprising a processor and a memory, the processor executing all the steps executed in the invention according to any one of claims 1 to 8.

11. A system comprising means for executing all the steps performed in any one of claims 1 to 8.

Images