Information processing device, update method, and program

Abstract

This information processing device updates information stored in an information storage unit, the information being used to generate prompts for instructing the design of application tests. The device includes an update unit that acquires a text output from a language model to which the prompt is input, extracts, from the text, test-related information that is information related to a test for the application, and updates the information stored in the information storage unit by using the test-related information.

Description

Information processing device, update method, and program

[0001] The present invention relates to exploratory testing of applications.

[0002] Exploratory testing is a testing method in which a tester simultaneously "learns and tests" the product and defects being tested, and "plans, designs, and executes the tests." It is generally used for applications with screens, such as web applications and mobile applications.

[0003] Compared to written testing, exploratory testing does not require test design in advance, and therefore does not necessarily require documents such as specifications or design documents. This not only reduces preparation time, but also allows the tester's knowledge to be utilized during testing, making it known to be highly efficient at finding bugs.

[0004] One of the challenges of exploratory testing is that the details of what tests to conduct are left up to the tester, making it highly subjective and difficult to be effective unless the tester is an experienced tester.

[0005] In relation to this problem, Non-Patent Document 1 discloses a method for expanding the variety of searches by recording the tester's operations during testing and information on the application being tested, and highlighting areas on the browser that have not yet been fully explored.

[0006] Bures, M., K. Frajtak, and BS Ahmed. "Tapir: Automation Support of Exploratory Testing Using Model Reconstruction of the System Under Test.", IEEE Transactions on Reliability 67, no. 2 (June 2018): 557-80. https: / / doi.org / 10.1109 / TR.2018.2799957.

[0007] However, the technology disclosed in Non-Patent Document 1 has problems such as "high search diversity does not necessarily mean high exploratory testing effectiveness" and "although it identifies pages or elements on a screen that have not been explored, it does not suggest tests that should be conducted."

[0008] Therefore, even if the technique disclosed in Non-Patent Document 1 is used, the problem of exploratory testing being highly dependent on the individual tester and being difficult to achieve unless the tester is an experienced tester, is not resolved.

[0009] The present invention has been made in consideration of the above points, and aims to provide a technology that enables even an unskilled tester to perform effective exploratory testing.

[0010] According to the disclosed technology, there is provided an information processing device that updates information stored in an information storage unit, which is used to generate a prompt that instructs performing test design for an application, and that includes an update unit that obtains text output from a language model to which the prompt is input, extracts test-related information from the text, which is information related to testing the application, and uses the test-related information to update the information stored in the information storage unit.

[0011] According to the disclosed technology, even if a tester is not an experienced tester, effective exploratory testing can be performed.

[0012] It is a system configuration diagram according to an embodiment of the present invention. It is a flowchart for explaining the operation of the information processing device 100. It is a diagram showing an image of input and output. It is a diagram showing an example of the hardware configuration of the information processing device 100.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0014] (Outline of the embodiment) As described above, exploratory testing has a problem in that it is highly dependent on the individual, and is difficult to demonstrate its effectiveness unless the tester is an experienced tester.

[0015] In this embodiment, in order to solve the above problems, large language models (LLMs) with sufficient versatility and linguistic capabilities are used, and the LLMs are given the necessary inputs in line with the thought patterns of exploratory testing as performed by humans, allowing the LLMs to suggest advanced tests.

[0016] When using LLM for testing, the question arises as to how to provide prompts (instructions).

[0017] It is difficult for a user to include application specifications and the current test status in a prompt each time. Therefore, the technology according to the present embodiment eliminates the effort of creating prompts by automatically generating prompts using information necessary for exploratory testing that has been manually or automatically accumulated.

[0018] Furthermore, to conduct effective exploratory testing, it is necessary to utilize knowledge accumulated during testing, but this knowledge is usually not documented, making it difficult to include in prompts. Therefore, in the technology according to the present embodiment, knowledge is extracted from recorded test logs and tester reports using LLM, and is updated sequentially, so that the accumulated knowledge can be used to propose the next test.

[0019] The technology according to this embodiment will be described in detail below.

[0020] (System Configuration Example) Fig. 1 shows an example of a system configuration according to the present embodiment. As shown in Fig. 1, an information processing device 100 that executes the operations of the technology according to the present invention is provided. A user terminal 10, a test log recording tool 20, and an LLM 30 are connected to the information processing device 100.

[0021] The user terminal 10 is, for example, a PC, a tablet, a smartphone, etc. Note that the user terminal 10 may also be an input / output interface (e.g., a keyboard, a touch panel, etc.) included in the information processing device 100.

[0022] The test log recording tool 20 is provided in a computer external to the information processing device 100. However, the test log recording tool 20 may be provided inside the information processing device 100. Furthermore, the LLM 30 is provided in a computer external to the information processing device 100. However, the LLM 30 may be provided inside the information processing device 100.

[0023] As shown in FIG. 1, the information processing device 100 includes an information storage unit 110, a prompt generation unit 120, and an update unit .

[0024] The information processing device 100 may include the prompt generation unit 120, and the information accumulation unit 110 and the update unit 130 may be provided outside the information processing device 100. Alternatively, the information processing device 100 may include the update unit 130, and the information accumulation unit 110 and the prompt generation unit 120 may be provided outside the information processing device 100.

[0025] The LLM 30 in this embodiment is not limited to a specific LLM, but for example, Open AI's GPT 4 can be used as the LLM 30. Note that the LLM 30 is not limited to what is called a "large-scale language model," and a language model that is not called a "large-scale language model" may be used as the LLM 30. A "large-scale language model" is an example of a "language model."

[0026] (Operation of Information Processing Device 100) During a test, the tester presses a button displayed by the information processing device 100 at a timing desired by the tester, and the information processing device 100 causes the LLM 30 to output a suggestion for the next test to be performed.

[0027] To obtain output (specifically, text) from the LLM 30, instructions called prompts are used. For the LLM 30 to make effective suggestions, the prompts must contain knowledge used in exploratory testing. Here, the knowledge used in exploratory testing refers to five types of information: "specifications, test guidelines, most recent test logs, lessons learned, and past tests" of the application under test, which are provided as text information. This information is stored in the information storage unit 110. Note that the specifications, test guidelines, most recent test logs, lessons learned, and past tests are all examples of test-related information, which is information related to application testing.

[0028] The prompt generation unit 120 reads out the information stored in the information storage unit 110 and automatically generates a prompt. The user can press a button to make an inquiry to the LLM 30 at any time using the prompt generated by the prompt generation unit 120.

[0029] The update unit 130 mechanically analyzes the output results from the LLM 30 to extract past tests and learning items (Insights), and reflects this information (updated information) in the information accumulation unit 110. In other words, the information accumulated in the information accumulation unit 110 is updated with the updated information.

[0030] In this embodiment, it is not necessarily required to use all five types of information for generating prompts: "specifications, test guidelines, most recent test log, learnings, and past tests" of the application under test. For example, it is also possible to use at least one of "specifications, test guidelines, most recent test log, learnings, and past tests." Furthermore, it is also possible to use information that is not included in "specifications, test guidelines, most recent test log, learnings, and past tests" as information for generating prompts.

[0031] The information that the LLM 30 updates may be either a past test or a learning item, or the information that the LLM 30 updates may be information that is different from either a past test or a learning item.

[0032] 2, the processing flow of the information processing device 100 will be described. This processing flow is executed, for example, by a program on the information processing device 100 (computer). Furthermore, this processing flow is premised on the assumption that a user is conducting exploratory testing of the application to be tested based on the output from the LLM 30, and that a test log is being acquired in real time using the test log recording tool 20. The test log is accumulated in the information accumulation unit 110.

[0033] In S1 (step 1), the information processing device 100 detects a user pressing a button, and in S2, the prompt generation unit 120 generates a prompt by applying the five types of information in the information storage unit 110 to a template.

[0034] In S3, the prompt generated by the prompt generating unit 120 is input to the LLM 30, thereby making an inquiry to the LLM 30.

[0035] In S4, the text (answer sentence) is output from the LLM 30 and input to the update unit 130.

[0036] In S5, the update unit 130 analyzes the answer sentences output from the LLM 30, extracts updated past tests and updated learning items, and updates the past tests and learning items stored in the information storage unit 110. Note that the user may update the information in the information storage unit 110 as necessary.

[0037] While the test continues (while the answer is No in S6), the above process is repeatedly executed.

[0038] An image of input and output is shown in Figure 3. As shown in Figure 3, prompts are generated based on the "specifications, test guidelines, most recent test log, learnings, and past tests" of the application under test, and the prompts are input into the LLM, resulting in output from the LLM. The next test design is extracted from the output from the LLM, and the learnings and past tests are updated based on the output from the LLM.

[0039] The information used in the information processing device 100 and the processing contents will be described in more detail below.

[0040] (Information storage unit 110) First, we will explain each of the following information stored in the information storage unit 110: (1) specifications, (2) test guidelines, (3) most recent test log, (4) learning points, and (5) past tests for the application under test.

[0041] (1) Specifications of the application to be tested Since it is difficult to conduct a test without knowing how the application should behave, the specifications of the application to be tested are necessary for conducting the test. In this embodiment, two types of specification information are used: specifications for the entire application and specifications for each screen. The specification information is assumed to be registered in advance in the information accumulation unit 110, but can be edited by the user at any time.

[0042] In this embodiment, in order to shorten the length of the prompt, the specifications for each screen are provided by mechanically extracting only the specifications of the screen visited during the test, which are included in the most recent test log below.

[0043] (2) Test Guidelines In exploratory testing, a method of dividing a test into sessions is often used, in which a test guideline called a charter is sometimes used to conduct an effective test within a limited time. In the technology according to this embodiment, similar to a test using a charter, a rough guideline for the test is given, such as the functions to be tested and the test perspective and type. The test guideline is also assumed to be registered in advance in the information accumulation unit 110, but can be edited by the user at any time.

[0044] (3) Recent Test Log: In exploratory testing, test design is based on observing the application behavior caused by the tester's actions. Therefore, prompts must include information about what the tester did and observed. Therefore, a mechanism is needed to record the tester's actions and intentions, as well as any insights gained during testing.

[0045] Specifically, for example, by using the tool disclosed in the reference document "H. Kirinuki, M. Tajima and T. Haruto, "LatteArt: A Platform for Recording and Analyzing Exploratory Testing," 2023 IEEE Conference on Software Testing, Verification and Validation (ICST), Dublin, Ireland, 2023, pp. 443-453, doi: 10.1109 / ICST57152.2023.00048.", it is possible to automatically or manually record information on test operations and operation targets, transitioned screens, tester observations, and tester intentions for web applications. The above tool can be used as the test log recording tool in this embodiment.

[0046] Since the test log must be in text format, if the recorded information is structured data, each operation etc. is converted into text format.

[0047] The term "most recent" in "most recent test log" refers to operations after the previous LLM inquiry, or the most recent N operations, where N is a natural number designated by the user.

[0048] (4) Learnings A key feature of exploratory testing is that test design utilizes the tester's knowledge and experience. Knowledge about the test subject, in particular, is accumulated during testing, but is typically not documented. Therefore, the technology according to this embodiment not only includes learnings acquired from previous tests in the prompt, but also updates them in the LLM 30 simultaneously with test proposals, thereby accumulating knowledge and experience as text information and utilizing it in proposals for future tests. Learnings can also be edited by the user at any time.

[0049] (5) Past Tests Knowledge of past tests is necessary because if a user does not have knowledge of what tests have been conducted in the past, the LLM 30 may suggest tests that have already been conducted. In the technology according to this embodiment, the LLM 30 updates these, just like learning items. Past tests can also be edited by the user at any time.

[0050] Specific examples of each of the above (1) to (5) are shown below as (a) to (e).

[0051] (a) Specific example of the specifications of the application under test Below is the common specification of the application. A bug tracking system that manages bug tickets. All pages have a common header with the following links. - Main: Press to transition to the "Main screen". - My View: Press to transition to the "My View screen". - Search: Press to transition to the "Ticket search screen". - Register: Press to transition to the "Ticket registration screen".

[0052] (Omitted) Below are the relevant page specifications. # [Registration - MantisBT](http: / / localhost:18080 / bug_report_page.php): Ticket registration screen A screen for registering a new bug ticket. ## Input fields - Category: A drop-down list to select the bug category. Choices include "(Please select)" and "[All projects] General". Required. The default is "(Please select)". - Reproducibility: A drop-down list to select the reproducibility of the bug. Choices include "Always", "Sometimes", "Various", "Untested", "Cannot be reproduced", and "Unknown". The default is "Untested".

[0053] (Omitted) - Additional information: A text area for entering any other supplementary information. The default is empty. - File registration: Select a file such as a bug trail. The default is unselected. - Public: A radio button to determine whether or not to make the ticket public. Choose from "Public" or "Private". The default is "Public". - Next ticket: A checkbox to determine whether or not to register another ticket immediately after registering a ticket. The default is unchecked.

[0054] (Omitted) ## Button - Register: Pressing this button will register a ticket with the information entered in the input fields.

[0055] The above is a concrete example of the specifications of the application under test.

[0056] (b) Specific examples of testing guidelines: Only functional testing should be performed, and non-functional testing should not be performed.

[0057] (c) Specific Example of the Most Recent Test Log A specific example of the most recent test log is as follows. The following test was performed for the purpose of "Select an option other than the default value from the reproducibility drop-down list and verify that the selected value is correctly reflected in the ticket." You clicked A: text="Register" href=" / bug_report_page.php" A screen transition occurs. The new page is [Register - MantisBT](http: / / localhost:18080 / bug_report_page.php). You clicked SELECT: name="category_id" You inputted "1" to SELECT: name="category_id" You clicked SELECT: name="category_id" You clicked INPUT: type="text" name="summary" You inputted "Summary 1" to INPUT: type="text" name="summary" You clicked TEXTAREA: name="description" You inputted "Details 1" to TEXTAREA: name="description" You clicked SELECT: name="reproducibility" You inputted "30" to SELECT: name="reproducibility" You clicked SELECT: name="reproducibility" clicked INPUT: value="Register" type="submit" A screen transition occurs. The new page is [MantisBT](http: / / localhost:18080 / bug_report.php). You clicked A: text="View registered ticket 3" href="view.php?id=3" A screen transition occurs.The new page is [0000003: Summary 1 - MantisBT](http: / / localhost:18080 / view.php?id=3). This is a concrete example from the most recent test log.

[0058] (d) Specific examples of learning points Specific examples of learning points are as follows: - Confirm that a ticket can be registered by filling in the required fields (category, summary, details) when registering a ticket. - Confirm that the transition to the ticket search screen occurs after registering a ticket. - Confirm that even if multiple required fields are left blank at the same time, error messages are only displayed for each field individually. - Confirmed a bug that allowed tickets to be registered without an error even when invalid values ​​were entered for reproducibility, severity, and priority. - Confirmed that tickets could be registered without an error even when special characters were entered in the platform, OS, and version fields. - Discovered a bug in the file attachment function that allowed files larger than the maximum file size of 2000 KB to be attached. - Confirmed that the assignee selection, public / private radio button selection, and next ticket checkbox were functioning. - Confirmed that tickets could be registered without an error even when long text was entered in the reproduction method and additional information text fields.

[0059] The above are specific examples of learning points.

[0060] (e) Past tests Past tests include the following: - Test of the basic flow of selecting a category, entering a summary and details, and registering a ticket. - Test of validation when required fields are not filled in. - Test of system behavior when invalid values ​​are entered for reproducibility, importance, and priority. - Test of whether a ticket can be registered when special characters are entered in the text fields for platform, OS, and version. - Test of system behavior when a file that exceeds the maximum file size that can be attached when registering a file. - Test to confirm the behavior of selecting an assignee, setting the publication status, and selecting to register the next ticket. - Test of entering long text into the text fields for reproduction methods and additional information, and whether that information is reflected correctly after the ticket is registered.

[0061] These are the past tests.

[0062] (Prompt Generator 120) Next, the prompt generation process executed by the prompt generator 120 will be described.

[0063] The prompt assigns the LLM 30 the role of a tester who will conduct exploratory testing and instructs them to design the next test. It also instructs them to update past tests and update what they have learned. By specifying the output format at the end of the prompt, it is possible to consistently obtain the desired results. When outputting, the Chain of Thoughts concept, which has been proven effective in creating prompts, is used to have the LLM analyze the next test before designing the test, increasing the likelihood that an effective test design will be proposed. This corresponds to the "## Analysis for the next test" written before "## Next test case" in the example described below.

[0064] The quality of the output depends on the performance of the LLM, but we have confirmed that by using Open AI's GPT 4, which is the world's best LLM as of February 2024, we can obtain highly accurate and effective output.

[0065] Below is an example of an automatically generated prompt template. The prompt generation unit 120 generates a prompt by providing the five pieces of input information described above in the sections enclosed in <>. In this example, the user can select the output language. If the user selects Japanese, instructions to output in Japanese are added to the prompt. ---Template--- You are an experienced tester and performed exploratory testing against a web application. Please design the next test by using the recent test log, test basis, your insights, and the test guideline. However, do not create the same test as previous tests. Additionally, update past tests and insights with the information from the newly performed tests. ### Test Log ### <Most Recent Test Log> ### Test Basis ### <Specifications of the Application to be Tested> ### Past Tests ### <Past Tests> ### Insights ### <Learning Points> ### Test Guideline ### <Test Guideline> ###### ** Please respond to the instructions in Japanese. However, please leave headings beginning with '##' in English.** Your output should be as follows: ## Test summary {explanation of the test log in one sentence} ## Past tests {update the given past test (bullet list)} ## Insights {update the given insight (bullet list)} ## Analysis for the next test ## Next test case What is verified in this test case: ### Test steps {describe in brief} ### Expected results ## Rationale ----End of template---- Below is an example of the output obtained using LLM30 with prompts generated based on the template above. Here, we show an example using Open AI's GPT-4 as LLM30. Note that Windows 10 and Windows in the examples below are both registered trademarks. --------Example output-------- ## Test summary A test to select a value other than the default value from the reproducibility drop-down list and confirm that the value is reflected in the ticket. ## Past tests - Test the basic flow of selecting a category, entering a summary and details, and registering a ticket. - Test validation when a required field is not filled in. - Testing system behavior when invalid values ​​are entered for reproducibility, importance, and priority. - Testing whether a ticket can be registered by entering special characters in the text fields for platform, OS, and version. - Testing system behavior when selecting a file that exceeds the maximum file size that can be attached when registering a file. - Testing to confirm the behavior of selecting an assignee, setting the publication status, and selecting to register the next ticket. - Testing whether long text is entered into the text fields for reproduction method and additional information, and whether that information is reflected correctly after the ticket is registered. - Testing to select an option other than the default value from the reproducibility pull-down list and verifying that the selected value is reflected correctly in the ticket.## Insights - We confirmed that tickets can be registered by filling in the required fields (category, summary, and details) when registering a ticket. - We confirmed that the user can proceed to the ticket search screen after registering a ticket. - We confirmed that error messages are only displayed for each required field if multiple fields are left blank at the same time. - We confirmed a bug that allowed tickets to be registered without an error even when invalid values ​​were entered for reproducibility, severity, and priority. - We confirmed that tickets can be registered without an error even when special characters are entered in the platform, OS, and version fields. - We discovered a bug in the file attachment function that allowed files larger than the maximum file size of 2000KB to be attached. - We confirmed that the assignee selection, public / private radio button selection, and next ticket checkbox are functional. - We confirmed that tickets can be registered without an error even when long text is entered in the reproduction method and additional information text fields. - We confirmed that tickets can be registered correctly when the reproducibility selection is set to "Sometimes." ## Analysis for the next test In the next test, we will verify the behavior of ticket registration by combining the "Severity" and "Priority" drop-down lists on the ticket registration screen. We also check whether valid values ​​are entered into the "Platform," "OS," and "Version" fields, which have not been verified in previous tests, and whether they are correctly reflected in the ticket. ## Next test case What is verified in this test case: Verify whether the values ​​selected in the severity and priority drop-down lists are reflected in the ticket, and whether the entered values ​​for platform, OS, and version are correctly reflected. ### Test steps 1. Click the "Register" link to go to the ticket registration screen. 2. Select "Category." 3. Enter text in "Summary" and "Details." 4. Select "Minor" from the "Severity" drop-down list. 5. Select "High" from the "Priority" drop-down list. 6. Enter "Windows 10" in "Platform." 7. Enter "Windows" in "OS."8. Enter "1.2.3" in "Version." 9. Click the "Register" button to register the ticket. 10. Confirm that the above input values ​​are correctly reflected on the details screen of the registered ticket. ### Expected results - The ticket is registered without any problems. - The registered ticket's severity is reflected as "Minor" and priority as "High." - "Platform" should be "Windows 10," "OS" should be "Windows," and "Version" should be "1.2.3." ## Rationale: Severity and priority are important information for determining bug fix priorities, so it is essential to verify that they are correctly reflected. Furthermore, platform, OS, and version information are closely related to bug reproducibility. Verifying the system's behavior and accurate reflection of information when entering correct values ​​in these fields can improve system reliability. ----------- Example output ends here-------- (Update Unit 130) Next, we will explain how the update unit 130 updates learnings and past tests.

[0066] The output from the LLM 30 described above shows that the LLM 30 recognizes that a test was conducted in which a value other than the default value was selected from the repeatability pull-down list and that the value was confirmed to be reflected in the ticket, and that the past tests (Past tests) and learnings (Insights) have been updated accordingly.

[0067] The update unit 130 mechanically analyzes the output results to extract past tests and learnings (Insights), and reflects this information in the information accumulation unit 110 .

[0068] Furthermore, in the example of output from LLM30 described above, the next test suggested is to verify whether the values ​​selected in the importance and priority pull-down lists are reflected in the ticket, and whether the input values ​​for platform, OS, and version are correctly reflected.

[0069] (Hardware Configuration Example) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0070] That is, the information processing device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the information processing device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0071] Fig. 4 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 4 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0072] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0073] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0074] (Summary, Effects, etc. of the Embodiment) As described above, the technology described in this embodiment makes it possible to automatically generate prompts that cause the LLM 30 to propose tests in accordance with the knowledge and thought patterns that humans use when conducting exploratory tests. This makes it possible to obtain effective test designs in real time by simply pressing a button during testing.

[0075] By having LLM30 propose tests, even novice testers can conduct effective tests, eliminating the problem of test design depending on the individual in exploratory testing. It is also useful for preventing overlooking tests that should be conducted even by non-novice testers, and for obtaining hints for tests that should be conducted.

[0076] Furthermore, the technology according to this embodiment can automatically generate prompts using five types of information that are manually or automatically accumulated, thereby eliminating the need to create prompts.

[0077] Furthermore, with the technology according to this embodiment, knowledge can be extracted using LLM30 from the recorded test logs and tester reports, and the knowledge can be updated sequentially, so that the accumulated knowledge can be used to propose the next test.

[0078] Regarding the above embodiment, the following Supplementary Notes 1 and 2 are further disclosed.

[0079] <Supplementary Note 1> (Supplementary Item 1) An information processing device that generates a prompt for causing a language model to design a test for an application, comprising: a prompt generation unit that reads test-related information from an information storage unit that stores test-related information, the test-related information being information related to testing of the application, and generates the prompt using the test-related information. (Supplementary Item 2) The information processing device according to Supplementary Item 1, wherein the test-related information includes at least one of specifications of the application, test guidelines for the application, the most recent test log for the application, lessons learned related to testing the application, and past tests for the application. (Supplementary Item 3) The information processing device according to Supplementary Item 1 or 2, wherein the prompt generation unit generates the prompt by applying the test-related information to a template. (Supplementary Item 4) The information processing device according to any one of Supplementary Items 1 to 3, wherein the prompt includes an instruction to analyze the next test and an instruction to design the next test that follows the instruction. (Supplementary Item 5) The information processing device of any one of Supplementary Items 1 to 4, wherein the prompt includes learnings related to testing the application, past tests for the application, an instruction to update the learnings, and an instruction to update the past tests. (Supplementary Item 6) A prompt generation method executed by an information processing device that generates a prompt for causing a language model to perform test design for an application, the prompt generation method comprising the steps of reading test-related information, which is information related to testing the application, from an information storage unit that stores the test-related information, and generating the prompt using the test-related information. (Supplementary Item 7) A non-transitory storage medium that stores a program for causing a computer to function as the prompt generation unit in the information processing device of any one of Supplementary Items 1 to 5.

[0080] <Supplementary Note 2> (Supplementary Item 1) An information processing device that updates information stored in an information storage unit used to generate a prompt that instructs performing test design for an application, comprising: an update unit that acquires text output from a language model to which the prompt is input, extracts test-related information related to testing of the application from the text, and updates the information stored in the information storage unit using the test-related information. (Supplementary Item 2) The information processing device according to Supplementary Item 1, wherein the prompt includes an instruction to perform the test design, the test-related information before the update, and an instruction to update the test-related information, and the text includes the test-related information after the update. (Supplementary Item 3) The information processing device according to Supplementary Item 1 or 2, wherein the test-related information is at least one of learnings related to testing the application and past tests for the application. (Supplementary Item 4) The information processing device according to any one of Supplementary Items 1 to 3, wherein the prompt includes, in addition to the test-related information, at least one of specifications of the application, test guidelines for the application, and the most recent test log for the application. (Supplementary Item 5) An updating method executed by an information processing device for updating information stored in an information storage unit used to generate a prompt that instructs performing test design for an application, the updating method comprising the steps of: acquiring text output from a language model to which the prompt is input, extracting test-related information related to testing of the application from the text, and updating the information stored in the information storage unit using the test-related information. (Supplementary Item 6) A non-transitory storage medium storing a program for causing a computer to function as an updating unit in the information processing device according to any one of Supplementary Items 1 to 4.

[0081] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0082] 10 User terminal 20 Test log recording tool 30 LLM 100 Information processing device 110 Information storage unit 120 Prompt generation unit 130 Update unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that updates information stored in an information storage unit and is used to generate a prompt that instructs test design for an application, comprising: an update unit that obtains text output from a language model to which the prompt is input, extracts test-related information from the text, which is information related to testing the application, and uses the test-related information to update the information stored in the information storage unit.

2. The information processing device of claim 1, wherein the prompt includes an instruction to perform the test design, the test-related information before the update, and an instruction to update the test-related information, and the text includes the test-related information after the update.

3. The information processing device according to claim 1, wherein the test-related information is at least one of learnings related to testing the application and past tests on the application.

4. The information processing device according to claim 1, wherein the prompt includes, in addition to the test-related information, at least one of a specification of the application, a test guideline for the application, and a most recent test log for the application.

5. An updating method executed by an information processing device for updating information stored in an information storage unit, which is used to generate a prompt that instructs performing test design for an application, comprising the steps of: acquiring text output from a language model to which the prompt is input; extracting test-related information from the text, which is information related to testing the application; and using the test-related information to update the information stored in the information storage unit.

6. A program for causing a computer to function as an update unit in the information processing device according to any one of claims 1 to 4.

Images