Information processing methods, computer programs, and information processing devices.

Abstract

The present invention provides an information processing method, a computer program, and an information processing device that are expected to support the implementation of software testing. [Solution] The information processing method according to this embodiment involves an information processing device acquiring screen definition information that defines the configuration of the operation screen displayed by the software under test, acquiring image data that captures the display content of the operation screen, acquiring test content information relating to the content of the test to be performed on the software under test, and generating identification information for identifying the operation elements provided on the operation screen using a language model that accepts text and image input, based on the acquired screen definition information, image data and test content information.

Description

Technical Field

[0001] The present disclosure relates to an information processing method, a computer program, and an information processing apparatus.

Background Art

[0002] In Patent Document 1, a software automatic test method has been proposed in which software design information based on design review or the like is set in a storage device, and a software test based on the set software design information is automatically performed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides an information processing method, a computer program, and an information processing apparatus that can be expected to assist in performing tests on software.

Means for Solving the Problems

[0005] An information processing method according to an embodiment includes an information processing apparatus obtaining screen definition information defining a configuration of an operation screen displayed by software to be tested, obtaining image data capturing display contents of the operation screen, obtaining test content information related to the content of a test to be performed on the software to be tested, and generating specific information for specifying operation elements provided on the operation screen using a language model that accepts input of text and images based on the obtained screen definition information, the image data, and the test content information.

Effects of the Invention

[0006] This disclosure is expected to support the implementation of software testing. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram illustrating the overview of the information processing system according to this embodiment. [Figure 2] This is a block diagram showing one example configuration of the information processing device according to this embodiment. [Figure 3] This is a schematic diagram illustrating the overview of the selector generation process performed by the information processing device according to this embodiment. [Figure 4] This is a schematic diagram showing an example of action definition information. [Figure 5] This flowchart shows an example of the processing procedure performed by the information processing device according to this embodiment as a selector generation tool. [Figure 6] This flowchart shows an example of the processing procedure performed by the information processing device according to this embodiment as a selector generation tool. [Figure 7] This flowchart shows an example of the procedure for searching for an operation element performed by the information processing device according to this embodiment. [Figure 8] This flowchart shows an example of the procedure for searching for an operation element performed by the information processing device according to this embodiment. [Figure 9] This flowchart shows an example of the processing procedure performed by the information processing device according to this embodiment as a test tool. [Modes for carrying out the invention]

[0008] Specific examples of information processing systems according to the embodiments of this disclosure will be described below with reference to the drawings. However, this disclosure is not limited to these examples, and all modifications within the meaning and scope of the claims are intended to be included.

[0009] <System Overview> Figure 1 is a schematic diagram illustrating the overview of the information processing system according to this embodiment. The information processing system according to this embodiment is a system that supports the testing of software 100. The software 100 to be tested in the information processing system according to this embodiment is software that displays a screen and accepts user input via a browser program executed on a user's personal computer or terminal device such as a smartphone, such as an online shopping site. However, the software 100 is not limited to software that displays a screen and accepts input via a browser program; it can be any software that displays a screen with operation elements such as buttons or text boxes and accepts input via these operation elements.

[0010] The information processing system according to this embodiment includes a test tool (test device) 101 that automatically performs tests on the software 100, and a selector generation tool 102 that generates selectors necessary for the test tool 101 to perform the tests. The test tool 101 receives test step information created in advance by a designer or the like as input, and performs tests on the software 100 according to the procedure set in the test step information. The test step information may include information that, for example, when performing a test step such as "click the login button," the action performed by the test tool 101 is "click" and the target is the "login button." If, for example, there are multiple buttons on the screen displayed by the software 100, the test tool 101 needs to identify which button is the login button in order to perform this test step.

[0011] A selector is information that allows the test tool 101 to identify which of the one or more operation elements provided on the operation screen displayed by the software 100 is the operation element to be tested. Conventionally, the designer of the software 100 would create selectors in advance along with test step information and provide them to the test tool 101. In the information processing system according to this embodiment, when the test tool 101, having been given test step information, determines that a selector is necessary to perform the test step, it outputs a selector request to the selector generation tool 102 requesting the generation of a selector. In response to the selector request from the test tool 101, the selector generation tool 102 uses a language model such as an LLM (Large Language Model) or VLM (Vision Language Model) that has been pre-machine-trained to identify the operation element to be tested, and generates a selector based on the identification result and outputs it to the test tool 101. The test tool 101 can obtain the selector generated by the selector generation tool 102 and perform the test step targeting the operation element identified by the obtained selector.

[0012] In this embodiment, the information processing system utilizes a trained language model, or so-called AI (Artificial Intelligence), to automatically generate selectors necessary for conducting tests using a selector generation tool 102. This eliminates the need for designers to create selectors themselves, which is expected to facilitate the testing of the software 100.

[0013] In this embodiment, the test tool 101 and the selector generation tool 102 are software tools implemented by an information processing device such as a personal computer or server computer executing the computer program according to this embodiment. In this embodiment, the selector generation process is described assuming that the test tool 101 and the selector generation tool 102 are executed on a single information processing device, but this is not the only way, and the test tool 101 and the selector generation tool 102 may be executed on separate information processing devices.

[0014] <Device configuration> Figure 2 is a block diagram showing an example configuration of an information processing device according to this embodiment. The information processing device 1 according to this embodiment can be realized by installing a predetermined application program on a general-purpose information processing device such as a personal computer or a server computer. The information processing device 1 according to this embodiment is configured to include a processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, and an operation unit 15, etc. In this embodiment, the explanation is given assuming that processing is performed by a single information processing device 1, but the processing of the information processing device 1 may be distributed among multiple devices.

[0015] The processing unit 11 is composed of an arithmetic processing unit such as a CPU (Central Processing Unit), MPU (Micro-Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Network Processing Unit), or quantum processor, and a storage device such as ROM (Read Only Memory) and RAM (Random Access Memory). The processing unit 11 reads and executes the program 12a stored in the storage unit 12, and performs various processes such as processing as a test tool 101 that performs tests on the software 100 based on test step information, and processing as a selector generation tool 102 that generates selectors necessary for performing tests.

[0016] The storage unit 12 is configured using a large-capacity storage device such as a hard disk or an SSD (Solid State Drive). The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In the present embodiment, the storage unit 12 stores a program 12a executed by the processing unit 11. The storage unit 12 is also provided with a model information storage unit 12b that stores information regarding a learning model that has been machine-learned in advance, and a history storage unit 12c that stores history information regarding the generated selector.

[0017] In the present embodiment, the program (computer program, program product) 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the information processing apparatus 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may be written in the storage unit 12, for example, at the manufacturing stage of the information processing apparatus 1. Also, for example, the program 12a may be acquired by the information processing apparatus 1 through communication of what is distributed by a remote server device or the like. For example, the program 12a may be read by a writing device from what is recorded on the recording medium 99 and written in the storage unit 12 of the information processing apparatus 1. The program 12a may be provided in a form of distribution via a network, or may be provided in a form recorded on the recording medium 99.

[0018] The model information storage unit 12b stores information about a learning model that has been machine-learned in advance. The information about the learning model may include, for example, information indicating the configuration of the learning model and information such as the values of predetermined internal parameters. In the present embodiment, the model information storage unit 12b stores information about the language model used in the processing of the selector generation tool 102. The language model may employ a learning model such as Transformer equipped with an attention mechanism in a large-scale neural network, BERT (Bidirectional Encoder Representations from Transformers), or GPT (registered trademark) (Generative Pre-trained Transformer). Also, in the present embodiment, a language model such as a VLM that can accept image data as input is adopted as the language model. The language model used by the information processing system according to the present embodiment may be a widely publicized general-purpose language model, and information regarding software testing may be learned by fine-tuning this general-purpose language model.

[0019] Note that in the present embodiment, a configuration in which the information processing device 1 includes a language model is adopted, but the present invention is not limited thereto, and for example, a configuration in which a language model is operated in an external server device may be adopted. In this case, a configuration may be adopted in which the information processing device 1 transmits input information to the language model to an external server device, the server device inputs the information from the information processing device 1 to the language model, transmits the output information of the language model to the information processing device 1, and the information processing device 1 acquires the output information of the language model from the server device.

[0020] The history storage unit 12c stores information about selectors generated by the selector generation tool 102. The history storage unit 12c stores information such as the test actions performed by the test tool 101 on the generated selector, and success / failure information indicating whether the test tool 101 was able to identify the operation element using the selected selector. If the selector generation request given by the test tool 101 relates to an operation element that can be identified by a previously generated selector, the selector generation tool 102 can read this selector from the history storage unit 12c and provide it to the test tool 101. Furthermore, by storing failure examples in the history storage unit 12c where the operation element could not be identified for previously generated selectors, the selector generation tool 102 can refer to these failure examples when generating new selectors, and an improvement in the accuracy of selector generation can be expected.

[0021] The communication unit 13 transmits and receives data with other devices via a network such as the Internet or a mobile phone communication network. In this embodiment, the communication unit 13 can communicate with other information processing devices used by the user for software development, server devices on which a language model operates, or other information processing devices on which software under test operates.

[0022] The display unit 14 is configured using a liquid crystal display or the like, and displays various images and characters based on the processing of the processing unit 11. In this embodiment, the display unit 14 displays various information, such as the display of information related to the test results of the software 100 by the test tool 101, and the display of information related to selectors generated by the selector generation tool 102.

[0023] The operation unit 15 receives user input and notifies the processing unit 11 of the received input. For example, the operation unit 15 receives user input via a mechanical button or an input device such as a touch panel provided on the surface of the display unit 14. Alternatively, the operation unit 15 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 1.

[0024] The memory unit 12 may be an external storage device connected to the information processing device 1. The information processing device 1 may be a multicomputer comprising multiple computers, or it may be a virtual machine virtually constructed by software. Furthermore, the information processing device 1 is not limited to the above configuration, and for example, it may not have a display unit 14 and an operation unit 15. If the display unit 14 and operation unit 15 are not provided, the information processing device 1 may, for example, use the display unit and operation unit of a terminal device used by the user to display information and receive user operations.

[0025] Furthermore, in the information processing device 1 according to this embodiment, the processing unit 11 reads and executes the program 12a stored in the storage unit 12, thereby realizing the data acquisition unit 11a, data simplification unit 11b, selector generation unit 11c, selector verification unit 11d, operation element search unit 11e, and test processing unit 11f, etc., as software functional units in the processing unit 11. The data acquisition unit 11a, data simplification unit 11b, selector generation unit 11c, selector verification unit 11d, and operation element search unit 11e correspond to the selector generation tool 102 described above, and the test processing unit 11f corresponds to the test tool 101 described above. In this figure, the functional units of the processing unit 11 that are related to supporting software testing are shown, and functional units related to other processing are omitted from the illustration.

[0026] The data acquisition unit 11a performs the process of acquiring various data necessary for generating selectors. In this embodiment, the data necessary for generating selectors includes HTML data (screen definition information) that defines the configuration of the operation screen displayed by the software 100 under test, screenshot image data captured from the operation screen displayed by the software 100, test step information (test content information) indicating the content of the tests performed by the test tool 101 on the software 100, and history information such as failure examples stored in the history storage unit 12c. The data acquisition unit 11a acquires the HTML data, screenshot image data, and test step information provided by the test tool 101 along with the selector generation request, and also acquires the history information from the history storage unit 12c.

[0027] The data simplification unit 11b performs a process to simplify the data by removing information unnecessary for selector generation from the HTML data acquired by the data acquisition unit 11a. For example, the data simplification unit 11b parses the document structure of the acquired HTML data as a DOM (Document Object Model) tree and simplifies the HTML data by removing predetermined tags, attributes, comment nodes, and blank text nodes contained in the HTML data.

[0028] The selector generation unit 11c generates selectors using a language model based on the HTML data simplified by the data simplification unit 11b, as well as screenshot image data, test step information, and history information acquired by the data acquisition unit 11a. However, in this embodiment, before generating selectors using the language model, the selector generation unit 11c determines whether the generated selectors can be reused. For selectors requested to be generated by the test tool 101, the selector generation unit 11c searches for reusable selectors among the selectors stored as history information in the history storage unit 12c, based on the test step information and other information provided along with the generation request. For example, when the selector generation unit 11c generates a selector to identify the login button targeted by an action in the test step information, it adds information indicating that this selector targets the login button and stores the generated selector in the history storage unit 12c. If the test tool 101 then requests the generation of a selector targeting the login button, the selector generation unit 11c searches the history storage unit 12c for a selector associated with the login button. If a suitable selector is stored, the unit reads this selector from the history storage unit 12c and outputs it to the test tool 101.

[0029] If no reusable selectors are stored in the history storage unit 12c, the selector generation unit 11c generates a selector using the language model. The selector generation unit 11c inputs simplified HTML data, screenshot image data, test step information, and history information, along with a prompt instructing the language model to generate a selector based on this information, and obtains the information output by the language model in response to this input. In this embodiment, the language model generates and outputs information including a selector for identifying an operation element, along with the action that the test tool 101 will perform on this selector.

[0030] The selector verification unit 11d performs a process to verify the validity of the selector generated by the selector generation unit 11c. In this embodiment, the selector verification unit 11d verifies the validity of the selector in two ways. The first verification method is a verification method that can be processed within the selector generation tool 102. The selector verification unit 11d determines whether the selector output by the language model can identify a unique operation element within the HTML data. The selector verification unit 11d searches the HTML data using the selector output by the language model and determines that the selector is valid if only one matching operation element is identified. If the language model generates a valid selector, the selector verification unit 11d adopts this selector and outputs it to the test tool 101.

[0031] In response, the selector verification unit 11d determines that the selector is invalid if there are zero matching operation elements (i.e., no operation elements are found) or if there are multiple matching operation elements (i.e., it is not possible to narrow down the operation elements to one), and instructs the language model to regenerate the selector, taking this determination result into account. For example, if there are multiple matching operation elements, the selector verification unit 11d inputs a prompt to the language model that includes an instruction to narrow down the generation conditions in order to regenerate a more specific selector, and repeats this until the operation elements identified by the selector output by the language model are narrowed down to one. Also, for example, if there are zero matching operation elements, the selector verification unit 11d inputs a prompt to the language model that includes an instruction to loosen the generation conditions in order to search a wider range, and attempts to regenerate the selector in a wider range, repeating this until the language model generates a selector that can identify at least one operation element. When the selector verification unit 11d obtains a selector that identifies one operation element from the language model, it outputs this selector to the test tool 101.

[0032] The second verification method involves verifying the validity of selectors not only by the selector generation tool 102, but also through the collaboration of the selector generation tool 102 and the test tool 101. Selectors deemed valid by the first verification method of the selector verification unit 11d are provided to the test tool 101, which then performs test steps based on these selectors. If the test tool 101 is unable to identify an operation element from the operation screen of the software 100 based on the provided selector, the test tool 101 notifies the selector generation tool 102 that it has failed to identify the operation element. When the selector verification unit 11d receives notification of identification failure from the test tool 101, it adds flag information indicating the identification failure to the selector output to the test tool 101 and stores it in the history storage unit 12c as a failure case. Based on this failure case, the selector verification unit 11d causes the selector generation unit 11c to regenerate selectors using the language model, provides the selector deemed valid by the first verification method to the test tool 101, and has it re-execute the test steps. The selector verification unit 11d repeats the above procedure until the test tool 101 successfully identifies the operation element by the selector.

[0033] The test tool 101 may be configured to notify the selector generation tool 102 if it successfully identifies an operation element based on a given selector, or it may be configured not to notify. If the test tool 101 is configured to notify success, the selector verification unit 11d adds flag information indicating successful identification to the selector in response to the success notification and stores it in the history storage unit 12c as a success case. If the test tool 101 is configured not to notify success, the selector verification unit 11d stores the selector in the history storage unit 12c as a success case if it does not receive a notification of failure from the selector generation tool 102 within a predetermined period of time.

[0034] The information processing device 1 according to this embodiment is expected to ensure the accuracy of the selectors generated by the language model by performing a double verification: a first verification performed internally by the selector generation tool 102 and a second verification performed using the test tool 101.

[0035] If the selector verification unit 11d fails to generate a selector that it deems valid, the operation element search unit 11e works in cooperation with the test tool 101 to search for an operation element from the operation screen. Possible reasons why an operation element cannot be identified include, for example, that the entire operation screen is not displayed and the operation element is located outside the displayed area, or that it is located on a different page. In such cases, the operation element search unit 11e requests the test tool 101 to perform an action such as scrolling the operation screen or transitioning to the next page, and obtains image data of a screenshot of the operation screen after the action as a result of the execution. Based on the image data obtained from the test tool 101, the operation element search unit 11e causes the selector generation unit 11c to generate a selector and the selector verification unit 11d to verify the selector.

[0036] Furthermore, if the operation element search unit 11e cannot identify an operation element even after performing the above actions such as scrolling and page transitions, it searches for selectors related to operation elements similar to the operation element being identified from among the successful selector cases stored in the history storage unit 12c. The operation element search unit 11e searches for similar operation elements from the history storage unit 12c by converting information such as the name of the operation element into a feature vector, calculating the cosine similarity of the feature vector, and obtaining successful selector cases that identified similar operation elements. The operation element search unit 11e provides the test tool 101 with the selector of the similar operation element obtained from the history storage unit 12c, allowing the test tool 101 to identify the operation element using this selector. The operation element search unit 11e attempts to search for similar operation elements a predetermined number of times, and if the test tool 101 still cannot identify the operation element, it displays an error message indicating that the selector generation failed.

[0037] The test processing unit 11f performs tests on the software 100 as a test tool 101. The test processing unit 11f receives HTML data defining the configuration of the operation screen, test step information indicating the content of the tests to be performed on the software 100, and test target information regarding the configuration of the software 100 to be tested. Based on this input information, it performs tests on the software 100. For example, the test step information contains information on multiple test steps to be performed on the software 100 in the order they will be performed. The test processing unit 11f performs various actions on the software 100 according to the test step information, obtains the results of those actions, and determines whether they are correct or incorrect. The test processing unit 11f displays information such as the progress of the tests being performed on the software 100 and the test results on the display unit 14.

[0038] In this embodiment, the test processing unit 11f requests the selector generation tool 102 to generate a selector when an operation element is specified as the target in the test step to be performed. At this time, the test processing unit 11f provides the selector generation tool 102 with HTML data, information on the test step (only for the test step to be performed), and image data of a screenshot of the operation screen displayed at that time by the software 100 under test. However, if there are no changes to the HTML data and image data from what was previously provided, the test processing unit 11f may omit providing this information to the selector generation tool 102.

[0039] Furthermore, the test processing unit 11f performs a test step using the selector generated by the selector generation tool 102, and if it cannot identify the operation element, it notifies the selector generation tool 102 that it failed to identify the operation element. At this time, the test processing unit 11f may also send information to the selector generation tool 102 regarding the circumstances under which the operation element could not be identified.

[0040] Furthermore, when the test processing unit 11f is requested to perform an action such as scrolling or page transition by the operation element search unit 11e, it performs the requested action, captures the operation screen displayed by the software 100 to obtain screenshot image data, and sends the obtained image data to the selector generation tool 102.

[0041] <Selector generation process> Figure 3 is a schematic diagram illustrating the overview of the selector generation process performed by the information processing device 1 according to this embodiment. In the information processing system according to this embodiment, for example, a designer or the like considers procedures for conducting various tests to confirm the functions and operation of software 100 under development, and pre-generates test step information for a test tool 101 to perform the tests according to these procedures. The test step information may be information generated by an automatic generation tool based on information such as test procedures generated by the designer or the like, or it may be information directly written by the designer or the like.

[0042] The designers of software 100 provide the test tool 101 with this test step information, along with HTML data showing the configuration of the user interface displayed by software 100, and have the test tool 101 perform tests of software 100 based on the test step information. In addition to the test step information and HTML data, the test tool 101 may also receive and process various other information from the designers, such as the source code or binary code of software 100.

[0043] The following is an example of test step information. Testing software 100 is represented as a sequence of steps, and the test step information contains information about multiple steps. The example below shows one step extracted from the multiple steps included in the test step information. This example is a step in a procedure to test the behavior of a user who uses a service provided by software 100 when logging into this service, and it tests whether the result is obtained when the user clicks the login button on the operation screen and is redirected to the dashboard.

[0044] { "step": { "label": "Login button", "action": "click", "value": "", "explain": "Click the login button to proceed to the dashboard", "target_item": { "name": "Login button", "description": "The blue login button in the upper right corner of the screen" } } }

[0045] The test tool 101, having received input information such as test step information and HTML data from the designer, performs tests on the software 100. For example, if the software 100 is configured to exchange information via a web browser, the test tool 101 launches the software 100 by starting the web browser and accessing a predetermined address. Note that the HTML data may be obtained by the test tool 101 from the software 100 via the web browser, rather than being input by the designer. The test tool 101 performs tests on the software 100 by sequentially executing multiple test steps set in the test step information and determining whether the expected results are obtained.

[0046] For example, when the test tool 101 is given the above test step, it performs a click operation, which is set as the action, on the login button, which is set as the operation element of the target item (target_item). Here, in order to perform this test step, the test tool 101 needs to identify which operation element on the operation screen displayed by the software 100 the login button, which is the target operation element, is, and obtains a selector, which is information for identifying the operation element. Conventionally, selectors for identifying operation elements were created in advance by designers, etc., but in the information processing system according to this embodiment, the selector generation tool 102 automatically generates the selector.

[0047] When the test tool 101 requires a selector to identify an operation element, it sends a selector request to the selector generation tool 102 requesting the generation of this selector. The selector request sent from the test tool 101 to the selector generation tool 102 includes information necessary for generating the selector, such as information on the test step to be performed, HTML data showing the configuration of the operation screen, and image data of a screenshot of the operation screen displayed by the software 100. The image data of the screenshot of the operation screen can be acquired by the test tool 101 at an appropriate time, for example, after the completion of the previous test step.

[0048] When the selector generation tool 102 receives a selector request from the test tool 101, it starts the process of generating selectors using a language model. In this embodiment, before generating selectors, the selector generation tool 102 analyzes the structure of the HTML data provided by the test tool 101 as a DOM tree and simplifies the data by removing unnecessary information from the HTML data. For example, the selector generation tool 102 removes information about tags in the HTML data that are not directly related to selector generation, such as "script", "style", "img", and "svg". Also, for example, the selector generation tool 102 removes information such as event handlers such as "onclick", custom attributes such as "data-*", or accessibility attributes such as "aria-*". Furthermore, for example, the selector generation tool 102 removes comment nodes or blank text nodes in the HTML data.

[0049] The selector generation tool 102 also determines whether a previously generated selector can be reused for a selector requested by the test tool 101. In this embodiment, the selector generated by the selector generation tool 102 is stored in the history storage unit 12c along with, for example, the test step information on which it was generated, and flag information indicating whether the test tool 101 succeeded in identifying the operation element using this selector. The selector generation tool 102 searches the history storage unit 12c for a match or similar selector from the test step information stored together with the selector, based on the test step information provided by the test tool 101, and determines that the selector stored in the history storage unit 12c is reusable if the flag information of the selector associated with the corresponding test step information is successful. If a reusable selector exists, the selector generation tool 102 retrieves this selector from the history storage unit 12c and outputs the retrieved selector to the test tool 101 as the generation result for the selector request. If no reusable selector exists, the selector generation tool 102 generates a selector using a language model.

[0050] The selector generation tool 102 generates the selector requested by the test tool 101 by inputting the following into the language model: test step information and screenshot image data provided by the test tool 101, simplified HTML data, action definition information defining the actions that the test tool 101 can perform, history information on selectors previously generated by the selector generation tool 102, and prompts instructing the generation of selectors based on this information. An example of the information and prompts to be input into the language model is shown below.

[0051] (Example of input information) { "content":"...", "image":"data:image / png;base64…", "step":{ "label": "Login button", "action": "click", "value": "", "explain": "Click the login button to proceed to the dashboard." }, "actions":[ ... ... ], "history":[ ... ] }

[0052] (Example prompt) " Analyze the provided HTML data, screenshots, and past operation history to generate an XPath selector that uniquely identifies the target element specified in the test step information. Please strictly adhere to the following rules during generation: 1. Utilize history: Avoid patterns that have caused errors in the past, and consider alternative operations such as scrolling as needed. 2. Element Identification: Prioritize ID, Name, Text Content, and Class in that order, and create unique selectors while avoiding dynamic attributes. 3. Operability: Exclude elements that cannot be manipulated, such as those with the display:none or disabled attribute. 4. Output format: Output in JSON format, including the specified action, XPath selector, and next action. "

[0053] The input information above is partially omitted, but "content" will contain simplified HTML data. "image" will contain the file name of the screenshot image data, etc. "step" will contain test step information provided by test tool 101. "actions" will contain action definition information regarding the actions that test tool 101 can perform. The action definition information is created in advance by the designer or the like and stored in the storage unit 12 of the information processing device 1. In this example, it is assumed that the language model does not know about the actions that test tool 101 can perform, but this is not the only case. If the language model has already learned about the actions that test tool 101 can perform, it is not necessary to input action definition information into the language model.

[0054] Furthermore, the "history" field contains information about past selectors stored in the history memory unit 12c that the test tool 101 failed to identify the operation elements for. By providing the language model with these failure examples in advance, the language model can avoid these failures and generate new selectors. The information provided to the language model as failure examples may include, for example, the generated selector and information about the results of the verification process performed on this selector. The information provided as a result of the verification process may include, for example, whether multiple operation elements were identified by the generated selector, or whether the operation elements could not be identified by the generated selector. By providing the language model with this history information of failure examples, it is expected that the language model will be able to determine whether to make the selector "more specific" or "more abstract" and generate the selector accordingly.

[0055] Figure 4 is a schematic diagram showing an example of action definition information. Action definition information includes "action," "description," and "parameter" information. In this example, the action definition information defines actions that test tool 101 can perform, such as "click," "double-click," "change input value," "file upload," "mouseover," "URL transition," and "scroll." Although the action definition information shown in the figure is presented in a table format, the action definition information input to the language model is structured text data. An example of structured action definition information is shown below. However, the following action definition information is a structured version of some of the information in the table shown in Figure 4. By inputting such action definition information to the language model, the language model is constrained to output only the defined actions; therefore, the action definition information functions as a constraint to limit the output of the language model.

[0056] { "actions":[ { "action":"click", "description": "Click on an element in the browser." "parameters":[ { "name":"target", "type":"test_target", "description": "Select the ID of the target element from the list of testTargets." }, { "name":"button", "type":"fixed_value", "description": "Specifies the mouse button to click." } ] }, { "action":"scroll", "description": "Scroll down the page." "parameters":[ { ... }, ... }

[0057] The selector generation tool 102 inputs the above information and a prompt instructing the language model to generate selectors, causing the language model to generate and output one or more selectors corresponding to the input information. In this embodiment, it is assumed that the language model may not be able to generate selectors based on the input information, and in this case, the language model outputs error information indicating that it cannot generate selectors. The selector generation tool 102 also instructs the language model in the prompt to generate output information that associates the generated selectors with the actions that can be performed on these selectors. An example of a prompt to be input to the language model is shown below.

[0058] (Example prompt) Analyze the input HTML data, screenshot images, and past validation history (identification results of manipulated elements and patterns of failed selectors) to generate an XPath selector to identify the 'target element' indicated in the test step information. When generating the selector, configure it to resolve the previous validation status (element not found or match to multiple elements) while following the following priority order. 1. Unique ID attribute (excluding dynamic ones) 2.Name attribute 3. Text content (normalize whitespace for detection) 4. Combinations of Class attribute and tags In particular, if multiple elements were identified in the previous trial, refine the conditions by adding information and attributes of the parent element. If no elements are found, try a different specification method that avoids past error patterns. The output should be structured data (in JSON format) including the specified action name, the identified selector, and whether processing can continue (whether scrolling, etc., is necessary).

[0059] The language model generates selectors to identify the operational elements based on the input information and prompts described above, and outputs these selectors as structured data, such as in JSON format. An example of the information output by the language model is shown below.

[0060] { "action":"click", "selector":" / / button[@id='login']" }

[0061] The output information above indicates that " / / button[@id='login']" is the selector used to identify the login button, and that a click action is associated with this selector. Based on this information, test tool 101 can perform the test action set in the test step information by clicking on the operation element identified by the selector.

[0062] The selector generation tool 102 generates selectors using the language model and then verifies the validity of the generated selectors. The selector generation tool 102 searches the HTML data using the selectors output by the language model and determines that the selector is valid if only one matching operation element is identified. On the other hand, if there are multiple matching operation elements, the selector generation tool 102 determines that the generated selector is not valid and instructs the language model to regenerate the selector. In this case, the selector generation tool 102 gives the language model a prompt instructing it to regenerate a more specific selector. An example of a prompt to be input to the language model is shown below.

[0063] (Example prompt) "The previously generated selector matched multiple elements in the current screen definition information. To uniquely identify the target operation element, please regenerate the selector with more specific conditions. Specifically, narrow down the match to only one element by combining the parent element's tag information, surrounding text information, or specific attribute values."

[0064] Furthermore, if there are zero operation elements corresponding to the selector generated by the language model, the selector generation tool 102 determines that this result is not valid and instructs the language model to regenerate the selector. In this case, the selector generation tool 102 prompts the language model to generate a new selector by giving it a prompt instructing it to generate a selector that searches a wider range. An example of a prompt to be input to the language model is shown below.

[0065] (Example prompt) "The conditions you provided did not allow us to find the target element on the user interface. The conditions may be too strict, so please loosen the generation conditions and search again, for example by using partial matching of attribute values ​​or excluding attributes that may change dynamically (such as IDs). If it is highly likely that the element is outside the current display area, please include in your response that additional actions such as scrolling are required."

[0066] An upper limit may be set on the number of iterations for regenerating selectors using the language model. The upper limit on the number of iterations is set in advance, for example, by the designer of software 100. If the number of iterations for regenerating selectors reaches the upper limit, the selector generation tool 102 outputs error information indicating that a valid selector cannot be generated.

[0067] If the selector generation tool 102 is able to generate a valid selector using the language model, it provides the generated selector to the test tool 101. The selector generation tool 102 also stores the generated selector in the history storage unit 12c.

[0068] The test tool 101 executes the test steps set in the test step information based on the selectors generated by the selector generation tool 102. If the test tool 101 succeeds in executing this test step, that is, if it can identify the operation element to be tested based on the generated selectors, it proceeds to the next test step. On the other hand, if the test step fails, that is, if it cannot identify the operation element to be tested based on the generated selectors, the test tool 101 notifies the selector generation tool 102 of the test step failure.

[0069] Upon receiving a test step failure notification from the test tool 101, the selector generation tool 102 adds flag information to the selectors stored in the history storage unit 12c, indicating that the test tool 101 failed to identify the operation element, and instructs the language model to regenerate the selectors. In this case, the selector generation tool 102 can, for example, instruct the language model to generate new selectors while avoiding the newly stored information in the history storage unit 12c by providing a prompt. An example of a prompt to be input to the language model is shown below.

[0070] (Example prompt) "When generating selectors, please analyze the 'patterns of failed selectors' included in the input history information. In the previous attempt, the use of specific attributes or structures caused errors (element not found or misidentified). Avoid these failure patterns and propose new, more reliable selectors using different attributes and structural approaches."

[0071] Furthermore, an upper limit may be set on the number of regeneration iterations for selectors in response to failure notifications from test tool 101. The number of regeneration iterations based on these failure notifications may be counted together with the number of regeneration iterations based on the number of selectors generated by the language model described above, or they may be counted separately. If the number of iterations reaches the upper limit, selector generation tool 102 will output error information indicating that a valid selector cannot be generated.

[0072] However, if the number of repetitions described above reaches the upper limit, or if the test tool 101 is unable to generate a selector that can identify the operation element, the selector generation tool 102 may search for the operation element in various ways before outputting error information or the like. In this embodiment, the selector generation tool 102 causes the test tool 101 to perform a predetermined operation on the operation screen of the software 100, and searches for the operation element by changing the display content of the operation screen.

[0073] One situation where the operation elements cannot be identified is when, for example, the entire operation screen is not displayed and the operation elements are located in the hidden parts. In this embodiment, the selector generation tool 102 changes the display content by having the test tool 101 scroll the operation screen or perform a screen transition, and attempts to regenerate the selector based on the display content of the operation screen after the change. For example, the selector generation tool 102 gives the test tool 101 a command to scroll the operation screen, and in response to this command, the test tool 101 scrolls the operation screen and provides the selector generation tool 102 with image data of a screenshot captured of the operation screen after scrolling. The selector generation tool 102 attempts to regenerate the selector by repeating the above procedure based on the new image data, and is expected to be able to identify the operation elements that could not be identified with the previous display content based on the new display content. The scroll direction (up, down, left, right) and scroll amount may be predetermined by the designer, etc., or may be determined by the language model.

[0074] For example, the selector generation tool 102 gives a command to the test tool 101 to perform a click operation on a button for screen transition provided on the operation screen. In response to this command, the test tool 101 transitions the operation screen from the current screen to another screen and provides the selector generation tool 102 with a screenshot image of the operation screen after the transition. The selector generation tool 102 attempts to regenerate the selector by repeating the above procedure based on the new image data, and is expected to be able to identify operation elements that could not be identified with the previous display content based on the new display content. Information such as the method of screen transition may be provided to the selector generation tool 102 in advance, or it may be determined by the language model.

[0075] The selector generation tool 102 may perform either scrolling the operation screen or a screen transition, or both. When performing both scrolling and a screen transition, it is preferable for the selector generation tool 102 to first scroll to search for an operation element, and only if this search is unsuccessful, to perform a screen transition to search for an operation element. By processing in this priority order, the selector generation tool 102 can be expected to improve the efficiency of searching for an operation element. However, the selector generation tool 102 may also perform a screen transition first and only scroll if that is unsuccessful.

[0076] Furthermore, the methods for changing the display of the operation screen are not limited to scrolling and screen transitions as described above. The selector generation tool 102 may change the display of the operation screen and search for operation elements using any other method.

[0077] Furthermore, if the selector generation tool 102 cannot identify an operation element by searching using the method described above, it searches for a selector of an operation element similar to the target operation element from among the selectors of successful cases stored in the history storage unit 12c. The history storage unit 12c stores information about the test action to be performed for each selector, and this information includes information such as the name of the operation element. The selector generation tool 102 converts the information such as the test action or name related to the operation element to be searched and the information such as the test action or name of the operation element related to the selector of the successful case stored in the history storage unit 12c into feature vectors and calculates the similarity of the feature vectors. The selector generation tool 102 searches for the selector information stored in the history storage unit 12c that has the highest similarity to the information of the operation element to be searched and provides the selector with the highest similarity to the test tool 101 as an alternative selector for the operation element to be searched. The test tool 101 attempts to perform the test using the given alternative selector.

[0078] If the test by the test tool 101 fails even when using the above-mentioned alternative selectors, the selector generation tool 102 may attempt to perform the test by sequentially providing the test tool 101 with alternative selectors, such as the second most similar selector, the third most similar selector, and so on, from among the selectors stored in the history storage unit 12c. If the selector generation tool 102 fails to succeed even after attempting to perform the test with a predetermined number of alternative selectors, it will determine that it has failed to generate a selector and output an error message or the like.

[0079] The test tool 101 sequentially executes all defined test steps for the software 100, requesting the selector generation tool 102 to generate selectors as needed, and outputs test result information determining whether the software 100 is functioning correctly or incorrectly. If the selector generation tool 102 fails to generate selectors, the test tool 101 outputs an error message indicating that the test steps cannot be executed and interrupts the test.

[0080] <Flowchart> Figures 5 and 6 are flowcharts illustrating an example of the processing procedure performed by the information processing device 1 according to this embodiment as a selector generation tool 102. The data acquisition unit 11a of the processing unit 11 of the information processing device 1 according to this embodiment acquires various data provided by the test tool 101, which performs tests on the software 100, along with the request for selector generation (step S1). Here, the data acquisition unit 11a may include, for example, HTML data defining the screen displayed by the software 100, image data of a screenshot capturing the displayed screen, and test step information indicating the content of the test to be performed.

[0081] The data simplification unit 11b of the processing unit 11 performs a process to simplify the HTML data included in the data acquired in step S1 (step S2). The data simplification unit 11b simplifies the acquired HTML data by deleting predetermined tags, attributes, comment nodes, and blank text nodes included in the HTML data.

[0082] The selector generation unit 11c of the processing unit 11 searches for a generated selector that matches the requested selector among the selectors stored in the history storage unit 12c based on the data acquired in step S1 (step S3). Based on the search result in step S3, the selector generation unit 11c determines whether or not a generated selector that matches the requested selector exists (step S4). If a matching selector exists in the history storage unit 12c (S4: YES), the selector generation unit 11c reads this selector from the history storage unit 12c (step S5) and proceeds to step S9.

[0083] If no matching selector exists in the history storage unit 12c (S4: NO), the selector generation unit 11c generates the selector requested by the test tool 101 using the language model whose information is stored in the model information storage unit 12b (step S6). The selector generation unit 11c inputs the test step information and screenshot image data obtained in step S1, the simplified HTML data obtained in step S2, the action definition information that defines the actions that the test tool 101 can perform, and the history information regarding the selector of the failure case stored in the history storage unit 12c, along with a prompt instructing the generation of a selector based on this information, to the language model, and generates the selector requested by the test tool 101 by obtaining the text information output by the language model.

[0084] The selector verification unit 11d of the processing unit 11 verifies the validity of the selector generated in step S6 (step S7). In this embodiment, the selector verification unit 11d determines that the selector is valid if the language model generates only one selector, and that it is not valid if the language model generates two or more selectors or if it does not generate any selectors. The selector generation unit 11c determines whether the selector is valid or not based on the verification result in step S7 (step S8).

[0085] If the selector is invalid (S8: NO), the selector verification unit 11d returns to step S6 and regenerates the selector using the language model. At this time, if the language model generates two or more selectors, the selector verification unit 11d prompts the language model to, for example, regenerate a more specific selector. If the language model cannot generate any selectors, the selector verification unit 11d prompts the language model to, for example, generate a selector that searches a wider range. Although not shown in this flowchart, if the number of selector regeneration iterations reaches the upper limit, the selector verification unit 11d outputs error information and terminates the selector generation process.

[0086] After reading the selector stored in the history storage unit 12c in step S5, or if the selector is determined to be valid in step S8 (S8: YES), the processing unit 11 outputs the read selector or the selector determined to be valid to the test tool 101 (step S9). The selector verification unit 11d determines whether the test performed by the test tool 101 using the selector output in step S9 was successful (i.e., whether the operating element was identified by the selector) (step S10). At this time, the selector verification unit 11d can determine whether the operating element was identified by the selector based on whether or not it received a test failure notification from the test tool 101.

[0087] If the test is successful (S10:YES), the selector verification unit 11d stores the selector output to the test tool 101 in step S9 as a success case in the history storage unit 12c (step S11), and terminates the selector generation process.

[0088] If the test fails (S10:NO), the selector verification unit 11d stores the selector output to the test tool 101 in step S9 as a failure case in the history storage unit 12c (step S12). The selector verification unit 11d determines whether the number of iterations of regenerating the selector based on the test failure using the selector output to the test tool 101 has reached the verification upper limit (step S13). If the number of iterations has not reached the upper limit (S13:NO), the selector verification unit 11d returns to step S6 and regenerates the selector using the language model. At this time, the language model is given the failure case stored in the history storage unit 12c in step S12, and the language model generates a new selector that avoids this failure case.

[0089] If the number of repetitions reaches the upper limit (S13:YES), the operation element search unit 11e of the processing unit 11 performs an operation element search process to find the operation element that will be the target of the selector to be generated from the operation screen displayed by the software 100 (step S14). Details of the search process will be described later. The operation element search unit 11e determines whether the search process in step S14 was successful, that is, whether the test tool 101 was able to identify the operation element using the selector obtained by the search process (step S15). If the search process is successful (S15:YES), the selector verification unit 11d stores the selector obtained by the search process as a success case in the history storage unit 12c (step S11), and terminates the selector generation process. If the search process fails (S15:NO), the operation element search unit 11e outputs error information indicating that a selector could not be generated (step S16), and terminates the process.

[0090] Figures 7 and 8 are flowcharts showing an example of the procedure for searching for an operation element performed by the information processing device 1 according to this embodiment, and represent the process performed in step S14 of the flowcharts shown in Figures 5 and 6. The operation element search unit 11e of the information processing device 1 according to this embodiment outputs a command to scroll the operation screen to the test tool 101 (step S31). The test tool 101, having received a command from the selector generation tool 102, scrolls the operation screen by performing appropriate operations on the software 100 and provides the selector generation tool 102 with image data of a screenshot captured of the scrolled operation screen. The operation element search unit 11e acquires the image data provided by the test tool 101 in response to the scroll command (step S32). Based on the image data acquired in step S32, the operation element search unit 11e regenerates the selector in the manner shown in steps S6 to S13 of the flowcharts in Figures 5 and 6 (step S33). The operation element search unit 11e outputs the selector regenerated in step S33 to the test tool 101 (step S34). The operation element search unit 11e determines whether the test using the selector was successful or not based on whether or not a failure notification was given from the test tool 101 that performed the test using the selector output in step S34 (step S35). If the test is successful (S35: YES), the operation element search unit 11e determines that it has successfully searched for the operation element (step S45) and terminates the process.

[0091] If the test fails (S35: NO), the operation element search unit 11e outputs a command to the test tool 101 to transition the operation screen (step S36). The test tool 101, having received the command from the selector generation tool 102, transitions the operation screen by performing appropriate operations on the software 100 and provides the selector generation tool 102 with image data of a screenshot of the operation screen after the transition. The operation element search unit 11e acquires the image data provided by the test tool 101 in accordance with the screen transition command (step S37). The operation element search unit 11e regenerates the selector based on the image data acquired in step S37 (step S38). The operation element search unit 11e outputs the selector regenerated in step S38 to the test tool 101 (step S39). The operation element search unit 11e determines whether the test using the selector was successful or not based on whether or not a failure notification was given from the test tool 101 that performed the test using the selector output in step S39 (step S40). If the test is successful (S40:YES), the operation element search unit 11e determines that it has successfully searched for the operation element (step S45) and terminates the process.

[0092] If the test fails (S40: NO), the operation element search unit 11e searches the selectors of successful cases stored in the history storage unit 12c for a selector similar to the selector requested by the test tool 101 (step S41). The operation element search unit 11e outputs the selector obtained from the history storage unit 12c in the search in step S41 to the test tool 101 (step S42). The operation element search unit 11e determines whether the test using the selector was successful or not based on whether or not a failure notification was given from the test tool 101 that performed the test using the selector output in step S42 (step S43). If the test is successful (S43: YES), the operation element search unit 11e determines that it has succeeded in searching for the operation element (step S45) and terminates the process. If the test fails (S43: NO), the operation element search unit 11e determines that it has failed to search for the operation element (step S44) and terminates the process.

[0093] Figure 9 is a flowchart showing an example of the processing procedure performed by the information processing device 1 according to this embodiment as a test tool 101. The test processing unit 11f of the processing unit 11 of the information processing device 1 according to this embodiment acquires various data from the user, such as test step information which lists the test steps to be performed in the test, HTML data which defines the configuration of the operation screen displayed by the software 100, and information regarding the configuration of the software 100 (step S61).

[0094] The test processing unit 11f obtains one test step in the order of execution from the test step information contained in the data obtained in step S61 (step S62). The test processing unit 11f provides the selector generation tool 102 with a request to generate a selector necessary to execute the test step obtained in step S62 (step S63). The test processing unit 11f obtains the selector output by the selector generation tool 102 in response to the selector generation request given in step S63 (step S64). Based on the selector obtained in step S64, the test processing unit 11f performs the test related to the test step obtained in step S62 (step S65).

[0095] The test processing unit 11f determines whether or not it was able to identify the operating element in the test step performed in step S65 (step S66). If the operating element was identified (S66: YES), the test processing unit 11f determines whether or not it has completed the execution of tests for all test steps included in the test step information obtained in step S61 (step S67). If it has not completed the execution of tests for all test steps (S67: NO), the test processing unit 11f returns to step S62 and performs the same process for the next test step. If it has completed the execution of tests for all test steps (S67: YES), the test processing unit 11f stores the test results in the storage unit 12 (step S68) and terminates the process.

[0096] Furthermore, if the operation element cannot be identified (S66: NO), the test processing unit 11f notifies the selector generation tool 102 that it failed to identify the operation element using the selector (step S69). Subsequently, the test processing unit 11f determines whether or not it has received a command such as a scroll command or a screen transition command from the selector generation tool 102 (step S70). If a command has been received (S70: YES), the test processing unit 11f performs a screen operation such as scrolling or a screen transition according to the given command (step S71). The test processing unit 11f captures the operation screen after the screen operation in step S71 (step S72) and obtains image data. The test processing unit 11f sends the image data of the operation screen captured in step S72 to the selector generation tool 102 (step S73) and returns to step S64.

[0097] If no command is given from the selector generation tool 102 (S70: NO), the test processing unit 11f determines whether or not it has received an error notification from the selector generation tool 102 indicating that it cannot generate a selector (step S74). If no error notification is given (S74: NO), the test processing unit 11f returns to step S64. If an error notification is given (S74: YES), the test processing unit 11f interrupts the test and terminates processing.

[0098] <Summary> In the information processing system according to this embodiment with the above configuration, the information processing device 1 acquires HTML data (screen definition information) that defines the configuration of the operation screen displayed by the software 100 under test, acquires image data captured from the content of the operation screen, acquires test step information (test content information) related to the content of the tests to be performed on the software 100 under test, and, based on this acquired information, generates selectors (identification information) for identifying operation elements provided on the operation screen using a language model that accepts text and image input. As a result, the information processing system according to this embodiment is expected to support the execution of tests on the software 100 because the user does not need to create selectors themselves in order to perform tests.

[0099] Furthermore, in the information processing system according to this embodiment, the screen definition information is HTML data written in a markup language. The information processing device 1 removes information of predetermined tags or attributes from the acquired HTML data and generates a selector using a language model based on the HTML data after the removal. As a result, the information processing system according to this embodiment is expected to reduce the amount of information input to the language model.

[0100] Furthermore, in the information processing system according to this embodiment, the test step information includes information about multiple steps to be performed in the test. The information processing device 1 repeatedly generates selectors for each test step included in the acquired test step information, thereby generating multiple selectors for multiple test steps. The information processing device 1 also generates and outputs information that associates the actions of the test tool 101 that perform the test on the software 100 in each test step with the selectors generated for that test step, using a language model. As a result, the information processing system according to this embodiment is expected to generate selectors for multiple test steps performed on the software 100 under test, thereby supporting the execution of tests by the test tool 101.

[0101] Furthermore, in the information processing system according to this embodiment, the information processing device 1 stores selectors generated using a language model in the history storage unit 12c, and determines whether the test content information relating to the operation element to be identified for the test step targeted for selector generation matches the test content information associated with the selector stored in the history storage unit 12c. If the information processing device 1 determines that they do not match, it generates a selector using the language model, and if it determines that they match, it reuses the selector stored in the history storage unit 12c. As a result, the information processing system according to this embodiment is expected to reduce the processing load of using the language model by reusing the generated selector.

[0102] Furthermore, in the information processing system according to this embodiment, the information processing device 1 determines whether the selector generated by the language model can uniquely identify the operation element. If multiple operation elements can be identified by the selector, the information processing device 1 gives an instruction to narrow down the generation conditions and causes the language model to regenerate the selector. Also, if no operation elements can be identified by the selector, the information processing device 1 gives an instruction to loosen the generation conditions and causes the language model to regenerate the selector. As a result, the information processing system according to this embodiment is expected to enable the language model to generate selectors with high accuracy.

[0103] Furthermore, in the information processing system according to this embodiment, if the test tool 101 is unable to identify an operation element based on the generated selector, the information processing device 1 stores information about this selector as a failure case in the history storage unit 12c. The information processing device 1 generates a selector using a language model based on the HTML data, image data, test step information, and the information stored as a failure case in the history storage unit 12c. As a result, in the information processing system according to this embodiment, it is expected that the language model will generate selectors while avoiding failure cases, and the accuracy of selector generation using the language model can be improved.

[0104] Furthermore, in the information processing system according to this embodiment, if the test tool 101 cannot identify an operation element based on the generated selector, the information processing device 1 causes the test tool 101 to perform a predetermined operation, such as scrolling or screen transition, on the operation screen displayed by the software 100 to be executed on 100. After performing the predetermined operation, the test tool 101 provides the selector generation tool 102 with image data of a screenshot capturing the operation screen that has changed as a result of the predetermined operation. The information processing device 1 generates a selector using a language model based on the image data that captures the display content of the operation screen after the predetermined operation has been performed. As a result, the information processing system according to this embodiment can be expected to search for an operation element by scrolling or transitioning the operation screen, etc., and generate a selector, for example, when the operation element to be operated is not displayed in the test step.

[0105] Furthermore, in the information processing system according to this embodiment, the information processing device 1 stores in the history storage unit 12c selectors that the test tool 101 was able to identify as successful cases. If the test tool 101 is unable to identify an operation element based on the generated selector, the information processing device 1 retrieves a selector related to an operation element similar to this one from the selectors stored in the history storage unit 12c, and causes the test tool 101 to identify the operation element based on the retrieved selector. As a result, the information processing system according to this embodiment can be expected to attempt to perform testing with the test tool 101 by reusing successful selectors when it is not possible to generate a selector using the language model.

[0106] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of this disclosure is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0107] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used. [Explanation of Symbols]

[0108] 1. Information processing equipment (computer) 11 Processing Section 11a Data acquisition unit 11b Data simplification section 11c Selector generation unit 11d Selector Verification Unit 11e Operation element search section 11f Test Processing Unit 12 Storage section 12a Program (Computer Program) 12b Model Information Storage Unit 12c History storage unit 13 Communications Department 14 Display section 15 Control section 99 Recording media 100 Software 101 Test Tools 102 Selector Generation Tool

Claims

1. Information processing device, Obtain screen definition information that defines the configuration of the user interface displayed by the software under test. Acquire image data that captures the display content of the aforementioned operation screen. The test content information relating to the content of the tests performed on the software under test is acquired. Based on the acquired screen definition information, image data, and test content information, a language model that accepts text and image input is used to generate specific information for identifying the operation elements provided on the operation screen. The language model determines whether the specific information generated by the language model can uniquely identify the operation element. If the aforementioned operating element cannot be uniquely identified, and multiple aforementioned operating elements can be identified by the identified information, the language model is instructed to regenerate the identified information by narrowing down the generation conditions. If the aforementioned operational element cannot be uniquely identified, and if none of the aforementioned operational elements can be identified by the identified information, an instruction is given to relax the generation conditions and cause the language model to regenerate the identified information. Information processing methods.

2. The aforementioned screen definition information is information written in a markup language, The aforementioned information processing device Delete the information of a predetermined tag or attribute from the acquired screen definition information. Based on the screen definition information after deletion, the language model is used to generate the specific information. The information processing method according to claim 1.

3. The aforementioned test content information includes information about multiple steps performed in the test, The aforementioned information processing device The generation of the specific information is repeated for each step included in the acquired test content information, thereby generating multiple pieces of the specific information for each of the multiple steps. The system outputs information that associates the actions of the test device that perform the test on the software in the step with the specific information generated for that step. The information processing method according to claim 1.

4. The aforementioned information processing device The generated specific information is stored in the memory unit. The system determines whether the test content information related to the target step matches the test content information associated with the specific information stored in the memory unit. If it is determined that they do not match, the specific information is generated using the language model. If a match is determined, the specific information stored in the storage unit is used. The information processing method according to claim 3.

5. The aforementioned information processing device The test device acquires action definition information that defines the actions it can perform, The language model is given a constraint to output only the actions included in the action definition information, thereby generating the specific information and the corresponding actions. The information processing method according to claim 3.

6. The aforementioned information processing device If a test device that performs tests on the software based on the generated specific information is unable to identify the operating element, the information relating to the specific information is stored in the storage unit. Based on the screen definition information, the image data, and the test content information, and the information stored in the storage unit, the language model is used to generate the specific information. The information processing method according to claim 1.

7. The aforementioned information processing device If the test device that performs tests on the software based on the generated specific information is unable to identify the operating element, the test device is instructed to perform a predetermined operation on the operation screen. After performing the predetermined operation, image data is obtained that captures the display content of the operation screen. Based on the acquired image data, the specific information is generated using the language model. The information processing method according to claim 1.

8. The predetermined operation is an operation to scroll the operation screen, or an operation to transition the operation screen to another screen. The information processing method according to claim 7.

9. The aforementioned information processing device The test device that performs the tests on the aforementioned software stores specific information in a storage unit that allows it to identify the operating elements. If the test device is unable to identify the operating element based on the generated specific information, specific information relating to an operating element similar to the aforementioned operating element is obtained from the specific information stored in the storage unit. Based on the acquired specific information, the test device is made to identify the operating element. The information processing method according to claim 1.

10. On the computer, Obtain screen definition information that defines the configuration of the user interface displayed by the software under test. Acquire image data that captures the display content of the aforementioned operation screen. The test content information relating to the content of the tests performed on the software under test is acquired. Based on the acquired screen definition information, image data, and test content information, a language model that accepts text and image input is used to generate specific information for identifying the operation elements provided on the operation screen. The language model determines whether the specific information generated by the language model can uniquely identify the operation element. If the aforementioned operating element cannot be uniquely identified, and multiple aforementioned operating elements can be identified by the identified information, the language model is instructed to regenerate the identified information by narrowing down the generation conditions. If the aforementioned operational element cannot be uniquely identified, and if none of the aforementioned operational elements can be identified by the identified information, the language model is instructed to relax the generation conditions and regenerate the identified information. A computer program that instructs a computer to perform a process.

11. Equipped with a processing unit, The aforementioned processing unit, Obtain screen definition information that defines the configuration of the user interface displayed by the software under test. Acquire image data that captures the display content of the aforementioned operation screen. The test content information relating to the content of the tests performed on the software under test is acquired. Based on the acquired screen definition information, image data, and test content information, a language model that accepts text and image input is used to generate specific information for identifying the operation elements provided on the operation screen. The language model determines whether the specific information generated by the language model can uniquely identify the operation element. If the aforementioned operating element cannot be uniquely identified, and multiple aforementioned operating elements can be identified by the identified information, the language model is instructed to regenerate the identified information by narrowing down the generation conditions. If the aforementioned operational element cannot be uniquely identified, and if none of the aforementioned operational elements can be identified by the identified information, an instruction is given to relax the generation conditions and cause the language model to regenerate the identified information. Information processing device.

Images