Information processing device, information processing method, and program

Abstract

The present invention provides an information processing device, an information processing method, and a program that can uniquely identify the target of operation during testing. [Solution] The information processing device includes a test specification document that shows the content of the test, an acquisition means that acquires screen configuration information and image information related to the screen from the test target, an extraction means that extracts test operations based on the content of the test shown in the test specification document, a conversion means that converts the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding, and an identification means that identifies the element of the target of operation in the test operation from among the elements on the screen based on the information of the target of operation, the screen configuration information, and the image information converted into vectors.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, in the development of systems, applications, etc., tests are conducted to determine whether the developed systems, applications, etc. operate properly. For example, in order to confirm whether a developed web system operates properly, predetermined inputs are made on the screen according to a test scenario, and tests are conducted to determine whether it operates correctly based on changes in the displayed content and screen transitions. Similarly, when functions are added after the release of a system, it is also necessary to conduct operation confirmation tests. Since it is extremely labor-intensive for the person in charge to perform these tests manually, tools, etc. that automatically perform the work related to testing have also been developed.

[0003] Patent Document 1 proposes a technique for acquiring screen information of an application to be tested and test information of the application, deriving correspondence relationship information corresponding to the display positions of the operation target objects and element names included in the acquired screen information based on the acquired screen information, and outputting an execution script for testing the application based on the derived correspondence relationship information and the element names included in the test information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 uses a concise specification of the target of operation in testing using element names, but the target of operation cannot always be uniquely determined by element names alone, in which case manual correction is required. The present invention aims to provide an information processing device, an information processing method, and a program that can uniquely identify the target of operation in testing. [Means for solving the problem]

[0006] The information processing device according to the present invention is characterized by comprising: a test specification document indicating the content of a test; an acquisition means for acquiring screen configuration information and image information relating to a screen from a test target; an extraction means for extracting test operations based on the content of the test indicated in the test specification document; a conversion means for converting information on the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that performs embedding; and an identification means for identifying the element that is the target of operation in the test operation from among the elements on the screen based on the information on the target of operation, the screen configuration information, and the image information converted into vectors. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing device, an information processing method, and a program that can uniquely identify the target of operation in a test. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2] This figure shows an example of the hardware configuration of an information processing device. [Figure 3] This figure shows an example of the functional configuration of the information processing device in the first embodiment. [Figure 4A] This figure illustrates an example of processing in the information processing device according to the first embodiment. [Figure 4B] This diagram illustrates an example of score calculation processing. [Figure 5] This is a diagram for explaining the division of test operations. [Figure 6] This is a diagram for explaining the description of test operations. [Figure 7] This is a diagram for explaining the description of test operations. [Figure 8] This is a diagram for explaining another example of score calculation processing. [Figure 9] This is a diagram showing an example of the functional configuration of an information processing apparatus in the second embodiment. [Figure 10] This is a diagram for explaining an example of the processing of an information processing apparatus in the second embodiment. [Figure 11] This is a diagram showing an example of a test case outline. [Figure 12] This is a diagram for explaining an example of a prompt regarding the generation of a description of test operations. [Figure 13A] This is a diagram for explaining an example of a prompt regarding the generation of a description of test operations. [Figure 13B] This is a diagram for explaining an example of a prompt regarding the generation of a description of test operations. [Figure 14] This is a diagram for explaining an example of description data of test operations. [Figure 15] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations. [Figure 16A] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations. [Figure 16B] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations. [Figure 17] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations. [Figure 18A] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations. [Figure 18B] This is a diagram for explaining an example of a prompt regarding the generation (correction) of a description of test operations.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described based on the drawings.

[0010] <First Embodiment> FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment of the present invention. The information processing system in the present embodiment includes an information processing apparatus 101 and a server apparatus 102. The information processing apparatus 101 and the server apparatus 102 are communicably connected via a network 103. Note that the network 103 only needs to be able to communicably connect the information processing apparatus 101 and the server apparatus 102, and its type is not particularly limited.

[0011] The information processing apparatus 101 automatically executes a test for confirming whether a system or application to be tested, etc. operates normally, based on a test specification indicating the content of the test. The server apparatus 102 is a server apparatus that provides a system or application to be tested, etc. Here, as an example, the information processing apparatus 101 automatically executes a test on a system (application) to be tested provided by the server apparatus 102, based on a test specification regarding the test.

[0012] The information processing apparatus 101 acquires a test specification, screen configuration information and image information regarding the screen of the system (application) to be tested, specifies elements to be operated on the screen based on them, and performs operations according to the test content. Further, the information processing apparatus 101 checks the validity of the result, such as whether the result obtained by the screen operation is the expected result indicated in the test specification, and captures and records and stores the screen as evidence.

[0013] The following explanation uses a web screen test of the system (application) under test provided by the server device 102 as an example. However, it is not limited to web screens and can be applied to any system or application where operations can be performed on the elements (UI, etc.) displayed on the screen by the system (application) under test. Furthermore, although the example shows the information processing device 101 that executes the test and the server device 102 that provides the system or application under test as separate devices, the information processing device 101 that executes the test may also be configured to have the system or application under test.

[0014] Figure 2 shows an example of the hardware configuration of the information processing device 101 in this embodiment. The information processing device 101 includes a CPU 201, ROM 202, RAM 203, auxiliary storage device 204, output device 205, input device 206, and network I / F 207. The CPU 201, ROM 202, RAM 203, auxiliary storage device 204, output device 205, input device 206, and network I / F 207 are connected to each other via a system bus 208.

[0015] The CPU (Central Processing Unit) 201 is a central processing unit that controls various operations of the information processing device 101. For example, the CPU 201 may control the operation of the entire information processing device 101. The ROM (Read Only Memory) 202 stores control programs, boot programs, etc., that can be executed by the CPU 201. The RAM (Random Access Memory) 203 is the main memory of the CPU 201 and is used as a work area or a temporary storage area for deploying various programs.

[0016] The auxiliary storage device 204 stores various data and programs. The auxiliary storage device 204 is implemented by a storage device capable of temporarily or permanently storing various data, such as an HDD (Hard Disk Drive) or non-volatile memory such as an SSD (Solid State Drive).

[0017] The output device 205 is a device that outputs various types of information and is used to present various types of information to the user. For example, the output device 205 may be implemented by a display device such as a display. The output device 205 may present information to the user by displaying various types of display information. As another example, the output device 205 may be implemented by an acoustic output device that outputs sounds such as voice or electronic sounds. In this case, the output device 205 may present information to the user by outputting sounds such as voice or electronic sounds. Furthermore, the device to which the output device 205 is applied may be appropriately changed depending on the medium used to present information to the user.

[0018] The input device 206 is used to receive various instructions from the user. For example, the input device 206 may include input devices such as a mouse, keyboard, or touch panel. As another example, the input device 206 may include a sound collection device such as a microphone to collect the voice spoken by the user. In this case, the collected voice may be subjected to various analysis processes such as acoustic analysis and natural language processing so that the content of the voice is recognized as an instruction from the user. Furthermore, the device applied as the input device 206 may be changed as appropriate depending on the method of recognizing the user's instructions. In addition, multiple types of devices may be applied as the input device 206.

[0019] Network I / F207 is used for communication with external devices via a network. The device used as Network I / F207 may be changed as appropriate depending on the type of communication path and the applicable communication method.

[0020] The CPU 201 loads the program stored in the ROM 202 or auxiliary storage device 204 into the RAM 203 and executes the program, thereby realizing the various functions and processes of the information processing device 101 described later. The program for the information processing device 101 may be provided to the information processing device 101 by means of a recording medium such as a CD-ROM, or it may be downloaded via a network or the like. When the program for the information processing device 101 is provided by a recording medium, the program recorded on the recording medium is installed in the auxiliary storage device 204 when the recording medium is set in a predetermined drive device.

[0021] The configuration shown in Figure 2 is merely an example and does not necessarily limit the hardware configuration of the information processing device 101 in this embodiment. For example, some components such as the output device 205 and the input device 206 may be omitted. Another example is that configurations appropriate to the functions to be realized by the information processing device 101 may be added as needed.

[0022] Figure 3 shows an example of the functional configuration of the information processing device 101 in the first embodiment. The information processing device 101 includes a communication unit 301, a control unit 302, an input / output control unit 303, a storage unit 304, an acquisition unit 305, a division unit 306, a conversion unit 307, an arithmetic unit 308, a specific unit 309, and an execution unit 310.

[0023] The communication unit 301 is a communication interface for each component of the information processing device 101 to send and receive information with other devices such as the server device 102 via the network 103. The communication unit 301 can be implemented, for example, by the network I / F 207. In the following description, when each component of the information processing device 101 sends and receives information with other devices, it will be assumed that the information is sent and received via the communication unit 301 unless otherwise specified.

[0024] The control unit 302 is responsible for controlling each component of the information processing device 101. The input / output control unit 303 performs various processes related to presenting various information to the user and receiving information input from the user (e.g., instructions). For example, the input / output control unit 303 may perform processes related to presenting a UI (User Interface) and processes related to receiving input via the UI. This enables the information processing device 101 to recognize instructions from the user and present the user with the results of the processing corresponding to those instructions.

[0025] The memory unit 304 schematically represents a memory area for storing various data and programs. For example, the memory unit 304 may store data and programs for each component of the information processing device 101 to execute processing. The memory unit 304 may also store test specifications, screen configuration information, and image information acquired by the acquisition unit 305, or it may store test operation description data obtained by dividing the test specifications. The memory unit 304 may also store screen images acquired as evidence of the executed tests. Furthermore, the memory unit 304 may store machine learning models that embed information about each element and convert it into vector representations (embedded representations, distributed representations).

[0026] The acquisition unit 305 acquires a test specification document that shows the content of the test to be performed. The test specification document shows the operations to be performed (operation instructions) and the confirmation items (confirmation instructions) to be confirmed during the test. The acquisition unit 305 also acquires screen configuration information and image information related to the screen of the system (application) under test. Screen configuration information is information related to the screen configuration, such as HTML (HyperText Markup Language) information and CSS (Cascade Style Sheet) information. Image information is information related to the images displayed on the screen, such as GUI (Graphical User Interface) images such as icon images.

[0027] The division unit 306 divides the test content, which consists of multiple operation contents and confirmation contents, etc., as shown in the test specification document acquired by the acquisition unit 305, into test operations. For example, the division unit 306 divides the test content shown in the test specification document into multiple test operations such that only one operation contents or one confirmation contents is included in each test operation. The division unit 306 also converts each divided test operation according to a predetermined notation to generate test operation description data. In this embodiment, as an example, BNF notation (Backus-Naur form) is applied to the description of the operation target in the test operation. The test operation description data may also be generated by having a test person or the like describe each divided test operation according to a predetermined notation.

[0028] The transformation unit 307 vectorizes the information about each element involved in the execution of the test using a machine learning model that embeds the information, and converts it into a vector representation (embedding representation, distributed representation). Here, the vector representation is a representation of the information about the elements using multi-dimensional real vectors. The machine learning model is trained so that vector representations of similar elements are placed close together in the vector space (the distance between vectors is small, the similarity is high).

[0029] Specifically, the conversion unit 307 converts the element names targeted in the test operation, which have been divided by the division unit 306, into vector representations using a machine learning model that performs text embedding. Furthermore, based on the screen configuration information acquired by the acquisition unit 305, the conversion unit 307 converts the screen configuration information for each element (screen component) that may be targeted in the execution of the test operation on the screen of the system (application) under test into vector representations using a machine learning model that performs text embedding. Additionally, based on the image information acquired by the acquisition unit 305, the conversion unit 307 converts the image information for each image element that may be targeted in the execution of the test operation on the screen of the system (application) under test into vector representations using a machine learning model that performs image embedding. For example, a machine learning model such as Sentence-BERT may be applied for text embedding. For image embedding, a so-called multimodal embedding model capable of embedding both images and text may be applied; for example, a model such as OpenAI's CLIP may be applied. In addition to these machine learning models, it is also possible to generate a machine learning model by performing machine learning to convert information about elements into vector representations (embedding representations, distributed representations), and then apply the generated machine learning model. Note that the machine learning model for image embedding is pre-trained to match the vector representation of the machine learning model for text embedding. For example, it is pre-trained to match the vector representation of a trash can icon image with the vector representation of the text "trash can". Alternatively, a multimodal embedding model may be applied as the machine learning model for text embedding, and the machine learning model for text embedding may be pre-trained to match the vector representation of the machine learning model for image embedding.

[0030] The calculation unit 308 calculates the similarity between the element to be operated on in the test operation and the screen components and image elements on the screen of the system (application) under test that may be targeted during the execution of the test operation, based on the vector representation of each piece of information obtained by the conversion unit 307. For example, the calculation unit 308 matches the vector representation of the element name to be operated on in the test operation with the vector representations of the screen configuration information and image information of the elements on the screen that may be targeted during the execution of the test operation, and calculates a score indicating the similarity to the element to be operated on in the test operation for each element on the screen that may be targeted during the execution of the test operation.

[0031] The identification unit 309 identifies the element to be operated on when executing a test operation from among the elements on the screen of the system (application) under test, based on the similarity obtained by the calculation unit 308. The identification unit 309 identifies the element with the highest similarity from among the elements on the screen of the system (application) under test as the element to be operated on in the test.

[0032] The execution unit 310 performs tests on the system (application) under test according to the test operation. The execution unit 310 performs the tests by operating the elements to be operated, which have been identified by the identification unit 309, according to the operation instructions (operation type) in the test operation. The execution unit 310 may also acquire evidence of the test execution (for example, a screen captured after the operation) and save it in the storage unit 304 or the like.

[0033] In the example described above, the machine learning model for embedding text or images is configured to be installed in the information processing device 101. However, it may also be configured to be installed in another information processing device that can communicate with the information processing device 101 via a network or the like. In this case, the information processing device 101 should send input to the machine learning model to the other information processing device 101 that is equipped with the machine learning model via a network interface 207 or the like, and receive the processing results for that input.

[0034] The processing in the information processing device 101 in the first embodiment will be described with reference to Figures 4A and 4B. Figures 4A and 4B are diagrams illustrating an example of processing in the information processing device 101 in the first embodiment, with Figure 4A showing the overall processing flow and Figure 4B showing the score calculation process. In the following explanation, we will use the case where the screen configuration information for the system (application) under test is HTML information created in HTML as an example, but we are not limited to this and the explanation can also be applied to screen configuration information created using other languages.

[0035] The information processing device 101 acquires a test specification 401 that indicates the contents of the test, such as the operations to be performed (operation instructions) and the items to be confirmed (confirmation instructions).

[0036] In process 402, the information processing device 101 divides the acquired test specification 401 into test operations. For example, the test specification 401 contains multiple test cases that define a series of operations to be performed in the test, and each test case contains multiple operations, confirmations, etc. The information processing device 101 divides the test content shown in the acquired test specification 401 into test cases, and further divides the divided test cases into multiple test operations such that one operation or one confirmation is included in one test operation. Then, for each divided test operation, the information processing device 101 converts each test operation according to a predetermined notation and generates test operation description data 403. Note that the generation of test operation description data 403 may also be performed by a test person or the like describing each test operation according to a predetermined notation for each divided test operation.

[0037] Let's take the following example test case as shown in Test Specification 401. (1) Open “URL / login”. (2) Enter "root" in the "Username or email" input area. (3) Enter "password" in the "Password" input area on the right side of the screen. (4) Click the "Sign in" button below the "Password" input area. (5) Confirm that it is the URL screen. ... The information processing device 101 divides the operation contents (confirmation contents) of (1), (2), (3), (4), (5), ... into one test operation each and generates test operation description data 403 as shown in Figure 5. Figure 5 shows an example of the description data for each test operation included in the test case 500 of the example described above, where, for example, test operation 501 corresponds to the operation contents of (1) described above. Here, the information processing device 101 selects the operation type from those that are predefined according to the operation contents. In addition, the input text item contains information other than the target of the operation in the input operation (operation type is input), etc. (for example, the text to be entered), as needed.

[0038] In this embodiment, the target of operation in the test operation description data 403 is described using BNF notation (Backusnaur notation) as an example. For example, the target of operation is described according to the notation shown in Figure 6(a), and the element units included in the target of operation are described according to the notation shown in Figure 6(b). Furthermore, the elements included in the target of operation and element units are described according to the notation shown in Figure 6(c). Thus, the description of the target of operation in the test operation description data 403 consists of an element name, attributes, and position, and is composed of at least an element name. For example, in the example shown in Figure 5, the target of operation "Password input area on the right side of the screen" is represented by "Password" as the element name, "input area" as the attribute related to the element name "Password", and "right side of the screen" as the position related to the element name "Password". Furthermore, for example, in the operation target "Sign in button below the "Password" input area", ""Sign in" corresponds to the element name, "button" corresponds to the attributes related to the element name "Sign in", and "below the "Password" input area" corresponds to the position related to the element name "Sign in". In addition, in the description of this position "below the "Password" input area", ""Password" corresponds to the element name, and "input area" corresponds to the attributes related to the element name "Password".

[0039] Furthermore, the names of the elements to be manipulated can be described using absolute or relative positions, or in matrix form. For example, suppose that the screen 701 shown in Figure 7 has a “Username or email” input area 702, a “Password” input area 703, and a “Sign in” button 704. In this case, depending on the description in the test specification, the description of the “Password” input area 703 as the target of manipulation can be an absolute position description such as “the “Password” input area on the right side of the screen,” or a relative position description such as “the “Password” input area below the “Username or email” input area” or “the “Password” input area above the “Sign in” button.” By enabling descriptions using absolute or relative positions or in matrix form in this way, it becomes possible to uniquely identify elements on the screen even when there are multiple elements with the same name on the screen or when the screen content is complex.

[0040] Returning to Figure 4A, in process 404, the information processing device 101 uses a parser to decompose the description of the target of operation in the test operation description data 403 generated for each test operation into element name 405, attribute 406, and position 407. The element name 405, attribute 406, and position 407 obtained by decomposing the test operation description data 403 by the parser are input to the score calculation process 421.

[0041] Furthermore, the information processing device 101 obtains HTML information (screen configuration information) 412 and image information 415 related to the web screen on which the test is executed from the system (application) 411 under test. The HTML information 412 is information about the screen configuration of the web screen created in HTML, and the image information 415 is information about the images displayed on the web screen.

[0042] In process 413, the information processing device 101 divides the acquired HTML information 412 related to the web screen into elements and outputs HTML information 414 for each element. For example, by dividing the acquired HTML information 412 related to the web screen into elements, the information processing device 101 can determine the status of a certain element (screen component). <input class="form-control gl-form-input bottom" required title="This field is required." data-qa-selector="password_field" type="password" name="user[password]" id="user_password"> If HTML information is obtained, it is output as the HTML information 414 for that element. The HTML information 414 for each element obtained by process 413 is input to the score calculation process 421.

[0043] In process 416, the information processing device 101 acquires image information for each element based on the acquired image information 415 related to the web screen, and outputs image information 417 for each element. For example, with respect to a certain element (image element), the information processing device 101 outputs an image obtained by cropping it from the acquired image information 415 related to the web screen based on the element's coordinates as the image information 417 for that element. The image information 417 for each element obtained by process 413 is input to the score calculation process 421.

[0044] In process 421, the information processing device 101 performs a score calculation process based on the information 405-407 regarding the target of operation in the test operation and the information 414, 417 regarding the Web screen of the test target, and calculates and outputs a score indicating the similarity to the target of operation in the test operation for each element on the Web screen. The information processing device 101 uses the element name 405, attribute 406, and position 407 of the input target of operation in the test operation, and the HTML information 414 for each element (screen component) and image information 417 for each element (image element) of the input Web screen of the test target to calculate the similarity between the target of operation in the test operation and the elements (screen components and image elements) on the Web screen. Details of the score calculation process in this process 421 will be described later.

[0045] In process 423, the information processing device 101 identifies the element to be operated on when executing the test operation from among the elements on the web screen under test, based on the score 422 for each element on the web screen under test obtained in process 421. Based on the score 422 for each element obtained in score calculation process 421, the information processing device 101 identifies the element on the web screen with the highest similarity to the element to be operated on in the test operation as the element to be operated on in the test operation. Here, if the description of the element to be operated on includes a specification such as "the nth element from the top...", the nth element from the top on the screen may be identified as the element to be operated on in the test operation.

[0046] In process 424, the information processing device 101 operates on the element to be operated on, which was identified in process 423, according to the test operation description data 403. As a result, the test corresponding to the test operation description data 403 is executed in the system (application) 411 under test.

[0047] In process 425, the information processing device 101 obtains the test execution results and saves the obtained execution results 426 as evidence related to the test. For example, the information processing device 101 obtains (captures) the screen of the system (application) 411 under test after an operation has been performed according to the test operation description data 403 and saves it as the execution result (evidence) 426.

[0048] In this way, after performing an operation according to the test operation description data 403 for a given test operation and executing the test of the system (application) 411 under test, the information processing device 101 obtains HTML information 412 and image information 415 related to the Web screen of the system (application) 411 under test after performing the operation corresponding to the test operation, and then processes for the next test operation. That is, the information processing device 101 performs a score calculation process based on the HTML information 412 and image information 415 related to the Web screen after the execution of the test operation, and the element name 405, attribute 406, and position 407 in the description data 403 for the next test operation to identify the element to be operated on in the next test operation, and then performs the operation according to the next test operation and executes the test of the system (application) 411 under test. If there is no next test operation, the information processing device 101 terminates the test of the system (application) 411 under test.

[0049] In other words, the information processing device 101 in this embodiment repeats the processing within the dashed frame shown in Figure 4A for each test operation, and each time it performs an operation corresponding to the test operation, it acquires HTML information 412 and image information 415 related to the Web screen of the system (application) 411 under test after the operation has been performed, and then performs the test for the next test operation. As a result, for each test operation, it becomes possible to perform a test on the Web screen of the system (application) 411 under test after the display content has changed or the screen has transitioned due to the previous test operation, and it can handle screens that can be dynamically changed by the test operation.

[0050] Next, the score calculation process (process 421 shown in Figure 4A) will be explained with reference to Figure 4B. In Figure 4B, the same components as those shown in Figure 4A are denoted by the same reference numerals.

[0051] In process 451, the information processing device 101 vectorizes the element names 405 related to the target of operation in the input test operation using a machine learning model that performs text embedding, and converts them into a vector representation 452 of the element names.

[0052] In process 453, the information processing device 101 filters the HTML information 414 for each element (screen component) and the image information 417 for each element (image element) of the input test target Web screen based on the attribute 406 related to the target of operation in the input test operation, and excludes information of elements whose attribute 406 does not match from the HTML information 414 and image information 417 for each element. The information processing device 101 classifies the attributes of each element on the Web screen based on the HTML information 414 and image information 417 for each element. Then, the information processing device 101 compares the obtained attributes of the elements on the Web screen with the attribute 406 related to the target of operation in the test operation for each element, and excludes the information of the element if the attributes do not match. The attributes of the elements on the Web screen may be classified using a rule-based classification method based on tag information, etc., or they may be classified using a machine learning model.

[0053] In process 454, the information processing device 101 vectorizes the HTML information 414 for each element after filtering in process 453 using a machine learning model that embeds text, and converts it into a vector representation 455 of the HTML information for each element. First, the information processing device 101 collects the descriptive information of the element in the HTML description (including the descriptive information of child elements if there are child elements) from the filtered HTML information 414 for each element, for example in a list format. The descriptive information collected here is the descriptive information that can be displayed as text information on the screen and specified as an element name. For example, information about "text", "placeholder", "name", "value", "aria-label", "title", "id", etc. of each element in the HTML description is collected as descriptive information. The example shown here is just one example and is not limited to this. Note that for child elements, descriptive information may be appropriately acquired depending on the distance from the parent element, etc. Also, for inputs, radio buttons, etc., neighboring descriptive information may also be collected. For example, if the HTML information 414 for each element is <input class="form-control gl-form-input bottom" required title="This field is required." data-qa-selector="password_field" type="password" name="user[password]" id="user_password"> In that case, ["This field is required.", "user[password]", "user_password"] This is collected as descriptive information.

[0054] Next, the information processing device 101 converts the collected descriptive information into a vector representation using a machine learning model that performs text embedding. For example, in the example ["This field is required.", "user[password]", "user_password"] mentioned above, "This field is required.", "user[password]", and "user_password" are each converted into a vector representation.

[0055] In process 456, the information processing device 101 vectorizes the element-specific image information 417 after filtering in process 453 using a machine learning model that performs image embedding, converting it into a vector representation 457 of the element-specific image information. Here, the machine learning model used in process 456 for image embedding is pre-trained to match the vector representation of the machine learning model that performs text embedding.

[0056] In process 458, the information processing device 101 matches the vector representation 452 of the element name with the vector representation 455 of the HTML information for each element and the vector representation 457 of the image information for each element to obtain a score for each element on the web screen under test that indicates the similarity to the target of operation in the test operation. The information processing device 101 obtains the score for each element as follows, for example: The information processing device 101 calculates the cosine similarity between the vector representation 452 of the element name and the vector representation 455 of the HTML information of the target element and saves it as a candidate score for the element. The information processing device 101 also calculates the cosine similarity between the vector representation 452 of the element name and the vector representation 457 of the image information of the target element, and saves the obtained cosine similarity corrected using a correction value as a candidate score for the element. Here, the correction value is a correction value that allows the vector representation of the HTML information and the vector representation of the image information to be compared on the same scale, and is assumed to be predetermined. The information processing device 101 then obtains the element with the highest similarity (or the one closest to 1 in the case of cosine similarity) from among the stored element score candidates, and outputs a score of 459 for each element. The above explanation uses cosine similarity, but it is not limited to this, and any index that can evaluate the similarity of each element can be applied. For example, the Levenshtein distance or Jalot-Winkler distance, which are indices that measure the similarity between strings, can be used, or the Pearson correlation coefficient, Euclidean distance or word carrying distance, which are indices that measure the similarity between vectors, can be used.

[0057] Process 460 is executed when the description of the target of operation includes information indicating a relative position, such as "the 'Password' input area above the 'Sign in' button." In process 460, the information processing device 101 obtains and outputs a score 461 for each element, similar to the above-mentioned process 458, for the element names of the elements used to indicate the relative position (hereinafter also referred to as relative elements), based on the position 407 of the target of operation in the input test operation. That is, the information processing device 101 obtains and outputs a score for each element on the web screen for the relative element, based on the vector representation of the element name of the relative element, the vector representation of the HTML information for each element on the web screen, and the vector representation of the image information for each element.

[0058] Process 462 is executed when the description of the object to be operated on includes location information. The information processing device 101 adjusts the element-specific score 459 obtained by process 458 according to the location and outputs the adjusted score as the element-specific score 422. If the description of the object to be operated on does not include location information, the information processing device 101 outputs the element-specific score 459 obtained by process 458 as the element-specific score 422. In process 462, if the description of the object to be operated on includes information indicating an absolute location, the information processing device 101 adjusts the element-specific score 459 according to the location description. For example, if the element-specific score is cosine similarity, the information processing device 101 adjusts the score by subtracting a value that increases as the distance from the described location increases. For example, if it is the "Password" input area on the right side of the screen, the information processing device 101 adjusts the element-specific score 459 by subtracting a value corresponding to the distance between the element and the right edge of the screen. Furthermore, if the description of the object to be manipulated includes information indicating relative position, the information processing device 101 adjusts the element-specific score 459 according to the element-specific score and position description of the relative element. For example, the information processing device 101 adds the relative element-specific score 461 to each of the element-specific scores 459 for the target element, and then adjusts the added score based on the distance between elements according to the position description. From the resulting score, the device outputs the element-specific score 422 that has the highest similarity for each element on the web screen. For example, if it is the "Password input area above the "Sign in" button", the information processing device 101 adds the relative element's element-specific score 461 for the element name "Sign in" to each of the element-specific scores 459 for the element name "Password". Furthermore, the information processing device 101 subtracts a value from the summed score that corresponds to the distance (which may be the difference in each coordinate) between the element that obtained an element-specific score of 459 and the element that obtained an element-specific score of 461, and sets the element-specific score that has the highest similarity for each element on the web screen as the element-specific score.

[0059] As described above, according to this embodiment, for each test operation divided from the test specification, the information of the target of operation in the test operation, the HTML information (screen configuration information) for each element on the screen of the test operation, and the image information for each element are converted into vectors using a machine learning model. The converted vectors are then compared to identify the element on the screen of the test operation that has the highest similarity to the target of operation in the test operation. By performing this processing, it becomes possible to uniquely identify the element of operation in the test operation, and tests corresponding to the test operation can be executed appropriately.

[0060] Furthermore, by obtaining HTML information (screen configuration information) and image information related to the screen under test after each test operation, and then executing the test for the next test operation, it becomes possible to appropriately perform tests in response to screens that may dynamically change due to test operations. In addition, by referring to the image information related to the screen under test when identifying the elements to be manipulated in a test operation, it becomes possible to identify elements that can only be specified visually.

[0061] In the explanation above, we described an example where the description of the target to be operated on in the test operation description data 403 consists of an element name 405, an attribute 406, and a position 407. However, the description of the target to be operated on only needs to include at least an element name 405. For example, if the description of the target to be operated on does not include an attribute 406, the processing (score calculation processing) 421 shown in Figure 4B can be simplified by performing processes 454 and 456 to vectorize the HTML information (screen configuration information) and image information for each element without performing the filtering in process 453. Also, for example, if the description of the target to be operated on does not include a position 407, the processing (score calculation processing) 421 shown in Figure 4B can be simplified by omitting processes 460 and 462, and outputting the score for each element obtained in process 458 as the score for each element 422. For example, if the description of the target to be operated on includes an element name 405 but does not include an attribute 406 or a position 407, the score calculation processing in process 421 can be implemented with the configuration shown in Figure 8. Figure 8 shows another example of the score calculation process, and the same components as those shown in Figure 4B are denoted by the same reference numerals.

[0062] <Second Embodiment> In the second embodiment described below, an overview of each test case is extracted from the test specification document that shows the content of the test, and the extracted overview of the test case is input into a machine learning model to perform inference using the machine learning model and generate a description of the test operation. Then, based on the description of the test operation generated by the machine learning model's inference, the test is executed on the system (application) under test in the same manner as in the first embodiment. The configuration of the information processing system and the hardware configuration of the information processing device in the second embodiment are the same as those in the first embodiment, so a description is omitted.

[0063] Figure 9 shows an example of the functional configuration of the information processing device 101 in the second embodiment. In Figure 9, components having the same function as those shown in Figure 3 are denoted by the same reference numerals, and redundant explanations are omitted. The information processing device 101 in the second embodiment includes a communication unit 301, a control unit 302, an input / output control unit 303, a storage unit 304, an acquisition unit 305, an extraction unit 901, a generation unit 902, a conversion unit 307, an arithmetic unit 308, a specific unit 309, and an execution unit 310.

[0064] The extraction unit 901 extracts an outline of each test case from the test specification document, which shows the content of the test to be performed, for each test case that defines a series of operations to be performed in the test. The extraction unit 901 divides the test content shown in the test specification document acquired by the acquisition unit 305 into test cases, and extracts an outline of each of the divided test cases. The outline of the test case may be modified by the test personnel or others according to the test operation description data generated based on the outline of the test case.

[0065] The generation unit 902 generates test operation description data for the test operations performed in a test case, based on the test case summary extracted by the extraction unit 901 and the screen configuration information and image information acquired by the acquisition unit 305. For example, the generation unit 902 inputs the test case summary, screen configuration information and image information, and a prompt including instructions to generate a description of the test operations into a machine learning model, performs inference using the machine learning model, and generates the test operation description data.

[0066] Here, the generation unit 902 performs inference using a trained machine learning model, such as a large-scale multimodal model (LMM), which is generated by machine learning (deep learning). A large-scale multimodal model (LMM) is a machine learning model constructed using a large amount of training data and deep learning technology. When data called a prompt, which indicates instructions, is input, it performs inference based on the prompt and generates and outputs data corresponding to the input prompt.

[0067] Furthermore, the generation unit 902 may generate descriptive data for test operations based on the test case overview and screen configuration information. In that case, if the input is only text data, inference may be performed using a Large Language Model (LLM), which is a language model constructed using a large amount of text data (such as a large corpus) and deep learning technology, rather than being limited to a Large Multimodal Model (LMM).

[0068] Similar to the first embodiment, the conversion unit 307 vectorizes information related to each element involved in the execution of the test using a machine learning model that embeds the respective information, and converts it into a vector representation (embedded representation, distributed representation). Specifically, based on the test operation description data generated by the generation unit 902, the conversion unit 307 converts the element names targeted in the test operation into vector representations using a machine learning model that embeds text. Furthermore, based on the screen configuration information acquired by the acquisition unit 305, the conversion unit 307 converts the screen configuration information for each element (screen component) that may be targeted in the execution of the test operation into a vector representation using a machine learning model that embeds text. In addition, based on the image information acquired by the acquisition unit 305, the conversion unit 307 converts the image information for each element of the image on the screen that may be targeted in the execution of the test operation into a vector representation using a machine learning model that embeds images.

[0069] In the example described above, the machine learning model may be configured to be installed on another information processing device that can communicate with the information processing device 101 via a network or the like. In this case, the information processing device 101 should send input to the machine learning model to the other information processing device 101 that has the machine learning model installed via a network interface 207 or the like, and receive the processing results for that input.

[0070] Referring to Figure 10, the processing in the information processing device 101 in the second embodiment will be described. Figure 10 is a diagram showing an example of processing in the information processing device 101 in the second embodiment. In Figure 10, elements identical to those shown in Figure 4A are denoted by the same reference numerals. In the following description, we will use the case where the screen configuration information for the system (application) under test is HTML information created in HTML as an example, but we are not limited to this and the explanation is also applicable to screen configuration information created using other languages.

[0071] The information processing device 101 acquires a test specification 401 that shows the test content, such as the operations to be performed (operation instructions) and the verification content (verification instructions) to be verified on the system (application) under test.

[0072] In process 1001, the information processing device 101 extracts a summary of each test case based on the test content shown in the acquired test specification 401. For example, the test specification 401 contains multiple test cases that define a series of operations to be performed in the test, and each test case contains multiple operations and verification items. The information processing device 101 divides the test content shown in the acquired test specification 401 into test cases and extracts a test case summary 1002 for each divided test case. An example of the extracted test case summary 1002 is shown in Figure 11. Figure 11 shows an example of a test case summary 1100 related to a test to confirm that the login process is completed successfully, and includes the screen to be tested, the operations to be performed in the test, and the verification items.

[0073] The process described below will be executed for each extracted test case summary (1002). In process 1003, the information processing device 101 generates and outputs test operation description data 1004 to be performed in the test case, based on the extracted test case summary 1002 and the HTML information (screen configuration information) 412 and image information 415 related to the Web screen (screen to be tested) on which the test is performed, obtained from the system (application) 411 under test. Note that the image information 415 is optional, and the information processing device 101 may generate and output the test operation description data 1004 based on the extracted test case summary 1002 and the HTML information (screen configuration information) 412 related to the Web screen (screen to be tested) on which the test is performed. In this process 1003, the information processing device 101 refers to the test operation description data 1004 already performed in the test case summary 1002 under processing and generates test operation description data 1004 for the next executable step (one test operation). The information processing device 101, for example, inputs the prompt 1200 shown in Figure 12 into the machine learning model, performs inference regarding the generation of a test operation description, and generates and outputs the description data 1004 for the next test operation.

[0074] Figure 12 illustrates an example of a prompt for generating a test action description. As shown in Figure 12, the prompt 1200 for generating a test action description includes a description of the test action generation 1210, a test case summary 1220, a description of previous test actions 1230, constraints for generating the test action description 1240, a description of the test action notation 1250, HTML information (screen configuration information) 1260, and image information 1270. The description of the test action generation 1210 is a description for executing the generation of test action description data. The test case summary 1220 is a summary of the test case to be processed, extracted from the test specification. The description of previous test actions 1230 is the description data of test actions that have already been executed. The constraints for generating the test action description 1240 are constraints when generating the test action description data, and include, for example, generating only one executable test action or prohibiting the generation of test actions that have already been executed. The explanation of the test operation notation 1250 describes the format for describing test operation data, for example, describing the operation procedure in the format "operation type 'target'", describing the target of the operation with an element name and a specified attribute, and specifying the element name with text that exists on the screen. The HTML information (screen configuration information) 1260 is the HTML information (screen configuration information) related to the web screen (test target screen) on which the test is executed. The image information 1270 is the image information related to the web screen (test target screen) on which the test is executed.

[0075] Specific examples of prompts for generating test operation descriptions are shown in Figures 13A and 13B. In Figures 13A and 13B, element 1310 corresponds to the explanation of test operation description generation 1210, element 1320 corresponds to the test case summary 1220, and element 1330 corresponds to the previous test operation description 1230. Element 1340 corresponds to the constraints 1240 when generating test operation descriptions, element 1350 corresponds to the explanation of the test operation notation 1250, and element 1360 is the part that prompts the machine learning model for output. Note that in the examples shown in Figures 13A and 13B, the elements corresponding to HTML information (screen configuration information) 1260 and image information 1270, respectively, are omitted from the illustration.

[0076] Returning to Figure 10, in process 404, the information processing device 101 uses a parser to decompose the description of the target of operation in the test operation description data 1004 generated in process 1003 into element name 405, attribute 406, and position 407. The element name 405, attribute 406, and position 407 obtained by decomposing the test operation description data 1004 by the parser are input to the score calculation process 421. Other processes (processes after the score calculation process) are the same as in the first embodiment, so their explanation is omitted.

[0077] The following describes a specific example of the process explained with reference to Figure 10, where the information processing device 101 performs processing based on, for example, the test case overview 1100 shown in Figure 11. In the following description, the information processing device 101 is described as inputting the prompt 1200 shown in Figure 12 into a machine learning model to perform inference regarding the generation of a test operation description, and then generating and outputting test operation description data.

[0078] In the first process 1003 performed after starting the processing for test case summary 1100, the information processing device 101 inputs the prompt 1200 shown in Figure 12 into the machine learning model, with test case summary 1220 set to test case summary 1100 shown in Figure 11, and the description of the test operation so far 1230 set to empty data (Null, or information indicating that it has not been performed). This generates, for example, the description data 1401 of a test operation, shown as number 1 in Figure 14, where the operation type is goto and the target of the operation is "URL / login". Based on this description data 1401 of the test operation, the information processing device 101 executes the operation to open the "URL / login" screen as a test operation.

[0079] In the second process 1003, after executing an action corresponding to the test action description data 1401, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case summary 1220 is set to the test case summary 1100 shown in Figure 11, and the description of the previous test actions 1230 is set to the test action description data 1401. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen obtained from the system (application) under test. As a result, for example, the test action description data 1402 shown as number 2 in Figure 14, where the operation type is input, the target of the operation is the "Username or email" input area, and the input text is root, is generated. Based on this test action description data 1402, the information processing device 101 performs score calculation processing and target element identification processing, and executes the operation of entering root as the username in the "Username or email" input area of ​​the "URL / login" screen as a test action.

[0080] In the third process 1003, after executing the action corresponding to the test action description data 1402, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case overview 1220 is set to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test actions 1230 are set to test action description data 1401 and 1402. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen obtained from the system (application) under test after executing the action corresponding to the test action description data 1402. As a result, for example, the test action description data 1403 shown as number 3 in Figure 14, where the operation type is input, the target of the operation is the "Password" input area, and the input text is password, is generated. Based on the test operation description data 1403, the information processing device 101 performs score calculation processing and processing to identify the element to be operated on, and as a test operation, it enters "password" as the password into the "Password" input area of ​​the "URL / login" screen.

[0081] In the processing 1003 after executing the action corresponding to the test action description data 1403, the information processing device 101 inputs a prompt 1200 into the machine learning model, where the test case summary 1220 is set to the test case summary 1100 shown in Figure 11, and the descriptions of the previous test actions 1230 are set to test action description data 1401-1403. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, of the HTML information and image information related to the "URL / login" screen obtained from the system (application) under test after executing the action corresponding to the test action description data 1403. As a result, for example, the test action description data 1404 shown as number 4 in Figure 14, where the operation type is click and the target of the operation is the "Sign in" button, is generated. Based on this test action description data 1404, the information processing device 101 performs score calculation processing and target element identification processing, and executes the operation of clicking the "Sign in" button on the "URL / login" screen as a test action.

[0082] In the processing 1003 after executing the action corresponding to the test action description data 1404, the information processing device 101 inputs a prompt 1200 to the machine learning model, where the test case summary 1220 is set to the test case summary 1100 shown in Figure 11, and the descriptions of the previous test actions 1230 are set to test action description data 1401 to 1404. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, which are HTML information and image information related to the screen obtained from the system (application) under test after executing the action corresponding to the test action description data 1404. As a result, for example, the test action description data 1405, shown as number 5 in Figure 14, with operation type assert_url and operation target "URL", is generated. Based on this test action description data 1405, the information processing device 101 confirms that the screen is a URL as a test action.

[0083] In the processing 1003 after executing the operation corresponding to the test operation description data 1405, the information processing device 101 inputs a prompt 1200 to the machine learning model, setting the test case overview 1220 to the test case overview 1100 shown in Figure 11, and the descriptions of the previous test operations 1230 to the test operation description data 1401-1405. In this prompt 1200, HTML information (screen configuration information) 1260 and image information 1270 are input, respectively, as HTML information and image information related to the screen obtained from the system (application) under test after executing the operation corresponding to the test operation description data 1405. As a result, for example, the test operation description data 1406, with operation type "end operations", shown as number 6 in Figure 14, is generated. Based on this test operation description data 1406, the information processing device 101 terminates processing, considering that the test operations to be performed in the test case are complete. If there are other test case overviews that have not yet been performed, processing for those test case overviews is started.

[0084] In the process described above, the test behavior description data generated by the machine learning model based on the test case overview may be inappropriate, potentially causing problems (errors) when attempting to execute the test behavior. For example, possible problems (errors) include: • The parser cannot properly decompose the test operation description data (for example, it should be written as "Password input area", but it is written as "area to enter "Password", which is a notational error). • The element to be manipulated is unknown (the element has a low score, or the element cannot be found). • The element to be manipulated cannot be uniquely identified. These are some examples.

[0085] When such a problem occurs, the information processing device 101 executes process 1003 by referring to the test operation description data in which the problem (error) occurred and the error message indicating the problem (error) that occurred, and regenerates (corrects) the test operation description data. For example, the information processing device 101 inputs the prompt 1500 shown in Figure 15 into a machine learning model to perform inference regarding the generation of the test operation description, and regenerates (corrects) the test operation description data and outputs it.

[0086] Figure 15 illustrates an example of a prompt related to the generation (modification) of a test operation description. As shown in Figure 15, the prompt 1500 for generating a test operation description includes a description of the test operation description generation 1510, a test case summary 1520, previous test operation descriptions 1530, constraints on generating the test operation description 1540, a description of the test operation notation 1550, the operation, error message, and solution to be modified 1560, HTML information (screen configuration information) 1570, and image information 1580. The description of the test operation description generation 1510 is a description to execute the regeneration (modification) of the test operation description data. The test case summary 1520, the description of previous test operations 1530, the constraints for generating the test operation description 1540, and the explanation of the test operation notation 1550 are the same as the test case summary 1220, the description of previous test operations 1230, the constraints for generating the test operation description 1240, and the explanation of the test operation notation 1250 for prompt 1200 shown in Figure 12. The operation to be corrected, error message, and solution 1560 are the description data of the test operation in which the problem (error) occurred, the error message indicating the problem (error) that occurred, and the suggestion of how to resolve the problem (error). The HTML information (screen configuration information) 1570 and image information 1580 are the HTML information (screen configuration information) and image information related to the screen under test.

[0087] Specific examples of prompts for generating (correcting) test operation descriptions are shown in Figures 16A and 16B. Figures 16A and 16B show examples of prompts for generating (correcting) test operation descriptions when the parser cannot properly decompose the test operation description data and a problem (error) occurs during parsing. In Figures 16A and 16B, element 1610 corresponds to the explanation of test operation description generation 1510, element 1620 corresponds to the test case summary 1520, and element 1630 corresponds to the previous test operation description 1530. Element 1640 corresponds to the constraints when generating test operation descriptions 1540, element 1650 corresponds to the explanation of test operation notation 1550, and element 1660 corresponds to the operation, error message, and solution to be corrected 1560. Element 1570 is the part that prompts the machine learning model for output. Note that in the examples shown in Figures 16A and 16B, the elements corresponding to HTML information (screen configuration information) 1570 and image information 1580, respectively, are omitted from the illustration.

[0088] If the element to be operated on is unknown and a problem (error) occurs, element 1660 enclosed by a dashed line in Figure 16B (corresponding to the operation to be corrected, error message, and solution 1560) should be replaced with element 1760 as shown in Figure 17. Furthermore, if the element to be operated on cannot be uniquely identified and a problem (error) occurs, element 1660 enclosed by a dashed line in Figure 16B (corresponding to the operation to be corrected, error message, and solution 1560) should be replaced with element 1860 as shown in Figures 18A and 18B.

[0089] If a problem (error) occurs when attempting to execute a test based on the generated test operation description data, the test operation description data can be regenerated (corrected) in this way, making it possible to generate highly accurate test operation description data. Furthermore, by switching prompts according to the problem (error) that occurred and regenerating (correcting) the test operation description data, it is possible to appropriately correct the test operation description data according to the problem (error) that occurred. Note that the test operation description data before correction that caused the problem (error) will be deleted, but the test operation description data before correction may be saved separately to be useful in generating test operation descriptions.

[0090] According to the second embodiment, similar to the first embodiment, it becomes possible to uniquely identify the element being manipulated in the test operation, and to appropriately execute tests corresponding to the test operation. Furthermore, by acquiring information about the screen after the operation is executed and executing tests related to the next test operation, it becomes possible to appropriately execute tests in response to screens that may be dynamically changed by the test operation.

[0091] It should be noted that the embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0092] The following devices and methods are also included in the disclosure of this embodiment. (1) A test specification document that shows the content of the test, and means for acquiring screen configuration information and image information related to the screen from the test target, An extraction means for extracting test operations based on the test content shown in the aforementioned test specification, A transformation means that converts the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that embeds each of these, An information processing apparatus characterized by having a means for identifying the element to be operated on in the test operation from among the elements on the screen, based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors. (2) The information processing apparatus according to (1), wherein the extraction means divides the content of the test shown in the test specification into a plurality of test operations, and generates test operation description data for each divided test operation by converting the test operation according to a predetermined notation. (3) The information processing apparatus according to (1), characterized in that the extraction means extracts an outline of a test for each test content shown in the test specification, and uses the extracted outline of a test to perform inference using a machine learning model to generate descriptive data of the test operation. (4) An information processing apparatus according to any one of (1) to (3), further comprising execution means for performing the test operation on the specified element to be operated on. (5) The information processing device according to any one of (1) to (4), characterized in that the acquisition means acquires the screen configuration information and the image information each time the test operation is performed. (6) The information processing apparatus according to any one of (1) to (5), wherein the identifying means obtains a similarity to the target for operation for each element on the screen based on the information of the target for operation converted into a vector, the screen configuration information, and the image information, and identifies the element with the highest similarity as the target for operation in the test operation. (7) The description of the object to be operated on in the description data of the test operation includes the element name, The information processing apparatus according to (2) or (3), wherein the conversion means is characterized by converting the element names into vectors using the machine learning model. (8) The description of the object to be operated in the description data of the test operation further includes a description indicating the position of the object to be operated, The information processing apparatus according to (7), wherein the identifying means identifies elements of the target to be operated in the test operation based on the information of the target to be operated, the screen configuration information, and the image information converted into vectors, and the described position of the target to be operated. (9) The specified means is Based on the operation target information, screen configuration information, and image information converted into vectors, the similarity to the operation target is obtained for each element on the screen. The similarity of the elements on the screen obtained is adjusted according to the position of the described target of operation. The information processing device according to (8), characterized in that it identifies the element with the highest similarity after adjustment as the element to be operated on in the test operation. (10) The information processing apparatus according to (8) or (9), characterized in that the position of the object to be operated is an absolute position on the screen. (11) The information processing apparatus according to (8) or (9), characterized in that the position of the object to be operated is a relative position to an element on the screen. (12) The description of the object to be operated in the test operation description data further includes a description indicating the attributes of the object to be operated, The information processing apparatus according to any one of (7) to (11), characterized in that the conversion means converts the screen configuration information for each element on the screen and the image information for each element on the screen into vectors, excluding elements whose attributes do not match those of the target of the operation. (13) The information processing apparatus according to (3), wherein if a problem occurs when executing a test operation based on the generated test operation description data, the extraction means refers to the problem that occurred, performs inference by a machine learning model, and modifies the test operation description data. (14) An information processing method performed by an information processing device, A test specification document outlining the test content, and an acquisition process for obtaining screen configuration information and image information related to the screen from the test target, An extraction step of extracting test actions based on the test content shown in the aforementioned test specification, A transformation step in which the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen are converted into vectors using a machine learning model that embeds them, An information processing method characterized by comprising a selection step of identifying the element to be operated on in the test operation from among the elements on the screen based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors. (15) To the computer, A test specification document outlining the test content, and an acquisition step to obtain screen configuration information and image information related to the screen from the test target, An extraction step to extract test actions based on the test content shown in the aforementioned test specification, A transformation step in which the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen are converted into vectors using a machine learning model that embeds them, A program (computer program product) for causing a program to perform a selection step of identifying the element to be operated on in the test operation from among the elements on the screen, based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors. (16) To the computer, A test specification document outlining the test content, and an acquisition step to obtain screen configuration information and image information related to the screen from the test target, An extraction step to extract test actions based on the test content shown in the aforementioned test specification, A transformation step in which the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen are converted into vectors using a machine learning model that embeds them, A computer-readable recording medium characterized by recording a program for executing a specific step of identifying the element to be operated on in the test operation from among the elements on the screen, based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors. [Explanation of symbols]

[0093] 101 Information Processing Device 102 Server Equipment 201 CPU 202 ROM 203 RAM 204 Auxiliary storage device 205 Output device 206 Input device 207 Network I / F 301 Communications Department 302 Control Unit 303 Input / Output Control Unit 304 Storage section 305 Acquisition Department 306 Split section 307 Conversion Unit 308 Arithmetic section 309 Specific section 310 Execution Unit 901 Extraction part 902 Generation part

Claims

1. A test specification document outlining the test content, and a means for acquiring screen configuration information and image information related to the screen from the test target. An extraction means for extracting test operations based on the test content shown in the aforementioned test specification, A transformation means that converts the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen into vectors using a machine learning model that embeds each of these, An information processing apparatus characterized by having a means for identifying the element to be operated on in the test operation from among the elements on the screen, based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors.

2. The information processing apparatus according to claim 1, characterized in that the extraction means divides the content of the test shown in the test specification into a plurality of test operations, and generates test operation description data for each of the divided test operations by converting the test operation according to a predetermined notation.

3. The information processing apparatus according to claim 1, characterized in that the extraction means extracts an outline of the test for each test content shown in the test specification, and uses the extracted outline of the test to perform inference using a machine learning model to generate descriptive data of the test operation.

4. The information processing apparatus according to claim 1, further comprising execution means for performing the test operation on the specified element to be operated on.

5. The information processing apparatus according to claim 1, characterized in that the acquisition means acquires the screen configuration information and the image information each time the test operation is performed.

6. The information processing apparatus according to claim 1, characterized in that the identifying means obtains a similarity to the target of operation for each element on the screen based on the information of the target of operation converted into a vector, the screen configuration information, and the image information, and identifies the element with the highest similarity as the target of operation in the test operation.

7. The description of the target of the operation in the aforementioned test operation description data includes the element name. The information processing apparatus according to claim 2 or 3, characterized in that the conversion means converts the element names into vectors using the machine learning model.

8. The description of the object being operated on in the test operation description data further includes a description indicating the position of the object being operated on. The information processing apparatus according to claim 7, wherein the identifying means identifies elements of the target to be operated in the test operation based on the information of the target to be operated, the screen configuration information, and the image information converted into vectors, and the described position of the target to be operated.

9. The aforementioned specifying means is, Based on the operation target information, screen configuration information, and image information converted into vectors, the similarity to the operation target is obtained for each element on the screen. The similarity of the elements on the screen obtained is adjusted according to the position of the described target of operation. The information processing apparatus according to claim 8, characterized in that it identifies the element with the highest similarity after adjustment as the element to be operated on in the test operation.

10. The information processing apparatus according to claim 8, characterized in that the position of the object to be operated on is an absolute position on the screen.

11. The information processing apparatus according to claim 8, characterized in that the position of the object to be operated on is a relative position to the element on the screen.

12. The description of the target of the operation in the test operation description data further includes a description indicating the attributes of the target of the operation, The information processing apparatus according to claim 7, characterized in that the conversion means converts the screen configuration information for each element on the screen and the image information for each element on the screen into vectors, excluding elements whose attributes do not match those of the target of the operation.

13. The information processing apparatus according to claim 3, wherein the extraction means, when a problem occurs during the execution of a test operation based on the generated test operation description data, refers to the problem that occurred, performs inference using a machine learning model, and modifies the test operation description data.

14. An information processing method performed by an information processing device, A test specification document outlining the test content, and an acquisition process for obtaining screen configuration information and image information related to the screen from the test target, An extraction step of extracting test actions based on the test content shown in the aforementioned test specification, A transformation step in which the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen are converted into vectors using a machine learning model that embeds them, An information processing method characterized by comprising a selection step of identifying the element to be operated on in the test operation from among the elements on the screen based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors.

15. On the computer, A test specification document outlining the test content, and an acquisition step to obtain screen configuration information and image information related to the screen from the test target, An extraction step to extract test actions based on the test content shown in the aforementioned test specification, A transformation step in which the information of the target of operation in the test operation, the screen configuration information for each element on the screen, and the image information for each element on the screen are converted into vectors using a machine learning model that embeds them, A program for performing a selection step of identifying the element to be operated on in the test operation from among the elements on the screen, based on the information of the target to be operated on, the screen configuration information, and the image information, which have been converted into vectors.

Images