Program, information processing method, and information processing device

Abstract

To provide a program, an information processing method, and an information processing device for efficiently outputting an execution script for testing an application.SOLUTION: In a test automation system in which an information processing device (test automation server) and an application server to be tested are communicably connected via the Internet, LAN, or the like, the information processing device obtains screen information of the application server to be tested, obtains test information of the application server, obtains an element image corresponding to an operation target object included in the obtained screen information, specifies an operation image indicating the operation target object based on the obtained element image, deriving a position of the operation image on a screen of the application server based on the specified operation image, and outputs an execution script for performing a test of the application server based on the derived position of the operation image and the test information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] There are known devices and programs for generating test specifications for computer programs (for example, Patent Document 1). According to the device described in Patent Document 1, when selecting factors and levels, the selection probability of factors selected together with a certain factor and the probability of a certain level being selected within a certain factor are displayed based on the factors and levels selected in past test specification designs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-125420 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the device and program described in Patent Document 1 have a problem in that they do not take into consideration the efficient output of an execution script for testing an application based on the screen information of the application to be tested and the element image corresponding to the object to be operated contained in the screen information.

[0005] The present disclosure has been made in consideration of such circumstances, and aims to provide a program or the like that can efficiently output an execution script for testing an application based on screen information of the application to be tested and an element image corresponding to an object to be operated that is included in the screen information. [Means for solving the problem]

[0006] In one proposal, the program causes a computer to execute a process of acquiring screen information of an application to be tested, acquiring test information for the application, acquiring an element image corresponding to an object to be operated that is included in the acquired screen information, identifying an operation image indicating the object to be operated based on the acquired element image, deriving the position of the operation image on the screen of the application based on the identified operation image, and outputting an execution script for testing the application based on the derived position of the operation image and the test information.

[0007] In one proposal, the information processing method causes a computer to execute a process of acquiring screen information of an application to be tested, acquiring test information for the application, acquiring an element image corresponding to an object to be operated included in the acquired screen information, identifying an operation image indicating the object to be operated based on the acquired element image, deriving a position of the operation image on the screen of the application based on the identified operation image, and outputting an execution script for testing the application based on the derived position of the operation image and the test information.

[0008] In one proposal, the information processing device includes an acquisition unit that acquires screen information of an application to be tested, acquires test information for the application, and acquires an element image corresponding to an object to be operated included in the acquired screen information; an identification unit that identifies an operation image indicating the object to be operated based on the acquired element image; a derivation unit that derives the position of the operation image on the screen of the application based on the identified operation image; and an output unit that outputs an execution script for testing the application based on the derived position of the operation image and the test information. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide a program or the like that efficiently outputs an execution script for testing an application based on screen information of the application to be tested and an element image corresponding to an object to be operated that is included in the screen information. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a system overview diagram illustrating a test automation system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram illustrating a configuration example of an information processing device included in a test automation system. [Figure 3] 10 is an explanatory diagram showing an example of a correspondence between an application screen including an operation image and an element image; FIG. [Figure 4] 10 is a flowchart illustrating a processing procedure performed by a control unit of an information processing device. [Figure 5] FIG. 10 is an explanatory diagram showing an example of processing (template matching) related to a Gaussian pyramid. [Figure 6] FIG. 10 is an explanatory diagram showing an example of processing related to a Gaussian pyramid (extraction in an original image). [Figure 7] FIG. 10 is an explanatory diagram showing an example of a process related to application of phase-only correlation. [Figure 8] 10 is a flowchart illustrating a processing procedure by a control unit of an information processing device according to a second embodiment (selection screen). [Figure 9] FIG. 10 is an explanatory diagram showing an example of a selection screen. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Embodiment 1) The present invention will be described in detail below with reference to the drawings illustrating embodiments thereof. FIG. 1 is a system overview diagram illustrating a test automation system S according to the first embodiment. FIG. 2 is a block diagram showing an example configuration of an information processing device 1 included in the test automation system S. The test automation system S is configured with the information processing device 1 as a main device (server), and the information processing device 1 is communicatively connected to an application server AS or the like via, for example, the Internet or a LAN. The application server AS is a web server or the like that provides an application to be tested. The information processing device 1 functions as a test automation server that generates and executes execution scripts for automatically executing tests on the application to be tested, and stores and manages various data related to the execution scripts.

[0012] The information processing device 1 acquires screen information of the application to be tested and test information that defines information related to the test, and generates an execution script for testing the application using the screen information and test information. In this embodiment, the information processing device 1 (test automation server) and the application server AS (web server that provides the application to be tested) have been described as separate devices, but this is not limiting, and it goes without saying that these server functions may be integrated into a single device in a server configuration.

[0013] The information processing device 1 includes a control unit 11, a storage unit 12, a communication unit 13, and an input / output I / F 14. The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc., and performs various information processing, control processing, etc. related to the information processing device 1 by reading and executing a program P (program product) stored in the storage unit 12.

[0014] The storage unit 12 includes a volatile storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and a non-volatile storage area such as an EEPROM or a hard disk. The storage unit 12 pre-stores a program P (program product) and data to be referenced during processing. The program P (program product) stored in the storage unit 12 may be a program P (program product) read from a recording medium 121 readable by the information processing device 1. Alternatively, the program P (program product) may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 12.

[0015] The communication unit 13 is a communication interface for communicating with an application server AS or a development management server (not shown) that manages development information such as application source files, via a LAN within the facility or the Internet, either wired or wirelessly.

[0016] The input / output I / F 14 is an interface for inputting and outputting data to the control unit 11 of the information processing device 1, such as a USB, and is connected to a display unit 141, such as an LCD display, and an input unit 142, such as a keyboard and a mouse.

[0017] 3 is an explanatory diagram showing an example of correspondence between an application screen including an operation image and an element image. The control unit 11 of the information processing device 1 acquires screen information of the application and an element image corresponding to an object to be operated as input factors for performing image recognition processing.

[0018] The application screen information is, for example, a URL indicating the website of the web application to be tested. Alternatively, the application screen information may be an application screen (screenshot image) of an operation window in which the web application is executed. The application screen (screenshot image) may be, for example, stored as an image file in the storage unit 12 of the information processing device 1. Alternatively, the application screen information may be code data expressed in a structured language such as HTML that is displayed on a browser when the web application is executed. When the control unit 11 of the information processing device 1 acquires a URL, HTML, or the like as the application screen information, the control unit 11 may perform a screen capture of the operation window that is displayed by accessing the URL, etc., and generate the application screen (screenshot image).

[0019] In the application screen (screenshot image), one or more operation target objects that are the targets of operation tests are arranged to perform test automation. The operation target objects are, for example, icons, function buttons, or link images (images with link settings, etc.), and the application screen includes operation images that indicate the operation target objects. In other words, test automation can be achieved by performing operations such as clicking or entering characters on the operation images that indicate the operation target objects using macro processing, etc.

[0020] The operation image is arranged at an arbitrary position on the application screen. The operation image arranged at an arbitrary position on the application screen is, for example, defined or set as an operation target object to be pressed or selected by a user clicking a mouse or the like, and is associated or associated with code written in a script language (for example, JavaScript, etc.). By clicking (pressing) an operation image arranged at an arbitrary position on the application screen with a mouse or the like, the code (script) associated with the operation image is executed, and, for example, a process of displaying an enlarged image of the operation image or a process of redirecting to another page is performed.

[0021] By acquiring the application screen (screenshot image), the control unit 11 of the information processing device 1 can define planar coordinates (client coordinates) on the application screen and identify icons, function buttons, image display areas, etc. arranged on the application screen using the planar coordinates. Therefore, the control unit 11 of the information processing device 1 can calculate the position (area in client coordinates) and size of an operation image indicating an object to be operated on the application screen. As will be described in detail later, the control unit 11 of the information processing device 1 uses the acquired application screen (screenshot image) as original data (level 0) and performs a reduction process on the original data using a Gaussian filter with gradually varying resolutions to generate a Gaussian pyramid (a group of application screens).

[0022] The element image corresponding to the operation target object is an image (in this embodiment, an image of a click button) corresponding to an icon, function button, or link image (an image with a link setting, etc.) that is the target of an operation test during test automation. The element image is an image obtained by cutting out an area of ​​the operation target object that is the target of the operation test from the application screen that is the target of the test, and may be stored in the storage unit 12 of the information processing device 1 as an image file, for example. That is, the element image is substantially the same image as the element (operation target object) that is the target of detection on the application screen, and when the element image is stored in the storage unit 12, it may be associated with the element name of the element (operation target object) that is the target of detection. In this embodiment, the element image is an image defined in HTML (button tag) and displayed on the browser. However, it goes without saying that the element image may also be a link image using, for example, PNG. As will be described in detail later, the control unit 11 of the information processing device 1 generates multiple element images of different sizes by enlarging or reducing the acquired element image. These multiple element images of different sizes constitute an element image group.

[0023] The control unit 11 of the information processing device 1 performs a comparison process or a comparison operation process between an application screen (source image), which is a screenshot image of an operation window in which an application is executed, and an element image (template image) corresponding to an operation target object. As a result, the control unit 11 of the information processing device 1 extracts or identifies an operation image (an area of ​​the operation image indicating an operation target object) that is substantially the same image as the element image (template image) on the application screen (source image). When extracting the operation image (operation image indicating an operation target object), the control unit 11 of the information processing device 1 may derive the position (client coordinates, etc.) and size on the application screen.

[0024] The control unit 11 of the information processing device 1 may identify, as an operation image, a comparison area whose degree of similarity (similarity) is equal to or greater than a threshold, based on the degree of similarity (similarity) between each partitioned area (comparison area) in the application screen (source image) and an element image (template image). In this case, the control unit 11 of the information processing device 1 may compare or contrast each image included in the application screen group and the element image group in a brute-force manner. The control unit 11 of the information processing device 1 may highlight the identified operation image (operation image indicating an object to be operated) on the application screen. The control unit 11 of the information processing device 1 uses the position (client coordinates, etc.) of the operation image (operation image indicating an object to be operated) derived in this manner, etc., to generate and output an execution script for automatically executing a test on the application to be tested.

[0025] 4 is a flowchart illustrating a processing procedure performed by the control unit 11 of the information processing device 1. The control unit 11 of the information processing device 1 acquires screen information of an application to be tested (S101). The control unit 11 of the information processing device 1 acquires screen information of the application to be tested and test information of the application, for example, from an application server AS to be tested or a development management server that manages development information of the application. The screen information of the application may include, for example, a URL indicating the website of the web application to be tested, an application screen (screenshot image) of an operation window in which the application is executed, or code data expressed in a structured language such as HTML that is displayed on a browser when the web application is executed.

[0026] The control unit 11 of the information processing device 1 acquires an element image corresponding to the operation target object (S102). The control unit 11 of the information processing device 1 acquires the element image (image file) from a development management server that manages development information of the application, or input from a keyboard or the like via the input / output I / F 14 or the like, and stores it in the storage unit 12. The element image (image file) may be generated as an image file such as png or jpeg by cutting out an area of ​​the operation target object that is the subject of the operation test.

[0027] The control unit 11 of the information processing device 1 derives the position and size of the operation image indicating the object to be operated based on the acquired element image (S103). The control unit 11 of the information processing device 1 identifies the position (area on the application screen) of the operation image (image of the UI element) indicating the object to be operated (UI element) by comparing an application screen (source image), which is a screenshot image of an operation window in which an application is executed, with an element image (template image) corresponding to the object to be operated (UI element). When performing this comparison process, the control unit 11 of the information processing device 1 is not limited to comparing a single application screen (source image) with an element image (template image), but may also compare image groups (application screen group, element image group) including each of these images.

[0028] The application screen group (source image group) includes a plurality of application screens (screenshot images) with gradually varying resolutions and sizes, created using a plurality of Gaussian filters with gradually varying resolutions (blurring levels). The element image group (template image group) includes a plurality of element images with different sizes, which are obtained by enlarging or reducing the acquired element image.

[0029] 5 is an explanatory diagram showing an example of processing (template matching) related to a Gaussian pyramid. The control unit 11 of the information processing device 1 applies a Gaussian filter to an original image (a screenshot image that is a screen capture of an operation window in which an application is executed) so that the resolution (degree of blurring) varies gradually. As a result, the control unit 11 of the information processing device 1 generates a group of application screens (a group of source images) that are a Gaussian pyramid including a plurality of screenshot images with gradually varying resolutions.

[0030] The level of an application screen included in the application screen group is defined according to the resolution, with the original image (a screenshot image, which is a screen capture of the operation window in which the application is executed) being defined as level 0. The gradually varying resolutions are expressed, for example, as 1 / (2^n) (n is a power and an integer (level) greater than or equal to 1), and increasing the level by one reduces the resolution by half. In the illustration of this embodiment, levels 0 (original image) to 4 (image with 1 / 16 resolution) are illustrated as examples, and the Gaussian pyramid (application screen group) may further include images at level 5 or higher. Using the Gaussian pyramid in this way, a source image group (application screen group) can be generated from screenshot images of a website (site page of the application to be tested). The control unit 11 of the information processing device 1 stores the application screen group (source image group) generated in this way in the storage unit 12.

[0031] The control unit 11 of the information processing device 1 generates an element image group (template image group) including a plurality of element images (template images) of different sizes generated from the acquired element images (images of UI elements), i.e., by enlarging or reducing the element images. The control unit 11 of the information processing device 1 stores the element image group (template image group) generated in this manner in the storage unit 12.

[0032] The control unit 11 of the information processing device 1 compares (template matching) each application screen (source image) included in the application screen group (source image group) with each element image (template image) included in the element image group (template image group). The control unit 11 of the information processing device 1 may perform template matching using an image processing library such as OpenCV [cv2.matchTemplate function]. By performing template matching, it is possible to search for a rectangular area on the source image that matches the template image, which is a small image, within the source image, which is a large image.

[0033] When performing the comparison process, the control unit 11 of the information processing device 1 may compare each of the images included in the application screen group and the element image group in a brute force manner (performing template matching for all combinations). When performing the template matching, the control unit 11 of the information processing device 1 may identify, as an operation image, a comparison area whose degree of match (similarity) is equal to or greater than a threshold, based on the degree of match (similarity) between each area (comparison area) partitioned in the application screen (screenshot image). By performing the template matching, the control unit 11 of the information processing device 1 extracts an area (operation image) that matches an element image in the application screen (screenshot image) of any level (level: 1).

[0034] FIG. 6 is an explanatory diagram showing an example of processing related to a Gaussian pyramid (extraction in an original image). Based on the position and size of an operation image (area showing UI elements) extracted from an application screen (matching screen) at any level (level: l) in the Gaussian pyramid as a result of template matching, the control unit 11 of the information processing device 1 calculates (derives) the position and size of an operation image (area showing UI elements) in an application screen at a level (l-1) that is one level lower than the matching screen (level: l). In this way, the control unit 11 of the information processing device 1 recursively repeats the matching process of the comparison target areas (operation images) in two screenshot images (levels: m, m-1) with different resolutions. This allows the position and size of the operation image (area showing UI elements) in the application screen (screen capture of the operation window in which the application is executed), which is the original data (level 0) when using a Gaussian filter, to be calculated.

[0035] 7 is an explanatory diagram showing an example of processing related to application of phase-only correlation. As a confirmation process for an operation image (area indicating a UI element) identified on an application screen that is the original data (level 0), the control unit 11 of the information processing device 1 may use phase-only correlation to compare a rectangular area indicated by the operation image (area indicating a UI element) with an element image enlarged or reduced to the same size as the rectangular area. By applying phase-only correlation, the control unit 11 of the information processing device 1 can efficiently confirm whether the position and size of the extracted operation image (area indicating a UI element) are correct.

[0036] By using phase-only correlation in this way, the amount of translation of the second image relative to the first image can be calculated using the phase components obtained by performing a discrete Fourier transform on each of the two images, and a value that serves as an index of similarity can also be obtained when calculating the amount of translation.By performing the above-described series of processes, the position and size of the operation image (UI element image) can be efficiently determined (specified) even if the operation image (UI element image) is embedded in an application screen (web page) while being enlarged or reduced.

[0037] If the determination result by the phase-only correlation method is positive, the control unit 11 of the information processing device 1 executes post-processing such as generating an execution script using the position of the extracted operation image (area indicating a UI element), etc. If the determination result by the phase-only correlation method is negative, the control unit 11 of the information processing device 1 may continue the process of extracting a comparison target area where the degree of match is equal to or greater than a threshold by again performing a comparison process (template matching) between the element image group and the application screen group.

[0038] The control unit 11 of the information processing device 1 acquires test information for the application (S104). The control unit 11 of the information processing device 1 acquires test information from a development management server that manages development information for the application, or from test information input from a keyboard or the like via the input / output I / F 14 or the like, and stores the test information in the storage unit 12. The test information includes information for performing test automation, such as clicking an operation image (object to be operated), or inputting text or the like if the operation image indicates an input field.

[0039] The control unit 11 of the information processing device 1 generates an execution script for testing the application based on the position of the operation image and the test information (S105). The control unit 11 of the information processing device 1 uses the position of the operation image on the application screen (client coordinates, etc.) and the test information (operation content for the operation image) to generate an execution script for performing operations such as clicking or character input for the operation image by macro processing, etc. The control unit 11 of the information processing device 1 outputs the generated execution script and stores it in the storage unit 12.

[0040] The control unit 11 of the information processing device 1 may extract, from screen information defined in HTML or the like, an element path (xpath) that uniquely identifies an object to be operated in the operation image, using the position of the operation image derived according to the degree of match with the element image. The control unit 11 of the information processing device 1 may generate an execution script using the element path (xpath) identified based on the position of the operation image.

[0041] The execution script generation process may be performed using the process described in Japanese Patent Application Laid-Open No. 2023-75780, for example. That is, the control unit 11 of the information processing device 1 may invoke all of the processing matters disclosed in Japanese Patent Application Laid-Open No. 2023-75780 and execute various processes related to the generation of the execution script.

[0042] The control unit 11 of the information processing device 1 executes the generated execution script (S106). By executing the execution script, the control unit 11 of the information processing device 1 accesses the application server AS that executes the application to be tested, and tests the application based on the processing in the execution script.

[0043] The control unit 11 of the information processing device 1 acquires the execution result of the execution script (S107). The control unit 11 of the information processing device 1 outputs the execution result of the execution script (S108). The control unit 11 of the information processing device 1 may acquire the execution result, such as a processing result or an error output obtained by executing the execution script, and output and display the execution result on the display unit 141.

[0044] According to an embodiment of the present disclosure, the information processing device 1 (controller 11) identifies an area of ​​an operation image (an image showing an operation object) that is substantially identical to an element image (an image of an element to be detected) on an application screen (a screenshot image) based on the element image. The information processing device 1 derives the position of the operation image whose area has been identified, and generates and outputs an execution script for testing the application based on the derived position of the operation image and test information (e.g., a click on the operation image (operation object)). Therefore, when testing an application, it is possible to efficiently detect or extract an operation object (an element to be tested), and it is possible to relatively easily generate an execution script for testing. That is, for example, even if the application to be tested is a web application executed on a browser, it is possible to generate an execution script for testing without specifying an element path (Xpath) that uniquely identifies the operation object (an element to be tested). Alternatively, even if the application to be tested is a rich client compatible application configured using, for example, a Java Applet or ActiveX (.NET Framework), and the object to be operated (element to be tested) does not support element paths (Xpath), the object to be operated (element to be tested) can be specified based on the identity (degree of match, similarity) between the operation image showing the object to be operated and the element image.

[0045] According to an embodiment of the present disclosure, the information processing device 1 (controller 11) performs a comparison process between the entire application screen (screenshot image) and an element image (image of an element to be detected). When performing the comparison process between these two images (application screen, element image), the information processing device 1 may derive a degree of match (similarity) of the element image for each partitioned area of ​​the application screen using an image processing library such as OpenCV [cv2.matchTemplate function]. The information processing device 1 identifies a comparison target area where the derived degree of match (similarity) is equal to or greater than a predetermined threshold as an operation image indicating a target object to be operated. Setting values ​​or parameters required for various calculations, such as thresholds used in the comparison process, are stored in the storage unit 12 of the information processing device 1, and the control unit 11 can acquire various parameters, such as thresholds, by referring to the storage unit 12. In this way, based on the degree of match (similarity) between each area (comparison area) partitioned on the application screen (screenshot image) and the element image, the comparison area whose degree of match (similarity) is above a threshold is identified as the operation image, thereby ensuring the accuracy of identifying the operation image.

[0046] According to an embodiment of the present disclosure, the information processing device 1 (control unit 11) uses an acquired element image to enlarge or reduce the element image, thereby generating a plurality of element images of different sizes. An element image group is formed from the plurality of element images of different sizes generated in this manner. That is, the element image group includes a plurality of element images of different sizes. Furthermore, the information processing device 1 (control unit 11) uses the acquired screen information (screenshot images) of the application to generate a plurality of screens (screenshot images) of different sizes by gradually reducing the screenshot images. An application screen group (source image group) is formed from the plurality of screenshot images of different sizes generated in this manner. That is, the application screen group (source image group) includes a plurality of screenshot images of different sizes. The information processing device 1 derives a degree of match in the extracted comparison area by comparing each element image included in the element image group with each screen included in the application screen group. This allows the degree of match to be determined while absorbing the size difference between the two images to be compared, thereby improving the accuracy of calculating the degree of match.

[0047] According to an embodiment of the present disclosure, the information processing device 1 (controller 11) uses multiple Gaussian filters with gradually varying resolutions (blurring levels) to generate multiple screenshot images with gradually varying resolutions and sizes from an original image (1 / 1 resolution) of a screenshot image (a screen capture of an operation window in which an application is executed). The gradually varying resolutions may be expressed, for example, as 1 / (2^n), where n is a power and an integer (level) greater than or equal to 1. A Gaussian pyramid is constructed by stacking screenshot images created by this repeated reduction process. In this case, the resolution decreases by half as the level (n) in the Gaussian pyramid increases. By using a Gaussian filter to convert an application screen (screenshot image) to be compared with an element image into an image (screenshot image) with a different resolution, fine noise and texture within the image can be smoothed, resulting in an overall smooth image. This improves the matching accuracy with the element image and efficiently extracts (identifies) a comparison target area (operation image showing a target object) that matches the element image in each of the application screens (screenshot images) with different resolutions.

[0048] According to an embodiment of the present disclosure, the information processing device 1 (control unit 11) extracts a comparison area (operation image) that matches an element image in an application screen (screenshot image) of any level (level: l) by, for example, performing a brute-force comparison (template matching) between a group of element images and a group of application screens. In this case, the information processing device 1 may identify the screenshot image (level: l) from which the comparison area (operation image) that matches the element image was extracted as the matching screen. The information processing device 1 may derive a comparison area (operation image) in a screenshot image (level: l-1) with a higher resolution than the matching screen “screenshot image (level: l)” from which the comparison area (operation image) was extracted. For two screenshot images (levels: l, l-1) with different resolutions, the position and size of the comparison area (operation image) in the screenshot image with a higher resolution (level: l-1) may be geometrically calculated based on the position and size of the comparison area (operation image) in the screenshot image with a lower resolution (level: l). The information processing device 1 can calculate the position and size of the comparison area (operation image) on the application screen (screen capture of the operation window in which the application is executed), which is the original data when using a Gaussian filter, by recursively repeating the matching process of the comparison area (operation image) on two screenshot images (levels: m, m-1) with different resolutions.

[0049] The information processing device 1 may compare the comparison area (operation image) extracted from the application screen (level 0 screenshot image), which is the original data, with the element image, for example, using phase-only correlation, to determine whether the derived degree of match is appropriate (confirm the determination result of the degree of match). When comparing using phase-only correlation, the information processing device 1 may enlarge or reduce the element image so that it is substantially the same size as the comparison area (operation image) extracted from the application screen (level 0 screenshot image), and then apply phase-only correlation to the element image of the same size. If the comparison result using phase-only correlation is negative, the information processing device 1 may again compare the element images with the application screen group to extract a comparison area whose degree of match is equal to or greater than a threshold. By applying phase-only correlation in this way, the position of the operation image on the application screen based on the operation image is derived in essentially two stages, thereby ensuring the accuracy of the derivation.

[0050] According to an embodiment of the present disclosure, the information processing device 1 extracts an element path (xpath) from the screen information, which uniquely identifies the operation target object of the operation image, using the position of the operation image derived according to the degree of match with the element image. When the screen information is defined in, for example, HTML, the element path (xpath) may be extracted (identified) based on the processing result of the scraping process on the screen information defined in the HTML and the position (client coordinates) of the operation image. By using the element path identified based on the position of the operation image in this way, it is possible to efficiently define an operation for the operation target object indicated by the operation image.

[0051] (Embodiment 2) 8 is a flowchart illustrating a processing procedure by the control unit 11 of the information processing device 1 according to the second embodiment (selection screen). The control unit 11 of the information processing device 1 acquires screen information of an application to be tested (S201). The control unit 11 of the information processing device 1 acquires an element image corresponding to an object to be operated (S202). The control unit 11 of the information processing device 1 derives the position and size of an operation image indicating the object to be operated based on the acquired element image (S203). The control unit 11 of the information processing device 1 performs the processes from S201 to S203, similar to the processes from S101 to S103 of the first embodiment.

[0052] In this embodiment, when deriving the position and size of an operation image indicating an object to be operated based on the acquired element image, if there are multiple comparison target areas (operation images that are substantially the same as the element image) whose degree of match is equal to or greater than a threshold, the control unit 11 of the information processing device 1 outputs each of these multiple comparison target areas as a candidate for the operation image. If multiple candidate operation images are extracted from the application screen (screenshot image), the control unit 11 of the information processing device 1 stores the position and size of each of these operation images (areas on the application screen) in the storage unit 12.

[0053] The control unit 11 of the information processing device 1 determines whether or not the number of extracted operation images is plural (S204). The control unit 11 of the information processing device 1 determines whether or not the number of extracted operation images is plural by referring to the storage unit 12.

[0054] If the number of derived operation images is plural (S204: YES), the control unit 11 of the information processing device 1 outputs a selection screen (S205). The control unit 11 of the information processing device 1 accepts an operation to select one of the operation images on the selection screen (S206). The control unit 11 of the information processing device 1 accepts a selection of one of the operation images by a user operation or the like from among the extracted plurality of candidate operation images (areas on the application screen) on the output selection screen. The control unit 11 of the information processing device 1 identifies the selected operation image as the operation image indicating the operation target object to be tested.

[0055] FIG. 9 is an explanatory diagram showing an example of a selection screen. The selection screen highlights an area (extracted comparison area) in the application screen to be tested that is substantially the same as the element image to be detected. The application screen to be tested includes icons, function buttons, or link images representing multiple operation target objects, and it is assumed that identical or similar images are set for these multiple operation target objects. In the illustration of this embodiment, for example, three operation images (comparison area) that are the same as the element image to be detected are displayed. Even if multiple operation images (comparison area) that are the same as the element image to be detected are arranged in the application screen, the control unit 11 of the information processing device 1 outputs the selection screen and accepts an operation to select one of the operation images, thereby efficiently identifying the operation image representing the operation target object to be tested.

[0056] If the number of derived operation images is not plural (S204: NO), that is, if the number of derived operation images is singular (one), the control unit 11 of the information processing device 1 acquires test information for the application (S207). If the number of derived operation images is plural, as described above, an operation image indicating an operation target object is identified by accepting the selection of any operation image from the extracted plurality of operation images. The control unit 11 of the information processing device 1 acquires information (test information) for performing test automation, such as clicking the operation image (operation image indicating an operation target object) in the same manner as in process S104 of the first embodiment, for the operation image identified in this way (operation image indicating an operation target object), or inputting text or the like if the operation image indicates an input field.

[0057] The control unit 11 of the information processing device 1 generates an execution script for testing the application based on the position of the operation image and the test information (S208). The control unit 11 of the information processing device 1 executes the generated execution script (S209). The control unit 11 of the information processing device 1 acquires the execution result of the execution script (S210). The control unit 11 of the information processing device 1 outputs the execution result of the execution script (S211). The control unit 11 of the information processing device 1 performs the processes from S208 to S211, similar to the processes from S105 to S108 in the first embodiment.

[0058] According to an embodiment of the present disclosure, when multiple operation images are identified by deriving multiple comparison target regions whose degree of match is equal to or greater than a predetermined threshold, the information processing device 1 (control unit 11) outputs information about the multiple operation images. When outputting information about the multiple operation images, the information processing device 1 may generate a selection screen for selecting one of the multiple operation images and output the selection screen to a display device or information terminal. The information processing device 1 accepts a user's operation to select one of the multiple operation images on the selection screen, and identifies the selected operation image as the operation image representing the operation target object to be tested. In this way, even when multiple images (comparison target regions) similar to the element image (image of the element to be detected) exist (are arranged in multiple locations) on the application screen (screenshot image), the operation image representing the operation target object can be efficiently identified by outputting a selection screen that allows the selection of any one of the operation images.

[0059] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0060] Multiple claims may be combined with each other regardless of the form of reference. Multiple dependent claims may be included in the claims, depending on multiple claims. Multiple dependent claims may be included in a multiple dependent claim. If multiple dependent claims are not included in a multiple dependent claim, this does not limit the number of multiple dependent claims that are included in a multiple dependent claim. [Explanation of symbols]

[0061] S Test Automation System AS Application Server 1. Information processing device (test automation server) 11 Control section 12 Storage section 121 Recording Media P Program (Program Product) 13 Communications Department 14 Input / Output Interface 141 Display section

Claims

1. On the computer, Obtain screen information for the application under test, obtaining test information for the application; acquire an element image corresponding to an operation target object included in the acquired screen information; Identifying an operation image indicating the operation target object based on the acquired element image; deriving a position of the operation image on the screen of the application based on the identified operation image; outputting an execution script for testing the application based on the derived position of the operation image and the test information; extracting one or more comparison target regions on the screen of the application; deriving a degree of match between each of the extracted comparison regions and the element image; identifying a comparison target area in which the derived degree of coincidence is equal to or greater than a predetermined threshold as the operation image representing the operation target object; generating an element image group including a plurality of element images of different sizes using the acquired element images; generating an application screen group including a plurality of screens of different sizes using the acquired screen information of the application; deriving a degree of match in the extracted comparison region by comparing each of the element images included in the element image group with each of the screens included in the application screen group; the plurality of screens of different sizes included in the group of application screens are generated using a Gaussian filter so that the resolutions are gradually varied; a Gaussian pyramid is formed by the application screen group; Identifying a matching screen from among a plurality of screens included in the group of application screens, the matching screen including a comparison target area having a degree of match equal to or greater than a predetermined threshold; Deriving a position of the operation image on a screen having a higher resolution than the resolution of the matching screen based on a comparison target area included in the matching screen. A program that executes a process.

2. a screen having a higher resolution than the matching screen is an application screen that is source data when using the Gaussian filter; The operation image on the application screen is compared with the element image using a phase-only correlation method, and the suitability of the derived degree of match is determined. The program according to claim 1.

3. The position of the operation image on the screen of the application is derived in client coordinates based on the operation window in which the application is executed. The program according to claim 1.

4. When a plurality of comparison target regions having the degree of match equal to or greater than a predetermined threshold are derived and a plurality of operation images are identified, an operation of selecting one of the plurality of identified operation images is accepted; Using the position of the selected operation image, an execution script for testing the application is output. The program according to claim 1.

5. extracting, from the screen information, an element path that uniquely identifies the operation target object of the operation image using the derived position of the operation image; Using the extracted element path, the execution script for testing the application is output. The program according to claim 1.

6. Execute the output execution script, Output the execution result of the execution script The program according to any one of claims 1 to 5.

7. Obtain screen information for the application under test, obtaining test information for the application; acquire an element image corresponding to an operation target object included in the acquired screen information; Identifying an operation image indicating the operation target object based on the acquired element image; deriving a position of the operation image on the screen of the application based on the identified operation image; outputting an execution script for testing the application based on the derived position of the operation image and the test information; extracting one or more comparison target regions on the screen of the application; deriving a degree of match between each of the extracted comparison regions and the element image; identifying a comparison target area in which the derived degree of coincidence is equal to or greater than a predetermined threshold as the operation image representing the operation target object; generating an element image group including a plurality of element images of different sizes using the acquired element images; generating an application screen group including a plurality of screens of different sizes using the acquired screen information of the application; deriving a degree of match in the extracted comparison region by comparing each of the element images included in the element image group with each of the screens included in the application screen group; the plurality of screens of different sizes included in the group of application screens are generated using a Gaussian filter so that the resolutions are gradually varied; a Gaussian pyramid is formed by the application screen group; Identifying a matching screen from among a plurality of screens included in the group of application screens, the matching screen including a comparison target area having a degree of match equal to or greater than a predetermined threshold; Deriving a position of the operation image on a screen having a higher resolution than the resolution of the matching screen based on a comparison target area included in the matching screen. An information processing method that causes a computer to execute a process.

8. Obtain screen information for the application under test, obtaining test information for the application; an acquisition unit that acquires an element image corresponding to an operation target object included in the acquired screen information; an identification unit that identifies an operation image indicating the operation target object based on the acquired element image; a derivation unit that derives a position of the operation image on the screen of the application based on the identified operation image; an output unit that outputs an execution script for testing the application based on the derived position of the operation image and the test information; Equipped with extracting one or more comparison target regions on the screen of the application; deriving a degree of match between each of the extracted comparison regions and the element image; identifying a comparison target area in which the derived degree of coincidence is equal to or greater than a predetermined threshold as the operation image representing the operation target object; generating an element image group including a plurality of element images of different sizes using the acquired element images; generating an application screen group including a plurality of screens of different sizes using the acquired screen information of the application; deriving a degree of match in the extracted comparison region by comparing each of the element images included in the element image group with each of the screens included in the application screen group; the plurality of screens of different sizes included in the group of application screens are generated using a Gaussian filter so that the resolutions are gradually varied; a Gaussian pyramid is formed by the application screen group; Identifying a matching screen from among a plurality of screens included in the group of application screens, the matching screen including a comparison target area having a degree of match equal to or greater than a predetermined threshold; Deriving a position of the operation image on a screen having a higher resolution than the resolution of the matching screen based on a comparison target area included in the matching screen. Information processing device.

Images