Verification support device, verification support system, and verification support method

Abstract

The purpose of the present invention is to provide a technology capable of generating a script file for operating software such as an application without analyzing the source code. This verification support device comprises: a UI component input information generation unit that generates input information on the basis of a UI component type, an input information estimation result, and an ID; a script file generation unit that generates a script file for operating software, on the basis of the generated input information; and a software operation unit that operates the software using the script file.

Description

Verification support device, verification support system, and verification support method

[0001] The present disclosure relates to a verification support device, a verification support system, and a verification support method.

[0002] For software such as web pages or smartphone applications that use a GUI (Graphical User Interface), screen verification is performed during development to check whether the screen is displayed correctly on the display device. One method for checking whether the screen is displayed correctly is to improve testing efficiency by supporting screen verification using a test automation tool such as a UI automation framework. This verification support method requires that a correct baseline screen image be prepared in advance.

[0003] There are two preparation methods: one in which a tester operates an application to perform screen transitions and takes screenshots of the screen, and another in which a script automatically operates the application to perform screen transitions and takes screenshots of the screen. Regarding the latter method using a script, for example, Patent Document 1 proposes a technology that analyzes the source code of an application to extract screen transition information and automatically generates a script file for performing screen transitions.

[0004] Patent No. 6723976

[0005] However, with source code analysis technology, if the screen components and therefore the source code of the application change due to an application modification, it is not possible to automatically generate a script file, which results in the problem of being unable to operate the application automatically.

[0006] Therefore, the present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology that makes it possible to generate script files for operating software such as applications without analyzing the source code.

[0007] a software operation unit that operates the software using the script file; a screen transition determination unit that determines whether the screen image has transitioned due to the operation of the software; and a screen transition information generation unit that generates screen transition information by linking the determination result of the transition of the screen image with information in the script file used to operate the software.

[0008] According to the present disclosure, a UI component is detected from a screen image, input information is generated based on the type of the UI component, the input information estimation result, and the ID, and a script file for operating software is generated based on the generated input information. With this configuration, it is possible to generate a script file for operating software such as an application without analyzing the source code.

[0009] The objects, features, aspects and advantages of the present disclosure will become more apparent from the following detailed description and the accompanying drawings.

[0010] FIG. 1 is a diagram illustrating a configuration of an application verification support device according to a first embodiment. FIG. 2 is a diagram illustrating an example of information related to a screen image. FIG. 3 is a flowchart illustrating processing performed by an algorithm of a UI component identification unit according to the first embodiment. FIG. 4 is a diagram illustrating an example of information related to a screen image. FIG. 5 is a flowchart illustrating processing performed by an algorithm of a UI component input information estimation unit according to the first embodiment. FIG. 6 is a diagram illustrating an example of information related to a screen image. FIG. 7 is a diagram illustrating an example of a template portion. FIG. 8 is a diagram illustrating an example of data stored in a UI component input data storage unit. FIG. 9 is a diagram illustrating an example of data stored in a script file storage unit. FIG. 10 is a diagram illustrating an example of transition of a screen image. FIG. 11 is a diagram illustrating an example of screen transition information. FIG. 12 is a block diagram illustrating a hardware configuration of an application verification support device according to another modified example. FIG. 13 is a block diagram illustrating a hardware configuration of an application verification support device according to another modified example.

[0011] 1 is a diagram showing the configuration of an application verification support device 20 which is a verification support device according to this embodiment 1. The application verification support device 20 includes an application screen image acquisition unit 21 which is a screen image acquisition unit, an application operation unit 22 which is a software operation unit, an application screen image storage unit 23, a UI component detection device 30, a screen transition information generation device 40, and a script generation device 50.

[0012] The UI component detection device 30 includes a UI component detection unit 31 , a UI component identification unit 32 , a UI component input information estimation unit 33 , a UI component ID assignment unit 34 , a UI component detection model storage unit 35 , and a UI component information storage unit 36 ​​.

[0013] The screen transition information generating device 40 includes a screen transition determining unit 41 , a screen transition information generating unit 42 , and a screen transition information storage unit 43 .

[0014] The script generating device 50 includes a UI component input information generating unit 51 , a script file generating unit 52 , a test constraint storing unit 53 , a UI component input data storing unit 54 , and a script file storing unit 55 .

[0015] The verification target device 10 appropriately executes a test target application 11, which is software to be tested. The verification target device 10 may be a device having hardware such as a smartphone, or may be a device that simulates the same operation on software such as an emulator. The test target application 11 is, for example, an application that outputs (displays) any image developed by a user.

[0016] The application screen image acquisition unit 21 acquires a screen image of the application 11 under test that is being executed in the verification target device 10. The screen image may be in any format, such as a still image, as long as it can be processed within the application verification support device 20. The application screen image acquisition unit 21 outputs the acquired screen image to the UI component detection unit 31 of the UI component detection device 30 and the screen transition determination unit 41 of the screen transition information generation device 40, and stores the screen image in the application screen image storage unit 23.

[0017] The UI component detection unit 31 of the UI component detection device 30 detects UI components from the screen image acquired by the application screen image acquisition unit 21 based on the detection model stored in the UI component detection model storage unit 35. UI components include, for example, buttons, labels, and text boxes. To detect a UI component, the UI component detection unit 31 may detect the type of the UI component and its position within the screen image, or may assign an ID (identifier) ​​to the UI component that can uniquely identify the UI component.

[0018] The UI component identification unit 32 determines whether the type of the UI component detected by the UI component detection unit 31 is a predetermined type. The predetermined types include types of UI components that are difficult to identify by their appearance alone, such as text boxes and select boxes, but whose appearance becomes identifiable by an operation such as clicking on the UI component.

[0019] The UI component input information estimation unit 33 estimates input information that is information to be input to the UI component detected by the UI component detection unit 31. For example, the UI component input information estimation unit 33 estimates the input information based on characters around the detected UI component. The input information may be numerical values, characters, or a combination thereof.

[0020] The UI component ID assigning unit 34 assigns an ID that is the same as the ID that can uniquely identify the UI component detected by the UI component detection unit 31 to the UI component in the screen image acquired by the application screen image acquisition unit 21. In other words, the UI component ID assigning unit 34 links the UI component detected by the UI component detection unit 31 with the UI component in the screen image of the application 11 under test by the ID.

[0021] The UI component ID assigning unit 34 associates the type of UI component, the estimation result of the input information, and the ID, and stores them in the UI component information storage unit 36 ​​.

[0022] The UI component input information generation unit 51 of the script generation device 50 generates input information based on the test constraints stored in the test constraint storage unit 53 and the data stored in the UI component information storage unit 36. Test constraints are constraints imposed on input information, such as valid value information for input items. For example, if the input information is a user ID, the constraint is "alphanumeric characters and symbols only," if the input information is a Japanese name, the constraint is "alphanumeric characters not allowed," and if the input information is an email address, the constraint is "a combination of alphanumeric characters, @, and alphanumeric characters." The data stored in the UI component information storage unit 36 ​​includes the type of UI component, the input information estimation result, and the ID.

[0023] As described above, the input information generated by UI component input information generation unit 51 is information to be input to the UI component detected by UI component detection unit 31, and a single piece of input information may be generated, or multiple pieces of input information may be generated. UI component input information generation unit 51 associates the generated input information with data used to generate the input information from the data stored in UI component information storage unit 36, and stores the associated data in UI component input data storage unit 54.

[0024] The UI component input information generation unit 51 may generate input information based on the position of the UI component in the screen image, as well as the data stored in the UI component information storage unit 36 ​​(i.e., the type of UI component, the input information estimation result, and the ID). The UI component input information generation unit 51 may also generate input information by receiving the same data as the data stored in the UI component information storage unit 36 ​​from the UI component ID assignment unit 34.

[0025] The script file generation unit 52 generates a script file for operating an application with input information input to a UI component, based on data including input information stored in the UI component input data storage unit 54. The script file generation unit 52 stores the generated script file in the script file storage unit 55. The script file generation unit 52 may generate one script file or multiple script files. The script file generation unit 52 may also generate a script file by receiving the same data as the data stored in the UI component input data storage unit 54 from the UI component input information generation unit 51.

[0026] The application operation unit 22 of the application verification support device 20 operates the application 11 under test using the generated script file stored in the script file storage unit 55. This operation may cause a transition of the screen image depending on the screen image of the application 11 under test. Note that the application operation unit 22 may operate the application 11 under test by receiving, from the script file generation unit 52, a script file that is the same as the script file stored in the script file storage unit 55.

[0027] The screen transition determination unit 41 of the screen transition information generating device 40 determines whether a screen image has transitioned due to an operation of the test target application 11 performed by the application operation unit 22. For example, the screen transition determination unit 41 compares the screen image currently acquired by the application screen image acquisition unit 21 with the screen image previously acquired by the application screen image acquisition unit 21, and determines whether a screen image of the test target application 11 has transitioned. The screen image previously acquired by the application screen image acquisition unit 21 is the screen image stored in the application screen image storage unit 23.

[0028] The screen transition information generating unit 42 generates screen transition information by linking the determination result by the screen transition determining unit 41 regarding the transition of screen images with information on the script files used to operate the application under test 11. For example, the screen transition information generating unit 42 stores in the screen transition information storage unit 43 screen transition information linking the executed script files with screen transition data in which screen images of the application under test 11 are arranged in the order in which the script files are executed.

[0029] Next, detailed operations of each component of the application verification support device 20 will be described as appropriate.

[0030] The verification target device 10 executes the application 11 to be tested within the verification target device 10, for example, by operating the application operation unit 22. The application 11 to be tested may be started manually or automatically.

[0031] The application screen image acquisition unit 21 acquires a screen image of the test target application 11, for example, by taking a screenshot of the screen of the test target application 11. The application screen image acquisition unit 21 assigns a screen image ID that can uniquely identify the acquired screen image to the screen image, outputs the screen image to the UI component detection unit 31 and the screen transition determination unit 41, and stores the screen image in the application screen image storage unit 23. The screen image ID may be a number, a character, or a combination thereof. An example of a screen image ID is shown as a screen image ID 100 in FIG. 2 .

[0032] The UI component detection unit 31 detects the type of UI component and the position of the UI component in the screen image from the screen image acquired by the application screen image acquisition unit 21 based on the detection model stored in the UI component detection model storage unit 35. Any detection model may be used as the detection model as long as it can detect the type and position of the UI component, and may be a machine learning (trained) detection model. For example, the type and position of the UI component may be detected by a deep learning model such as YOLOv7 that performs object detection and object recognition in parallel using instance segmentation, or may be detected by multiple deep learning models that perform object detection and object recognition separately.

[0033] The position of a UI component is indicated, for example, by the coordinates (x, y) of the upper left corner of the detection area, with the upper left corner of the screen image being the reference position (0, 0), the width of the detection area, and the height of the detection area. The position of a UI component may be specified by other parameters as long as the position can be specified by the detection area surrounding the UI component. Examples of UI component types and positions are shown in UI component type 101 and detection area 102 in FIG. 2.

[0034] The UI component detection unit 31 also assigns a UI component ID to each detected UI component, which can uniquely identify the UI component. The UI component ID may be a number, a character, or a combination thereof. An example of a UI component ID is shown as a UI component ID 103 in FIG. 2.

[0035] The UI component identification unit 32 identifies whether the type of UI component detected by the UI component detection unit 31 is a type of UI component that is difficult to identify by its appearance as it is, but whose appearance becomes identifiable by operating the UI component.

[0036] 3 is a flowchart showing the processing performed by the algorithm of the UI component identification unit 32 according to embodiment 1. As an example, the processing for identifying a text box and a select box will be described below.

[0037] First, in step S1, UI component identification unit 32 reads the type of UI component detected by UI component detection unit 31. In step S2, UI component identification unit 32 determines whether the type of the read UI component is a text box. If the type is a text box, the process proceeds to step S3, and if the type is not a text box, the process returns to step S1.

[0038] In step S3, the UI component identification unit 32 takes a screenshot of the UI component whose type has been read, thereby acquiring an image of the UI component and its vicinity. In step S4, the UI component identification unit 32 generates and executes a script file for clicking the UI component whose type has been read. In step S5, the UI component identification unit 32 takes a screenshot of the UI component clicked by executing the script file, thereby acquiring an image of the UI component and its vicinity.

[0039] In step S6, the UI component identification unit 32 compares the pre-click image acquired in step S3 with the post-click image acquired in step S5 to determine whether the image has changed. The determination of image change may be performed using a difference method, feature point matching, or other methods. If it is determined that the image has changed, the process proceeds to step S7, and if it is determined that the image has not changed, the process proceeds to step S9.

[0040] In step S7, the UI component identification unit 32 determines that the type of the loaded UI component is a select box. In step S8, the UI component identification unit 32 changes the type of the loaded UI component from a text box to a select box. Then, the process in FIG. 3 ends.

[0041] In step S9, the UI component identification unit 32 determines that the type of the loaded UI component is a text box, and then the process of FIG.

[0042] Note that the input information for a select box is usually fixed. Therefore, in the first embodiment, an OCR (Optical Character Recognition / Reader) is applied to the image area changed in step S6 to recognize the character area, and the information obtained thereby is acquired as the input information for the select box. In Figure 4, the fourth text box from the top in Figure 2 has been changed to a select box by the UI component identification unit 32, and the acquired input information is set as the input information 104 for that UI component.

[0043] The UI component input information estimation unit 33 estimates input information for UI components such as text boxes into which any character can be input, among the UI components detected by the UI component detection unit 31. However, as described above, for UI components such as select boxes whose input information is usually fixed, the input information has already been acquired by the UI component identification unit 32, and therefore the UI component input information estimation unit 33 does not estimate the input information.

[0044] FIG. 5 is a flowchart showing the processing performed by the algorithm of the UI component input information estimation unit 33 according to the first embodiment.

[0045] First, in step S21, the UI component input information estimation unit 33 reads the type of UI component detected by the UI component detection unit 31. In step S22, the UI component identification unit 32 determines whether the type of the read UI component is a UI component that allows arbitrary characters to be input. The determination of whether a UI component allows arbitrary characters to be input is made based on, for example, the type of UI component, but is not limited to this. If it is determined that the UI component allows arbitrary characters to be input, the process proceeds to step S23, and if it is determined that the UI component does not allow arbitrary characters to be input, the process returns to step S21.

[0046] In step S23, the UI component input information estimation unit 33 applies OCR to the area in front of the detection position of the UI component whose type has been read, and determines whether or not a character area is present in that area. If it is determined that a character area is present, the process proceeds to step S25, and if it is determined that a character area is not present, the process proceeds to step S24.

[0047] In step S24, the UI component input information estimation unit 33 applies OCR to the area above the detection position of the UI component whose type has been read, and determines whether or not a character area is present in that area. If it is determined that a character area is present, the process proceeds to step S25, and if it is determined that a character area is not present, the process in FIG. 5 ends.

[0048] In step S25, the UI component input information estimation unit 33 acquires the characters in the character area as a result of estimation of the input information of the UI component whose type has been read, and associates the input information estimation result with the UI component. Then, the processing in FIG. 5 ends. An example of the input information estimation result is shown as input information 104 in FIG. 6.

[0049] For example, if "Name" is written around a text box for inputting a name, the UI component input information estimation unit 33 estimates "Name" as the input information for the text box. Also, if "Input Conditions" are written around the UI component, the UI component input information estimation unit 33 estimates "Input Conditions" that can improve the estimation accuracy of the input information as the input information for the UI component.

[0050] 6 to the UI component in the application 11 under test, for the UI component detected by the UI component detection unit 31. This associates the UI component detected by the UI component detection unit 31 with the UI component in the application 11 under test. The UI component ID assignment unit 34 then stores the data in FIG. 6 , which associates the UI component type, the input information estimation result, the ID, and the like, in the UI component information storage unit 36.

[0051] The UI component input information generation unit 51 generates input information for a UI component based on the test constraints stored in the test constraint storage unit 53 and the data stored in the UI component information storage unit 36. For example, the UI component input information generation unit 51 may display a template statement in which the test constraints and the data from the UI component information storage unit 36 ​​are embedded, and generate information input by the user as input information.

[0052] Fig. 7 is a diagram showing an example of a template sentence. The contents of the UI component type 101 and input information 104 in Fig. 4 are embedded in the <UI component type> and <input information> in Fig. 7, and the contents of the <test constraint> in the test constraint storage unit 53 are embedded in the <test constraint> in Fig. 7. The UI component input information generation unit 51 then generates information input by the user in the input example in Fig. 7 as input information for the UI component. The UI component input information generation unit 51 may perform machine learning (training) of the template sentence using a natural language generation AI model such as ChatGPT.

[0053] Note that the operation of the UI component input information generation unit 51 to generate input information for UI components is not limited to the above. For example, the UI component input information generation unit 51 may generate input information using a table that previously associates test constraints with data stored in the UI component information storage unit 36. Furthermore, the UI component input information generation unit 51 may learn the association between test constraints and data stored in the UI component information storage unit 36 ​​through machine learning (training), such as deep learning.

[0054] The UI component input information generation unit 51 associates the generated input information with the data used to generate the input information among the data stored in the UI component information storage unit 36 ​​and stores the associated data in the UI component input data storage unit 54 .

[0055] FIG. 8 is a diagram showing an example of data stored in the UI component input data storage unit 54. Test constraint 105 in FIG. 8 is a test constraint used to generate input information, and first input information 106a and second input information 106b in FIG. 8 are input information generated by the UI component input information generation unit 51. Note that input information 104 for UI components whose input information is usually fixed, such as select boxes, is set as first input information 106a and second input information 106b. The UI component input information generation unit 51 may generate multiple pieces of input information as long as it generates one or more valid pieces of input information, or may generate invalid input information for displaying an error screen.

[0056] The script file generation unit 52 obtains the input information generated by the UI component input information generation unit 51 from the UI component input data storage unit 54, and generates a script file for operating an application based on the input information. The combination of input information used to generate the script file may be a brute force search or a combination of one valid input information and one invalid input information, as long as valid input information is input to all UI components in the screen image. Note that the method of generating the script file is not limited to these.

[0057] The script file generation unit 52 assigns a script file ID that can uniquely identify the generated script file to the script file, and stores the script file in the script file storage unit 55. The script file ID may be a number, a character, or a combination thereof.

[0058] Fig. 9 is a diagram showing an example of data stored in the script file storage unit 55. The data in the example of Fig. 9 includes a screen image ID 100, a script file ID 107 that can uniquely identify a script file, file content 108, and a script file execution time 109, which are all linked together. Note that the format of the script file execution time 109 is not important as long as the execution time can be identified.

[0059] The application operation unit 22 executes the generated script file to operate the application under test 11. After executing the script file, the application operation unit 22 stores the execution time in the script file execution time 109 in FIG.

[0060] 10 is a diagram showing an example of a transition of a screen image when the application operation unit 22 executes all the generated script files and operates the test target application 11. When all the generated script files have been executed and there is no transition of the screen image, the application operation unit 22 ends the operation of the test target application 11.

[0061] The screen transition determination unit 41 determines whether or not a screen image has transitioned due to an operation of the test target application 11. An example will be described below in which the application operation unit 22 determines whether or not a screen image has transitioned in the reverse order to the order in which the script files were executed.

[0062] First, the screen transition determination unit 41 acquires a screen image after the latest script file has been executed from the application screen image acquisition unit 21. The screen transition determination unit 41 also acquires an image ID, a script file ID, and an execution time for the screen image after the latest script file has been executed from the script file storage unit 55.

[0063] Next, screen transition determination unit 41 obtains the image ID, script file ID, and execution time of the screen image before the execution of the latest script file that is linked to an execution time that is closest to and earlier than the obtained execution time from script file storage unit 55. Screen transition determination unit 41 also obtains, from application screen image storage unit 23, the screen image to which the screen image ID of the latest script file before the execution has been assigned.

[0064] Then, the screen transition determination unit 41 compares the screen image after the execution of the latest script file with the screen image before the execution of the latest script file. The method of comparing the screen images is not limited.

[0065] When the screen transition determination unit 41 determines that the screen images before and after the execution of the script file are different from each other, it determines that the screen image before the execution of the most recent script file has transitioned.Then, the screen transition determination unit 41 determines the screen transition determination result for the screen image before the execution of the most recent script file as "screen transition has occurred" and determines whether there is a transition destination as "yes."The screen transition determination unit 41 also sets the screen image before the execution of the most recent script file as the screen image after the execution of the second-most recent script file.

[0066] Next, screen transition determination unit 41 acquires the screen image ID, script file ID, and execution time of the screen image before the execution of the second-most recent script file, which is linked to an execution time that is closest to and earlier than the execution time of the screen image after the execution of the second-most recent script file, from script file storage unit 55. Screen transition determination unit 41 acquires, from application screen image storage unit 23, the screen image to which the screen image ID of the screen image before the execution of the second-most recent script file is assigned.

[0067] Then, the screen transition determination unit 41 compares the screen image after the execution of the second most recent script file with the screen image before the execution of the second most recent script file.

[0068] The screen transition determination unit 41 repeats the above operation until it determines that the screen images before and after the execution of the script file are the same and determines that the screen image before the execution of the script file has not transitioned. For screen images that have not been determined to have a screen transition or have a screen transition, the screen transition determination unit 41 determines the screen transition determination result as "no screen transition" and determines whether there is a transition destination as "no."

[0069] The screen transition determination unit 41 outputs the determination result regarding the transition of the screen image (e.g., screen transition determination result, whether there is a transition destination) and information about the script file (e.g., the screen image ID used for the determination, the execution time, and the script file ID) to the screen transition information generation unit 42.

[0070] The screen transition information generating unit 42 generates screen transition information based on the information from the screen transition determining unit 41, and stores the generated screen transition information in the screen transition information storing unit 43. An example of the screen transition information is shown in FIG.

[0071] Summary of First Embodiment As described above, the application verification support device 20 according to the first embodiment detects UI components from a screen image, generates input information based on the type of the UI component, the input information estimation result, and the ID, and generates a script file for operating the software based on the generated input information. This makes it possible to generate a script file for operating the application on an image basis without analyzing the development language of the application. Therefore, even if the screen components and therefore the source code of the application change, the application can be automatically operated.

[0072] Furthermore, in the first embodiment, it is determined whether or not a screen image has transitioned due to an operation of the application 11 under test, and screen transition information is generated by linking the determination result with information on the script file used to operate the application 11 under test. With this configuration, the user can easily check the transition of the screen image.

[0073] <Other Modifications> The application screen image acquisition unit 21, UI component detection unit 31, UI component identification unit 32, UI component input information estimation unit 33, UI component ID assignment unit 34, UI component input information generation unit 51, script file generation unit 52, application operation unit 22, screen transition determination unit 41, and screen transition information generation unit 42 shown in FIG. 1 are hereinafter referred to as the "application screen image acquisition unit 21, etc." The application screen image acquisition unit 21, etc. are realized by a processing circuit 81 shown in FIG. 12. That is, the processing circuit 81 includes the application screen image acquisition unit 21, UI component detection unit 31, UI component identification unit 32, UI component input information estimation unit 33, UI component ID assignment unit 34, UI component input information generation unit 51, script file generation unit 52, application operation unit 22, screen transition determination unit 41, and screen transition information generation unit 42. Dedicated hardware or a processor that executes a program stored in a memory may be applied to the processing circuit 81. Examples of processors include central processing units, processing units, arithmetic units, microprocessors, microcomputers, and DSPs (Digital Signal Processors).

[0074] When the processing circuitry 81 is dedicated hardware, the processing circuitry 81 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each unit, such as the application screen image acquisition unit 21, may be realized by a circuit in which processing circuits are distributed, or the functions of each unit may be realized together by a single processing circuit.

[0075] When the processing circuit 81 is a processor, the functions of the application screen image acquisition unit 21 and the like are realized in combination with software and the like. The software and the like may include, for example, software, firmware, or software and firmware. The software and the like are written as a program and stored in a memory. As shown in FIG. 13 , the processor 82 applied to the processing circuit 81 realizes the functions of each unit by reading and executing a program stored in a memory 83. That is, the application verification support device 20 includes a memory 83 for storing a program that, when executed by the processing circuit 81, results in the execution of an application screen image acquisition step, a UI component detection step, a UI component identification step, a UI component input information estimation step, a UI component ID assignment step, a UI component input information generation step, a script file generation step, an application operation step, a screen transition determination step, and a screen transition information generation step. In other words, this program can be said to cause a computer to execute the procedures and methods of the application screen image acquisition unit 21 and the like. Here, the memory 83 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory), a HDD (Hard Disk Drive), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), a drive device for any of these, or any storage medium to be used in the future.

[0076] The above describes a configuration in which each function of the application screen image acquisition unit 21 and the like is realized either by hardware or software, etc. However, the present invention is not limited to this, and a configuration in which part of the application screen image acquisition unit 21 and the like is realized by dedicated hardware and another part is realized by software, etc. For example, the function of the application screen image acquisition unit 21 can be realized by the processing circuitry 81 as dedicated hardware, and the other functions can be realized by the processing circuitry 81 as the processor 82 reading and executing programs stored in the memory 83.

[0077] As described above, the processing circuitry 81 can realize the above-described functions by hardware, software, or a combination of these. Furthermore, although not shown, a verification support system may be realized that includes the application verification support device 20 and a notification device (not shown) that can communicate with the application verification support device 20 and notifies the user of screen transition information by displaying it, etc.

[0078] The contents of the embodiments can be modified or omitted as appropriate.

[0079] The above description is illustrative in all respects and is not restrictive. It is understood that countless variations not illustrated can be envisioned.

[0080] 11 Test target application, 20 Application verification support device, 21 Application screen image acquisition unit, 22 Application operation unit, 31 UI component detection unit, 32 UI component identification unit, 33 UI component input information estimation unit, 34 UI component ID assignment unit, 36 UI component information storage unit, 41 Screen transition determination unit, 42 Screen transition information generation unit, 51 UI component input information generation unit, 52 Script file generation unit.

Claims

1. a screen image acquisition unit for acquiring a screen image of the software under test; a UI component detection unit that detects UI components from the screen image; a UI component identification unit that identifies whether the type of the UI component is a predetermined type; a UI component input information estimation unit that estimates input information that is information to be input to the UI component; a UI component ID assigning unit that associates the UI component with a UI component in the screen image of the software by an ID; a UI component input information generation unit that generates the input information based on the type of the UI component, the estimation result of the input information, and the ID; a script file generating unit that generates a script file for operating the software based on the generated input information; a software operation unit that operates the software using the script file; a screen transition determination unit that determines whether or not the screen image has transitioned due to an operation of the software; a screen transition information generating unit that generates screen transition information by linking the determination result regarding the transition of the screen image with information of the script file used to operate the software; A verification support device comprising:

2. 2. The verification support device according to claim 1, The predetermined types include types of the UI components whose appearances become identifiable by operations on the UI components.

3. 3. The verification support device according to claim 1, The verification support device further includes a UI component information storage unit that stores the type of the UI component, the estimation result of the input information, and the ID in association with each other.

4. 3. The verification support device according to claim 1, The UI component input information generation unit a verification support device that generates the input information based on the type of the UI component, the estimation result of the input information, the ID, and a constraint imposed on the input information;

5. a verification support device according to claim 1 or 2; a notification device that can communicate with the verification support device and notifies the screen transition information; A verification support system comprising:

6. Capture a screen image of the software under test, Detecting UI components from the screen image; Identifying whether the type of the UI component is a predetermined type; Inferring input information, which is information to be input to the UI component; The UI component is associated with a UI component in the screen image of the software by an ID; generating the input information based on the type of the UI component, the estimation result of the input information, and the ID; generating a script file for operating the software based on the generated input information; operating the software using the script file; determining whether the screen image has transitioned due to an operation of the software; A verification support method that generates screen transition information by linking the determination result regarding the transition of the screen image with information on the script file used to operate the software.