Program and information processing apparatus

Abstract

To provide a program which can automatically and timely generate screen transition data fo a target program and can realize prompt execution of testing.SOLUTION: A computer executes the steps of: detecting a kind of an object capable of being operated and included in a GUI screen displayed by execution of a target program; acquiring data relating to transition of a screen displayed when a detected object is operated; associating a GUI screen before transition, the kind of the object in the GUI screen and the data relating to the transition of the screen obtained when the object is operated to generate screen transition data for the target program; and comparing the image transition data obtained before updating and the image transition data obtained after updating, and if the comparison result shows any difference in transition of the GUI screen, displaying the difference on a display unit in a form different from the ordinary transition in the screen transition diagram.SELECTED DRAWING: Figure 21

Description

[Technical field]

[0001] The present invention relates to a program and an information processing device for verifying the operation of a program that transitionally displays a plurality of screens in response to an input command. [Background technology]

[0002] For example, Patent Document 1 discloses a method of acquiring image data of a screen displayed on a display of a target device, analyzing the acquired image data to extract one or more buttons contained in the screen, generating a signal in the target device when the extracted button is operated, recording screen transition data including image data before and after the screen transition caused by the button operation and data related to the operated button, and generating a screen transition diagram using the screen transition data.

[0003] Patent Document 2 discloses a method for automatically generating a test scenario by reading design documents, correcting the design documents using error information output to the scenario generator, and automatically generating a test scenario based on the corrected design documents. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-15476 [Patent Document 2] JP 2020-204847 A Summary of the Invention [Problem to be solved by the invention]

[0005] Even if there are no changes to the specifications of the program being tested, the program may no longer function properly if the version of the OS (Operating System) in which the program runs is updated to a newer version. For example, a program that worked properly on an older version of the OS may no longer function on a newer version of the OS, and may not transition to a specified screen even when a specified button is pressed.

[0006] Furthermore, when the specifications of the program to be tested are updated and a specific GUI button is added or deleted, the screen transitions often change between before and after the specification update.

[0007] Conventionally, there has been no function that clearly shows how screen transitions have changed as a result of updating the execution environment of a target program or updating the specifications of the target program.

[0008] In one aspect, the present disclosure aims to provide a program or the like that can automatically generate screen transition data for a target program as appropriate and realize rapid testing. [Means for solving the problem]

[0009] (1) A program according to an embodiment of the present disclosure causes a computer to execute a process including: an execution step of executing a target program accompanied by the display of a GUI screen; a detection step of detecting a type of an object that can be operated included in the GUI screen displayed by the execution of the target program; an acquisition step of acquiring data regarding a screen transition that is displayed when the detected object is operated; a creation step of creating screen transition data for the target program by associating the GUI screen before the transition, the type of object in the GUI screen, and data regarding the screen transition when the object is operated; a comparison step of comparing pre-update image transition data, which is screen transition data created before the target program or the execution environment of the target program is updated, with post-update image transition data, which is screen transition data created after the target program or the execution environment of the target program is updated; and a display step, if there are differences in the GUI screen transitions as a result of the comparison, of displaying the differences on a screen transition diagram based on the pre-update image transition data or a screen transition diagram based on the post-update image transition data on a display unit in a manner different from normal transitions.

[0010] (2) In the program of (1), the comparison step may calculate a similarity between a GUI screen included in the pre-update image transition data and a GUI screen included in the corresponding post-update image transition data, and compare whether the calculated similarity is less than a predetermined value. If the calculated similarity is less than the predetermined value, the display step may display the GUI screen on the display unit in a manner different from that of a GUI screen that does not have differences, as being a difference between the GUI screens included in the pre-update image transition data and the corresponding post-update image transition data.

[0011] (3) In the program of (1), the comparison step calculates a similarity between a sound signal output in a GUI screen included in the pre-update image transition data and a sound signal output in a GUI screen included in the corresponding post-update image transition data, and compares whether the calculated similarity is less than a predetermined value. If the calculated similarity is less than the predetermined value, the display step may display the GUI screen on the display unit in a manner different from that of a GUI screen that does not have differences in the screen transition diagram, as a result of the GUI screen being considered to have differences.

[0012] (4) An information processing device according to an embodiment of the present disclosure includes a control unit that executes a target program accompanied by display of a GUI screen. The control unit executes the following processes: detecting a type of an operable object included in a GUI screen displayed by execution of the target program, acquiring data on a screen transition displayed when the detected object is operated, creating screen transition data for the target program by associating a GUI screen before the transition, a type of an object in the GUI screen, and data on a screen transition when the object is operated, comparing pre-update image transition data, which is screen transition data created before the target program or the execution environment of the target program is updated, with post-update image transition data, which is screen transition data created after the target program or the execution environment of the target program is updated, and, if there are differences in the transition of the GUI screen as a result of the comparison, displaying the differences on a screen transition diagram based on the pre-update image transition data or a screen transition diagram based on the post-update image transition data on the display unit in a manner different from normal transitions. Effect of the Invention

[0013] According to one aspect, screen transition data for the target program is automatically created as appropriate, and differences due to updates to the execution environment of the target program or updates to the specifications of the target program are displayed on a display unit, thereby enabling rapid testing to be performed. [Brief description of the drawings]

[0014] [Figure 1]1 is a block diagram showing an example of the configuration of an information processing device; [Diagram 2] 11 is an explanatory diagram showing an example of a record layout of an object DB and a screen transition DB. FIG. [Diagram 3] FIG. 1 is an explanatory diagram of a learning model. [Figure 4] 13 is a flowchart illustrating an example of a learning model generation processing procedure. [Diagram 5] 13 is a flowchart showing an example of a procedure for creating a screen transition DB. [Figure 6] 13 is a flowchart showing an example of a procedure for creating a screen transition DB. [Figure 7] FIG. 13 is an explanatory diagram showing a state in the middle of creating a screen transition DB. [Figure 8] FIG. 13 is an explanatory diagram showing a state in the middle of creating a screen transition DB. [Figure 9] 13 is a flowchart illustrating an example of a verification process procedure for a target program. [Figure 10] FIG. 13 is an explanatory diagram showing an example of a screen. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of an information processing device according to a second embodiment. [Figure 12] FIG. 13 is an explanatory diagram illustrating an example of a record layout of a screen transition DB according to the second embodiment. [Figure 13] 13 is a flowchart showing an example of a screen transition DB creation processing procedure according to the second embodiment. [Figure 14] 13 is a flowchart showing an example of a screen transition DB creation processing procedure according to the second embodiment. [Figure 15] FIG. 13 is an explanatory diagram illustrating an example of a record layout of a screen transition DB according to the third embodiment. [Figure 16] 13 is a flowchart showing an example of a screen transition DB creation processing procedure according to the third embodiment. [Figure 17] 13 is a flowchart showing an example of a screen transition DB creation processing procedure according to the third embodiment. [Figure 18] FIG. 13 is an explanatory diagram showing a modified example of an object DB. [Figure 19]13 is an example of a transition diagram before updating in the fourth embodiment. [Figure 20] 13 is an example of a transition diagram after updating in the fourth embodiment. [Figure 21] 13 is an example of a transition diagram after updating according to the fourth embodiment, showing differences. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the program, information processing method, and model generation method of the present disclosure will be specifically described with reference to the drawings showing embodiments thereof.

[0016] (Embodiment 1) In this embodiment, an information processing device that performs operation verification of a GUI program to be developed will be described. The application program to be verified (hereinafter referred to as the target program) is software for transitionally displaying a plurality of GUI screens in response to input commands, and may be any program that involves displaying a GUI screen. For example, the application program to be verified may be an application program that operates on a device that has a display or a device that can be connected to a display, such as a personal computer, a smartphone, a tablet terminal, or a game terminal.

[0017] The GUI screen in this embodiment is configured in a hierarchical structure having multiple layers, and by operating an object (icon, button, etc.) in the GUI screen, the screen transitions to a GUI screen linked to the object (the GUI screen one layer lower). Also, each GUI screen is provided with a Return object for returning to the screen before the transition, and by operating the Return object, the screen transitions to the link source GUI screen (the GUI screen one layer higher).

[0018] 1 is a block diagram showing an example of the configuration of an information processing device 10. The information processing device 10 is a device capable of various information processes and transmitting and receiving information, and is a server computer, a personal computer, or the like.

[0019] The information processing device 10 is managed, for example, by a device manufacturer of a device in which an application program to be developed is installed. When the target program is updated by a developer, the device manufacturer uses the information processing device 10 to verify the operation of the target program.

[0020] Furthermore, if the device employs an open source OS, and the OS is updated, the operation verification of the target program is also performed by the information processing device 10. Note that the information processing device 10 may be a device on the developer's side, and the operation verification of the target program may be performed on the developer's side.

[0021] The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (semiconductor for AI).

[0022] The control unit 11 uses built-in memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory) to appropriately execute programs P and the like stored in the storage unit 12, thereby executing various information processing and control processing to be performed by the information processing device 10. The control unit 11 may be configured as a single piece of hardware (SoC: System On a Chip) that integrates a processor, a memory, a communication device, and the like.

[0023] The storage unit 12 includes a hard disk, a solid state drive (SSD), a flash memory, etc. The storage unit 12 stores a program P (program product) executed by the control unit 11 and various data required for the execution of the program P.

[0024] The storage unit 12 also stores a learning model M that has learned training data by, for example, machine learning. The learning model M is a learned model that has been trained to detect an operable object included in a GUI screen when the GUI screen is input.

[0025] The learning model M is expected to be used as a program module constituting artificial intelligence software. The learning model M performs a predetermined calculation on input values ​​and outputs the calculation results, and the memory unit 12 stores data such as coefficients and thresholds of functions that define this calculation as the learning model M.

[0026] The storage unit 12 also stores a screen transition creation program P1 and a screen transition verification program P2. The screen transition creation program P1 is a program for realizing a process of creating screen transition data (screen transition DB 12b described below) used for verifying the operation of a target program based on the target program. The screen transition verification program P2 is a program for realizing a simulation of a GUI screen output by execution of a target program based on the updated target program when the target program or the execution environment of the target program is updated. The screen transition verification program P2 is also called an emulator.

[0027] A part or all of the programs P, P1, P2 stored in the storage unit 12 may be written to the storage unit 12 during the manufacturing stage of the information processing device 10, or the control unit 11 may download them from another device via the communication unit 13 and store them in the storage unit 12. The storage unit 12 further stores an object DB 12a and a screen transition DB 12b, which will be described later. A part or all of the learning model M, the object DB 12a, and the screen transition DB 12b may be stored in another storage device connected to the information processing device 10, or in another storage device with which the information processing device 10 can communicate.

[0028] The communication unit 13 is a communication module for performing processes related to wired or wireless communication, and transmits and receives information to and from other devices via a network. The network may be the Internet or a public telephone line network, or may be a LAN (Local Area Network) constructed within the facility where the information processing device 10 is installed.

[0029] The input unit 14 receives an operation input by a user, and sends a control signal corresponding to the operation content to the control unit 11.

[0030] The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit, or the touch panel may be configured to be externally attached to the information processing device 10.

[0031] The reading unit 16 reads information stored in the portable storage medium 10a, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD card, a micro SD card, a Compact Flash (registered trademark), etc. The control unit 11 may read a part or all of the programs P, P1, P2 stored in the storage unit 12 from the portable storage medium 10a via the reading unit 16 and store them in the storage unit 12.

[0032] In this embodiment, the information processing device 10 may be a multi-computer consisting of multiple computers, may be a virtual machine virtually constructed by software, or may be a cloud server. The information processing device 10 does not necessarily include the input unit 14 and the display unit 15, and may be configured to accept operations through a connected computer or to output information to be displayed to an external display device. The programs P, P1, and P2 may be executed on a single computer, or may be executed on multiple computers connected to each other via a network.

[0033] Fig. 2 is an explanatory diagram showing an example of the record layout of the object DB 12a and the screen transition DB 12b. Fig. 2A shows the object DB 12a, and Fig. 2B shows the screen transition DB 12b. The object DB 12a is a database that stores information about operable objects that may be displayed on a GUI screen.

[0034] The object DB12a shown in FIG. 2A includes an object ID string, an object name string, a processing instruction string, and a priority string, and stores the processing instruction (e.g., a command) and priority assigned to each object in association with the object ID uniquely assigned to each object and the object name given to each object.

[0035] In the example of FIG. 2A, the priorities 1, 2, 3, and 4 are assigned to each object, but the configuration may be such that the larger the value, the higher the priority. Also, it is not necessary to assign priorities to all objects. Note that the graphic data of each object is assigned an object ID or object name and stored in a predetermined area (e.g., an object folder prepared in advance) of the storage unit 12. The object DB 12a may be provided for each target program, and in this case, it is stored in the storage unit 12 in association with the program ID assigned to the target program.

[0036] The screen transition DB 12b is a database in which information indicating the transition state of a GUI screen displayed when a target program is executed is registered. The screen transition DB 12b is a DB created by the control unit 11 executing the screen transition creation program P1, and is created for each target program and stored in the storage unit 12 in association with the program ID assigned to the target program.

[0037] The screen transition DB 12b is created when a verification process of a target program is performed, and therefore may be stored in the storage unit 12 in association with a program ID and a process ID assigned to the verification process.

[0038] The screen transition DB 12b shown in FIG. 2B includes a pre-transition screen name column, an object name column, a number of operations column, and a post-transition screen name column, and stores the object names of objects displayed in the GUI screens, the number of operations of each object, and the names of GUI screens (post-transition screens) to which the transition occurs when each object is operated, in association with the names of each GUI screen (pre-transition screen) that is displayed when the target program is executed.

[0039] The screen data of each GUI screen is stored in a predetermined area of ​​the storage unit 12 (for example, a screen DB prepared in advance) in association with the screen name.

[0040] In the screen transition DB12b shown in FIG. 2B, objects with object names A, B, and C are displayed on the HOME screen, and it can be seen that when object A is operated, a transition to screen A occurs, and when object B is operated, a transition to screen B occurs.

[0041] Also, when object C is operated on the HOME screen, the first operation transitions to screen C, and the second operation transitions to screen C-1.

[0042] In addition, screen A displays a Return object and an object named A-1, and it can be seen that when the Return object is operated, the screen transitions to the HOME screen, and when the A-1 object is operated, the screen transitions to screen A-1.

[0043] Fig. 3 is an explanatory diagram of learning model M, Fig. 3A shows an example of the configuration of learning model M, and Fig. 3B shows an example of training data used for learning learning model M. Learning model M is trained to receive an input of a screen image of a GUI screen, perform a calculation to recognize an object to be operated on the GUI screen based on the input GUI screen, and output the recognition result.

[0044] The objects to be recognized include a Return object, a Menu object, a Delete object, and a Home object as shown in Fig. 3B, and are registered in advance in the object folder and the object DB 12a. The learning model M may be a model that realizes single-label classification that recognizes one type of object included in a GUI screen, or may be a model that realizes multi-label classification that recognizes multiple types of objects.

[0045] The learning model M shown in Fig. 3A shows a model that realizes multi-label classification. The learning model M can be configured using object detection algorithms such as YOLO (You Only Look Once), R-CNN (Regions with Convolution Neural Network), SSD (Single Shot Multibook Detector), etc., and may be configured by combining multiple algorithms.

[0046] The learning model M may be a model that recognizes objects in an input GUI screen on a pixel-by-pixel basis by semantic segmentation. In this case, the learning model M can be configured using algorithms such as U-Net, FCN (Fully Convolutional Network), and SegNet.

[0047] The learning model M has an input layer to which a GUI screen is input, an intermediate layer that extracts features from the input GUI screen, and an output layer that outputs an image in which objects in the GUI screen are detected based on the calculation results of the intermediate layer.

[0048] The intermediate layer uses various functions, thresholds, etc. to calculate output values ​​based on the GUI screen input from the input layer.

[0049] The output layer outputs, for the input GUI screen, an image (hereinafter referred to as a label image) that includes a bounding box (a rectangle indicated by a dashed line in Figure 3A) surrounding the detected object, a discrimination label indicating the type of the detected object, and a confidence level for the discrimination label.

[0050] With this configuration, when a GUI screen is input, the learning model M adds a bounding box to the object in the GUI screen and outputs a label image (information about the object) that displays the object name and confidence level of the recognition result for the object.

[0051] The learning model M can be generated by machine learning using training data that includes a training GUI screen and images (correct label images) in which objects in the GUI screen are marked (bounding boxes) and each mark is associated with an object name (correct label).

[0052] The training data is generated by associating an image (correct label image) in which the annotation technician adds marks (bounding boxes) and object names (correct labels) to the objects in the GUI screen with the GUI screen.

[0053] The training GUI screen may be an image of an object as shown in Fig. 3B, and in this case, the image of the object and the name of the object (correct label) can be used as training data. In the example shown in Fig. 3B, objects with the same function (processing command) associated with different graphic designs are shown, and in this embodiment, training data in which the same object name is associated with objects with the same function (processing command) associated with different graphic designs is used.

[0054] The learning model M learns to output a correct labeled image included in the training data when a GUI screen included in the training data is input. In the learning process, the learning model M performs calculations in the intermediate layer and the output layer based on the input GUI screen, and calculates a labeled image to be output from the output layer.

[0055] The learning model M compares the calculated label image with the correct label image, and optimizes parameters used in the calculation process in the intermediate layer and the output layer so that the two are close to each other. Specifically, the learning model M compares the discrimination label and confidence for the object detected in the calculated label image with a value corresponding to the correct label in the correct label image (1 for the correct object, 0 for other objects), and optimizes the two so that they are close to each other. The parameters are weights (coupling coefficients) between nodes in the intermediate layer and the output layer, etc. The method of optimizing the parameters is not particularly limited, but the backpropagation method, the steepest descent method, etc. can be used.

[0056] This results in a learning model M that, when a GUI screen is input, determines the type of object in the GUI screen and outputs a label image indicating the determination result (bounding box, discrimination label, and confidence level).

[0057] As shown in Fig. 3B, the learning model M of this embodiment learns objects associated with the same function (processing command) as objects of the same type, even if they have different graphic designs. Therefore, even if an unlearned graphic design is input, the learning model M can identify an appropriate object (similar object) from among the learned objects.

[0058] The information processing device 10 prepares the learning model M as described above in advance, and uses it when generating image transition data for a target program and when performing a verification process for the target program. Learning of the learning model M may be performed by another learning device. The trained learning model M generated by learning on the other learning device is downloaded from the learning device to the information processing device 10 via a network or a portable storage medium 10a, for example, and stored in the storage unit 12.

[0059] The generation process of the learning model M will be described below. FIG. 4 is a flowchart showing an example of the generation process procedure of the learning model M. "S" in each flowchart used in the following description indicates a step. The following process is executed by the control unit 11 of the information processing device 10 in accordance with the program P stored in the storage unit 12, but may be executed by another learning device. It is assumed that the above-mentioned training data is generated in advance and stored in a predetermined area (a predetermined DB) of the storage unit 12.

[0060] The control unit 11 acquires one of the training data stored in the storage unit 12 (S11). Then, the control unit 11 performs a learning process for the learning model M based on the acquired training data (S12).

[0061] Here, the control unit 11 inputs a GUI screen included in the training data to the learning model M, and obtains an output value (label image) output from the learning model M in response to the input of the GUI screen. The control unit 11 compares the label image output from the learning model M with a correct label image included in the training data, and trains the learning model M so that the two are close to each other. In the learning process, the learning model M optimizes parameters used in the calculation process in the intermediate layer and the output layer, for example, using an error backpropagation method that sequentially updates the parameters from the output layer to the input layer.

[0062] The control unit 11 determines whether or not there is unprocessed training data that has not been subjected to learning processing among the training data stored in the storage unit 12 (S13). When the control unit 11 determines that there is unprocessed training data (S13: YES), the control unit 11 returns to S11 and performs the processing of S11 to S12 on the training data that has not been subjected to learning processing.

[0063] When the control unit 11 determines that there is no unprocessed training data (S13: NO), the control unit 11 ends the series of processes. By the above-mentioned learning process, when a GUI screen is input, a learning model M is generated that outputs a label image in which objects in the GUI screen are surrounded by bounding boxes and a discrimination label and a confidence factor are added to each bounding box.

[0064] In addition, a learning model M that has already been trained can also be retrained by performing the above-mentioned processing, in which case a learning model M with higher discrimination accuracy can be generated.

[0065] The following describes the processing performed by the information processing device 10. The information processing device 10 of this embodiment executes a screen transition creation program P1 to create a screen transition DB 12b in a target program, and executes a screen transition verification program P2 to verify a screen transition state in the target program.

[0066] The information processing device 10 regards the screen transition DB 12b created based on the target program released initially as the correct screen transition data (expected value in the verification process). When the target program or the execution environment of the target program is updated, the information processing device 10 creates the updated screen transition DB 12b based on the target program and performs the verification process by comparing the updated screen transition DB 12b with the correct screen transition DB 12b.

[0067] The target program used when creating the correct screen transition DB 12b may be the first delivered target program in addition to the first released target program.

[0068] The process of creating the screen transition DB 12b in the target program will be described below. Figures 5 and 6 are flowcharts showing an example of the process of creating the screen transition DB 12b, and Figures 7 and 8 are explanatory diagrams showing the state of the screen transition DB 12b during its creation.

[0069] After a target program developed and created by a developer is released for the first time, the information processing device 10 performs the following process to create a screen transition DB 12b that stores screen transition data in the target program.

[0070] In addition, the released target program is in a state where bugs and the like have been eliminated, and the screen transition DB 12b created based on this target program can be regarded as correct screen transition data.

[0071] The control unit 11 of the information processing device 10 starts the screen transition creation program P1, and executes a target program to be processed here in response to the start of the screen transition creation program P1 (S21).

[0072] The control unit 11 executes the target program to generate and capture a GUI screen to be displayed (S22).

[0073] The control unit 11 captures the GUI screen to be displayed, for example, by using a screen capture (screen shot) function. Note that the control unit 11 may capture the GUI screen by executing the target program to display the GUI screen to be displayed on the display unit 15, and acquiring an image obtained by photographing the GUI screen displayed on the display unit 15 with a camera.

[0074] The control unit 11 searches the pre-transition screens stored in the screen transition DB 12b being created to see whether there is a screen identical to the captured GUI screen (S23).

[0075] For example, the control unit 11 reads out from the memory unit 12 the GUI screen stored in the screen transition DB 12b at this point as a pre-transition screen, calculates the similarity between each of the read GUI screens and the captured GUI screen, and if the maximum similarity is equal to or greater than a threshold value, determines that there are identical screens, and identifies the GUI screen with the calculated maximum similarity as the identical screen.

[0076] The similarity can be, for example, a correlation coefficient or a cosine similarity. The control unit 11 may be configured to estimate the similarity between two GUI screens using a learning model constructed by machine learning.

[0077] For example, the learning model may be configured with a CNN (Convolutional Neural Network), and may be used that has been trained to output the similarity between two GUI screens when two GUI screens are input. In this case, the control unit 11 inputs the two GUI screens to the trained learning model, and can estimate the similarity between the two GUI screens based on the output information from the learning model. Note that the threshold value as a criterion for determining whether the two screens are the same may be changeable for each target program or according to the contents of the verification process.

[0078] The control unit 11 determines whether or not the same screen is found as a result of the search (S24), and if it determines that the same screen is not found (S24: NO), it registers the current GUI screen (here, the GUI screen captured in S22) in the screen transition DB 12b as the pre-transition screen (S25).

[0079] Here, the control unit 11 assigns a screen name to the current GUI screen, stores the assigned screen name in the pre-transition screen name column of the screen transition DB 12b, and gives the screen name to the image data of the current GUI screen and stores it in a specified area (screen DB) of the memory unit 12.

[0080] Then, the control unit 11 recognizes operable objects in the current GUI screen (S26). Specifically, the control unit 11 inputs the current GUI screen to the learning model M and obtains a label image output from the learning model M.

[0081] For example, when a GUI screen including objects A, B, and C is input to the learning model M as shown in FIG. 7A, the control unit 11 acquires a label image showing each object as a bounding box.

[0082] In this embodiment, since the processing command for each object is stored in the object DB 12a, the control unit 11 can use the learning model M to detect not only the name of the object on the GUI screen but also the processing command (type) of each object.

[0083] Furthermore, the control unit 11 may recognize an object whose graphic design includes predetermined text by extracting text from a GUI screen using an OCR (Optical Character Recognition) function in addition to image recognition using the learning model M. For example, in the case of a Menu object whose graphic design includes the text "MENU" as shown in Fig. 3B, the Menu object can be recognized by extracting the text "MENU" using the OCR function.

[0084] The control unit 11 registers each recognized object in the screen transition DB 12b (S27).

[0085] Here, the control unit 11 stores the object name (distinguishing label) of each recognized object in the object name column of the screen transition DB 12b, assigns the object name to the graphic data of each object, and stores it in a predetermined area (object folder) of the memory unit 12.

[0086] 7A shows the stored contents of the screen transition DB 12b when the name of the HOME screen is assigned to the current GUI screen, and objects with object names A, B, and C are recognized on the HOME screen. At this point, the control unit 11 stores 0 in the number of operations for each object.

[0087] As a result of the search, if it is determined that the same screen is present (S24: YES), the current GUI screen has already been registered in the screen transition DB 12b, so the control unit 11 skips S25 to S27 and proceeds to S28.

[0088] The control unit 11 refers to the screen transition DB 12b being created and determines whether or not the objects in the current GUI screen include an object with an operation count of 0 (S28). If it is determined that there is an object with an operation count of 0 (S28: YES), the control unit 11 identifies an object with a high priority among the objects with an operation count of 0 (S29).

[0089] Here, the control unit 11 acquires the priority set for each object with a zero number of operations from the object DB 12a, and identifies the object with the highest priority.

[0090] Note that the lowest priority may be assigned to an object to which no priority is assigned. Also, in a screen having a plurality of objects as shown in Fig. 7A, when a priority is not assigned to each object on the screen, the control unit 11 identifies one arbitrary object. In this case, the control unit 11 may take into consideration, for example, the display position (arrangement position) of each object on the screen, or may identify one arbitrary object in alphabetical order or in hiragana order based on the object name assigned to each object.

[0091] The control unit 11 identifies a processing command (type of object) associated with the identified object (S30). In this embodiment, since the processing command of each object is stored in the object DB 12a, the control unit 11 can identify the processing command of the identified object by the object DB 12a.

[0092] Then, the control unit 11 operates the specified object (S31) and executes a processing command associated with the object. Note that the object operation here is a software process, and the control unit 11 accepts a command (control signal) sent when the specified object is operated, and executes a processing command corresponding to the command.

[0093] Moreover, the operation is, for example, a tap, a double tap, a swipe, etc., and a preset operation is performed. Also, the type of operation may be different for each object. For example, the type of operation to be performed on each object is registered in association with each object, and the control unit 11 receives a command corresponding to each object and the type of operation for each object, thereby making it possible to execute a processing command corresponding to the operation content (type of operation) for the object.

[0094] The control unit 11 executes a processing command associated with the object to generate a GUI screen to be displayed next (a GUI screen after transition) and captures it (S32).

[0095] The GUI screen after the transition may be a completely different screen from the GUI screen before the transition, or only a part of the GUI screen before the transition may be different. The process of capturing the GUI screen is the same as that of S22.

[0096] The control unit 11 stores the GUI screen captured in S32 in the screen transition DB 12b as a post-transition screen in association with the pre-transition GUI screen and the object that was the object of operation (S33). Here, the control unit 11 assigns a screen name to the captured GUI screen, stores the assigned screen name in a post-transition screen name column in the screen transition DB 12b, and gives the screen name to the captured GUI screen and stores it in a predetermined area (screen DB) of the storage unit 12.

[0097] Moreover, the control unit 11 updates the number of operations for this object to 1. This enables the control unit 11 to create screen transition data including a GUI screen before the transition, an object in the GUI screen, and a GUI screen after the transition when the object is operated, and to store the data in the screen transition DB 12 b.

[0098] Then, the control unit 11 proceeds to S23, and searches whether there is a screen identical to the current GUI screen (here, the GUI screen captured in S32) (S23), and if it is determined that there is no identical screen (S24: NO), it registers the current GUI screen as a pre-transition screen in the screen transition DB 12b (S25).

[0099] Furthermore, the control unit 11 recognizes operable objects in the current GUI screen (S26), and registers each recognized object in the screen transition DB 12b (S27). In the process up to this point, for example, when object A in the HOME screen is operated to transition to screen A, and the screen A displays the Return object and object A-1, the screen transition data is stored in the screen transition DB 12b as shown in FIG. 7B.

[0100] The control unit 11 repeats S23 to S33 until it determines that there is no object in the current GUI screen whose operation count is 0. In this way, for example, a screen transition DB 12b as shown in FIG.

[0101] FIG. 8 shows the stored contents of screen transition DB12b when object A in the HOME screen is operated to transition to screen A, the Return object with the highest priority on screen A is operated to return to the HOME screen, object B in the HOME screen is operated to transition to screen B, the Return object with the highest priority on screen B is operated to return to the HOME screen, object C in the HOME screen is operated to transition to screen C, and the Return object with the highest priority on screen C is operated to return to the HOME screen.

[0102] When the control unit 11 determines that there is no object with an operation count of 0 among the objects in the current GUI screen (S28: NO), it determines whether or not there is an object with an operation count of 0 in a GUI screen to be transitioned after operating any object in the current GUI screen (S34). That is, it determines whether or not there is an object with an operation count of 0 in a GUI screen lower than the current GUI screen.

[0103] In the screen transition DB 12b shown in FIG. 8, there is no object with an operation count of 0 on the HOME screen, but there are objects with an operation count of 0 on screens A, B, and C to which a transition can be made from the HOME screen. When the control unit 11 determines that the post-transition GUI screen has an object with zero operation count (S34: YES), it operates the object for transitioning to the post-transition GUI screen, and transitions to the post-transition GUI screen (S35).

[0104] For example, the control unit 11 operates an object A in the HOME screen to transition to screen A. Specifically, the control unit 11 identifies a processing command associated with object A, and operates object A to execute the processing command associated with object A, and generates screen A to be displayed next (GUI screen after transition). Note that the control unit 11 may transition to screen B or screen C instead of screen A, or may transition to a screen with a large number of objects with a zero number of operations.

[0105] The control unit 11 proceeds to S33, where it stores the GUI screen after the transition in the screen transition DB 12b in association with the GUI screen before the transition and the operated object (S33). Here, the control unit 11 adds a record corresponding to this object to the screen transition DB 12b, and stores the operation count obtained by adding 1 to the most recent (maximum) operation count of this object and the GUI screen after the transition in the new record. That is, if this object is operated for the second time, the control unit 11 stores the GUI screen after the transition in association with the operation count 2. After that, the control unit 11 proceeds to S23.

[0106] If it is determined that there is no object with an operation count of 0 on the GUI screen after the transition (S34: NO), that is, if there is no object with an operation count of 0 on a GUI screen lower than the current GUI screen, the control unit 11 determines whether or not there is a GUI screen (the screen one level above) before the transition of the current GUI screen (S36).

[0107] When it is determined that there is a GUI screen before the transition (S36: YES), the control unit 11 operates the Return object to transition (return) to the GUI screen before the transition (the screen one layer above) (S37).

[0108] Here, the control unit 11 identifies the processing command associated with the Return object, and by manipulating the Return object, executes the processing command associated with the Return object, and generates the GUI screen to be displayed next (the GUI screen before the transition). Then, the control unit 11 proceeds to S28 and determines whether or not there is an object with a number of operations of 0 on the current GUI screen (S28), and repeats S28, S34, and S36 to S37 until it determines that there is an object with a number of operations of 0 on the current GUI screen or on a GUI screen lower than the current GUI screen, and transitions (returns) to the GUI screen on the upper layer.

[0109] If the control unit 11 determines that there is an object with a count of 0 on the current GUI screen (S28: YES), it proceeds to S29, and if it determines that there is an object with a count of 0 on the transitioned GUI screen (S34: YES), it proceeds to S35.

[0110] If it is determined in S36 that there is no GUI screen before the transition (S36: NO), that is, if the current GUI screen becomes the topmost GUI screen (e.g., the HOME screen) as a result of transition to a GUI screen in a higher layer, the control unit 11 ends the series of processes. The screen transition DB 12b created at this point becomes the correct (expected) screen transition data to be used for verifying the operation of the target program when the target program or the execution environment is updated.

[0111] In the above-described process, the screen transition DB 12b is created when each object in the GUI screen is operated at least once, but it may be configured to create the screen transition DB 12b when each object in the GUI screen is operated at least twice. In this case, in S28, the control unit 11 may determine whether or not there is an object in the current GUI screen that has been operated less than twice.

[0112] According to the above-mentioned process, in this embodiment, it is possible to automatically create an image transition DB12b (image transition data, a test script showing a screen transition state) obtained by actually running the target program. Also, the screen transition DB12b created for the first time for the target program, for example, the screen transition DB12b created based on the target program released for the first time, is used as a correct answer (expected value) in the operation verification process of the target program that is subsequently updated. This makes it possible to carry out the operation verification process without the work of creating the screen transition DB12b (test script) from the device specifications.

[0113] In addition, when the screen transition DB 12b is created for the first time for the target program, the person in charge of verification checks whether the screen transition data for each screen registered in the screen transition DB 12b is correct, and by appropriately correcting any incorrect screen transition data, it can be used as the correct answer in the operation verification process.

[0114] Next, a verification process using the screen transition DB 12b (hereinafter referred to as the correct screen transition DB 12b) created for the first time based on the target program by the above-mentioned process will be described.

[0115] When the target program for which the correct screen transition DB 12b was created is updated, or when the execution environment of the target program (e.g., the OS) is updated, the information processing device 10 executes the screen transition creation program P1 again to create the updated screen transition DB 12b. Then, the information processing device 10 compares the created updated screen transition DB 12b with the correct screen transition DB 12b, and presents the location where the difference occurs as the location where the problem occurs.

[0116] Fig. 9 is a flowchart showing an example of a verification process procedure for a target program, and Fig. 10 is an explanatory diagram showing an example screen. When the target program for which the correct screen transition DB 12b has been created by the processes shown in Figs. 5 and 6 is updated, or when the execution environment (e.g., the OS) of the target program is updated, the control unit 11 of the information processing device 10 performs the following verification process.

[0117] The control unit 11 of the information processing device 10 determines whether or not either the target program or the execution environment of the target program has been updated (S41). The target program is determined to have been updated when, for example, an updated target program is delivered by a developer and stored in a predetermined area (for example, a folder prepared in advance) of the storage unit 12. In addition, updates to the target program include not only upgrades made by the developer but also automatic upgrades of the target program.

[0118] The execution environment is determined to have been updated when, for example, the execution environment (eg, OS) of the device on which the target program is installed is upgraded. The update status of the target program and the execution environment may be received via the input unit 14 or the communication unit 13. When it is determined that both the target program and the execution environment have not been updated (S41: NO), the control unit 11 waits until either one is updated.

[0119] When it is determined that either the target program or the execution environment has been updated (S41: YES), the control unit 11 executes a process of creating the screen transition DB 12b for the updated target program (S42).

[0120] The processing of S42 is a process of creating the screen transition DB 12b shown in Figures 5 and 6, and the control unit 11 starts the screen transition creation program P1 and creates an updated screen transition DB 12b in which screen transition data is accumulated by executing the target program to be processed in response to the start of the screen transition creation program P1.

[0121] Then, the control unit 11 compares the correct screen transition DB 12b created in advance with the updated screen transition DB 12b created in S42, performs video verification on the screen transition data in the updated screen transition DB 12b, and determines whether or not there is a defect.

[0122] Specifically, the control unit 11 reads one piece of screen transition data from the updated screen transition DB 12b (S43). Here, the control unit 11 reads out, as the screen transition data, the pre-transition screen name, the object name, the number of operations, and the post-transition screen name stored in one record of the updated screen transition DB 12b.

[0123] Next, the control unit 11 reads out the correct post-transition screen corresponding to the pre-transition screen name, object name, and number of operations in the read screen transition data (S44). Here, the control unit 11 reads out the post-transition screen name (correct post-transition screen name) stored in the correct screen transition DB 12b in association with the pre-transition screen name, object name, and number of operations in the read screen transition data, and reads out the post-transition screen (the correct post-transition GUI screen) stored in the screen DB based on the read correct post-transition screen name.

[0124] The control unit 11 reads out the post-transition screen (post-transition GUI screen) stored in the screen DB based on the post-transition screen name included in the screen transition data read out in S43, and compares the read-out post-transition GUI screen with the correct post-transition GUI screen read out in S44 (S45).

[0125] Here, the control unit 11 calculates the similarity (degree of similarity) between the two GUI screens being compared, and if the similarity is equal to or greater than a threshold, it determines that the GUI screen after the transition is the correct screen (verification result: OK), and if the similarity is less than the threshold (less than a specified value), it determines that the GUI screen after the transition is not the correct screen (verification result: NG).

[0126] Here, the similarity can be calculated using a correlation coefficient, a cosine similarity, or the like, and may be estimated using a learning model constructed by machine learning. Note that a threshold value as a criterion for determining whether the verification result is OK or NG based on the comparison result between the two screens may be changeable depending on, for example, the contents of the verification process.

[0127] For example, during the verification process, the GUI screen after the transition may be in a white screen state, where the entire screen is white, a black screen state, or a blue screen state, where the entire screen is blue. In such a state, the similarity between the GUI screen after the transition and the correct GUI screen is low.

[0128] Therefore, in the case of a verification process that aims to detect such a state as a verification NG result, a small value should be set as the threshold value.Also, even if the difference from the correct GUI screen is small, in the case of a verification process that aims to detect the difference as a verification NG result, a large value should be set as the threshold value because the similarity with the correct GUI screen is high.

[0129] The control unit 11 determines whether the verification result obtained by comparing the post-transition GUI screen included in the screen transition data read out in S43 with the correct post-transition GUI screen is NG (S46), and if it is determined to be NG (S46: YES), it identifies the reason why the verification result is NG (S47).

[0130] Reasons for the verification result being NG include a white screen state, a black screen state, and a blue screen state, as well as a freeze state in which objects on the screen become unresponsive and a protrude text (abnormal character string) state in which text on the screen does not fit within an appropriate range. Therefore, the control unit 11 determines whether the entire GUI screen after the transition is white, black, or blue, and if it determines that it is any of these, it specifies that it is a white screen state, a black screen state, or a blue screen state.

[0131] Furthermore, when the process of switching to the next GUI screen by operating an object on the GUI screen after the transition is repeated (retried) a predetermined number of times or for a predetermined period of time and the retry expires without switching to the next GUI screen, the control unit 11 determines that the state is Freeze. Furthermore, the control unit 11 determines whether the background of a series of text on the GUI screen after the transition is multiple colors or not, and when it is determined that the background is multiple colors, determines that the state is Protrude Text, in which the text protrudes from the text area.

[0132] Generally, a series of text is displayed on the same background, so if the series of text has a background of multiple colors, it can be identified as a Protrude Text state in which the series of text does not fit within the text area.

[0133] The control unit 11 may use an OCR function to read the text in the GUI screen after the transition, and if the text contains words or terms that are not registered in a dictionary prepared in advance, may determine that the state is Protrude Text. The control unit 11 may also be configured to estimate the reason why the verification result is NG by using a learning model constructed by machine learning.

[0134] For example, a learning model may be used that is configured with CNN and that has been trained to output information indicating whether the input GUI screen is in a white screen state, a black screen state, a blue screen state, a freeze state, a protrude text state, etc., when a post-transition GUI screen is input. In this case, the control unit 11 inputs the post-transition GUI screen to the trained learning model, and can estimate whether the post-transition GUI screen is in a state where the verification result is NG and the reason for the NG verification result, based on the output information from the learning model.

[0135] The control unit 11 stores the screen transition data for which the verification result is determined to be NG in a predetermined area (for example, a verification result DB prepared in advance) of the storage unit 12 (S48). Here, the control unit 11 stores the pre-transition screen name, object name, number of operations, and post-transition screen name included in the screen transition data in association with the reason for the verification result being NG.

[0136] Furthermore, the control unit 11 may also store the GUI screen after the correct transition for this screen transition data in association with the screen transition data and the reason for NG. If the control unit 11 determines that the verification result is not NG (S46: NO), it skips S47 to S48.

[0137] The control unit 11 determines whether or not there is any unprocessed screen transition data that has not been subjected to the above-mentioned verification process among the screen transition data stored in the updated screen transition DB 12b created in S42 (S49).

[0138] If it is determined that there is unprocessed screen transition data (S49: YES), the control unit 11 returns to S43 and performs the processes of S43 to S48 on the screen transition data for which the verification process has not been performed. If it is determined that there is no unprocessed screen transition data (S49: NO), the control unit 11 ends the series of verification processes and generates a verification result screen based on the verification results accumulated in the verification result DB (S50).

[0139] Fig. 10 shows an example of a verification result screen. The screen shown in Fig. 10A displays screen transition data in which, when a screen X is a pre-transition screen and an object Y in the screen X is operated, the screen Y after the transition is different from the correct screen Y, and the verification result is NG.

[0140] In addition, when the GUI screen after the correct transition is stored in the verification result DB, the GUI screen after the correct transition can be displayed in addition to the screen transition data whose verification result is NG, as shown in Fig. 10A. The screen shown in Fig. 10B displays the screen transition data whose verification result is NG, in which the GUI screen after the transition becomes a black screen when the Return object in screen B is operated when screen B is the pre-transition screen.

[0141] The control unit 11 outputs the generated verification result screen (S51) and ends the series of processes. The control unit 11 may store the verification result screen in the storage unit 12, may output it to the display unit 15 for display, or may transmit it to a device specified by the communication unit 13. The control unit 11 may also transmit the verification result screen to a printer with which the information processing device 10 can communicate, via the communication unit 13, and execute a printing process. This makes it possible to provide the results of the verification process by any method.

[0142] The verifier, etc. shall check the screen transition data for which the verification result is NG on the verification result screen, and determine whether the screen transition data that differs from the correct screen transition data is due to the occurrence of a bug or other malfunction, a change in specifications in the updated target program, etc., and take action according to the determination.

[0143] According to the above-described verification process, in this embodiment, when the target program or the execution environment of the target program is updated, the updated screen transition DB12b is automatically created, and a process of verifying whether each screen transition data stored in the updated screen transition DB12b shows an appropriate screen transition based on a comparison with the correct screen transition DB12b (test script) created in advance can be automatically executed. Therefore, even if the target program or the execution environment is frequently improved or upgraded, the creation of the updated screen transition DB12b and the verification process are automatically executed every time the target program or the execution environment is updated, so that the person in charge of verification only needs to check the verification result screen, and an increase in the workload due to the verification process is suppressed.

[0144] In the process of creating the screen transition DB 12b in this embodiment, a priority is set for each object in the GUI screen, and screen transition data is created by operating objects in order of priority, starting from the object with the highest priority, and is stored in the screen transition DB 12b.

[0145] For example, when a high priority is set to the Return object, as shown in FIG. 8, after each object A, B, C in the HOME screen is operated to transition to each screen A, B, C, the Return object can be operated on each screen A, B, C to return to the HOME screen. That is, screen transition data from the HOME screen to the screens A, B, C one level below and screen transition data from the screens A, B, C to the HOME screen are created preferentially, and then screen transition data from the screen A (or B, C) to the screen A-1 (or B-1, C-1) one level below and screen transition data from the screen A-1 (or B-1, C-1) to the screen A (or B, C) are created. According to such a process, in each GUI screen configured in a hierarchical structure, screen transition data can be collected from the upper level to each GUI screen at the same level of depth in order of hierarchical depth.

[0146] By performing this type of processing, for example, when there are a large number of areas that need to be improved or upgraded, or when the quality of the target program is poor, it is possible to verify a wide range of screen transitions resulting from the execution of the target program, starting from the top hierarchical levels and by hierarchical depth, thereby making it possible to efficiently create screen transition data.

[0147] 8, an object other than the Return object (e.g., object A) is operated on the HOME screen to transition to screen A, and then an object other than the Return object (e.g., object A-1) is operated on screen A to transition to screen A-1, which is a lower level. That is, after screen transition data from the HOME screen to screen A, which is one level lower, is created, screen transition data from screen A to screen A-1, which is one level lower, is created.

[0148] According to this process, in each GUI screen configured in a hierarchical structure, it is possible to collect screen transition data in each GUI screen that transitions sequentially in the depth direction when an object in the HOME screen is operated. By performing this process when the quality of the target program is stable to a certain extent, for example, it is possible to focus on verifying screen transitions in the depth direction from an arbitrary GUI screen to the lower layers. Therefore, it is possible to efficiently collect screen transition data in, for example, a place where bugs are likely to occur, a place where a certain function is realized, or a place where verification is desired to focus on.

[0149] As described above, by setting an appropriate priority for each object, it is possible to switch between prioritizing the process of creating screen transition data for each GUI screen for each hierarchy (each hierarchy of the same depth) starting from the top hierarchy of the GUI screen, and prioritizing the process of creating screen transition data for the depth direction of the GUI screen hierarchy.

[0150] Furthermore, in the process of creating screen transition data for each GUI screen for each hierarchy of the same depth as described above, and the process of creating screen transition data for each GUI screen sequentially in the depth direction of the hierarchy, it is possible to transition to each GUI screen for each hierarchy or in the depth direction of the hierarchy, so that it is possible to prevent each GUI screen from being omitted from the creation of screen transition data, and it is possible to create highly accurate screen transition data.

[0151] In contrast, for example, when creating screen transition data for each GUI screen by randomly operating objects in a GUI screen, there is a possibility that bias will occur in the objects that are operated, resulting in objects that are not operated. In this case, there is a possibility that some GUI screens will be omitted from the screen transition data creation targets, which may lead to a decrease in the accuracy of the screen transition data. Note that the priority set for each object may be set manually by a person in charge of the verification process, or may be set automatically according to a preset rule.

[0152] In this embodiment, the process of recognizing objects on a GUI screen is performed by image recognition using a learning model M. This makes it possible to classify an object with an unlearned graphic design as one of the learned objects. For example, the object is classified as an object with a similar graphic design.

[0153] The object recognition process may be performed by, for example, rule-based image recognition. For example, a template image generated from an image (graphic design) of each object may be prepared in advance, and the object on the GUI screen may be identified by pattern matching using the template image.

[0154] In the above embodiment, the GUI program to be processed has been described as being installed in the device, but the GUI program may be an application program for a general-purpose personal computer or may be a Web application.

[0155] In this embodiment, the creation process of the screen transition DB 12b by the screen transition creation program P1 and the verification process by the screen transition verification program P2 are not limited to being locally performed by the information processing device 10. For example, a server that executes the creation process of the screen transition DB 12b may be provided.

[0156] In this case, the information processing device 10 is configured to transmit a target program for which the screen transition DB 12b is to be created to the server, and the server starts the screen transition creation program P1 to create the screen transition DB 12b of the target program and transmit it to the information processing device 10. In this case, the information processing device 10 acquires the correct screen transition DB 12b generated by the server and the updated screen transition DB 12b, and can execute the verification process by the screen transition verification program P2 using the two acquired screen transition DBs 12b.

[0157] Also, a server that executes the verification process may be provided. In this case, the information processing device 10 is configured to transmit the correct screen transition DB 12b and the updated screen transition DB 12b to the server, and the server performs the verification process by comparing the two screen transition DBs 12b, creates a verification result screen, and transmits it to the information processing device 10. In this case, the information processing device 10 can obtain the verification result screen generated by the server and display it on the display unit 15, for example. Even in such a configuration, the same process as in this embodiment is possible, and the same effect can be obtained.

[0158] (Embodiment 2) In the above-mentioned embodiment 1, the processing command associated with the object detected in the GUI screen is registered in the object DB 12a, but depending on the execution environment of the target program, the processing command associated with the object may not be made public. In this case, the processing command associated with the detected object cannot be executed, and the GUI screen (post-transition GUI screen) to be displayed when the object is operated cannot be generated.

[0159] In this embodiment, an information processing device is described that actually executes a target program, displays a GUI screen to be displayed on the display unit 15, detects an object in the displayed GUI screen, and actually operates the detected object with a robot arm, thereby realizing a screen transition to the GUI screen to be displayed when the object is operated, and creating screen transition data.

[0160] 11 is a block diagram showing an example of the configuration of an information processing device according to embodiment 2. The information processing device 10 of this embodiment includes a camera connection unit 17 and a robot arm driving unit 18 in addition to the configuration of the information processing device 10 of embodiment 1 shown in FIG.

[0161] Description of the same configuration as in embodiment 1 will be omitted. In the information processing device 10 of this embodiment, the display unit 15 includes a capture display unit 15a and an operation display unit 15b, and the operation display unit 15b is configured with a touch panel.

[0162] Camera 17a is connected to camera connection unit 17 via, for example, a cable. Camera 17a acquires image data such as a still image or a video, and outputs the acquired image data to information processing device 10. Camera connection unit 17 acquires the image data output from camera 17a.

[0163] The control unit 11 of the information processing device 10 acquires the image data sent from the camera 17a via the camera connection unit 17, and stores the image data in the storage unit 12. The camera connection unit 17 may be configured to be capable of wireless communication with the camera 17a.

[0164] A robot arm 18b is connected to the robot arm driving unit 18, and a finger unit 18a capable of operating a touch panel is attached to the tip of the robot arm 18b.

[0165] The robot arm driving unit 18 performs driving processing of the robot arm 18b in accordance with instructions from the control unit 11, moves the finger unit 18a to an arbitrary position on the operation display unit 15b, and performs an operation on the arbitrary position with the finger unit 18a. The operation by the finger unit 18a includes, for example, tapping, double tapping, swiping, and the like.

[0166] When creating the screen transition DB 12b, the information processing device 10 of the present embodiment having the above-mentioned configuration sequentially displays the GUI screens to be displayed by executing the target program on the capture display unit 15a and the operation display unit 15b. The information processing device 10 detects objects in the GUI screens by capturing the GUI screens displayed on the capture display unit 15a with the camera 17a.

[0167] Furthermore, the information processing device 10 drives the robot arm 18b by the robot arm driving unit 18 in response to the GUI screen displayed on the operation display unit 15b, thereby allowing the finger unit 18a to perform an operation on the display screen of the operation display unit 15b.

[0168] Fig. 12 is an explanatory diagram showing an example of a record layout of the screen transition DB 12b of the second embodiment. The screen transition DB 12b of the second embodiment includes an arrangement position column in addition to the configuration of the screen transition DB 12b of the first embodiment shown in Fig. 2B. The arrangement position column stores information indicating the arrangement position of an object on a GUI screen before the transition. The position of an object is expressed by the coordinate values ​​of the top left and bottom right pixels of the display area (e.g., a rectangular area) of the object in a coordinate system in which the top left corner of the display area of ​​the GUI screen displayed on the display unit 15 (capture display unit 15a, operation display unit 15b) is the origin, the X axis runs to the right, and the Y axis runs downward.

[0169] Figures 13 and 14 are flowcharts showing an example of a creation process procedure of the screen transition DB 12b of the second embodiment. The process shown in Figures 13 and 14 is obtained by adding S61 between S21 and S22, S62 between S26 and S27, S63 instead of S30, S64 between S31 and S32, S65 between YES of S34 and S35, and S66 between YES of S36 and S37 in the process shown in Figures 5 and 6. Explanations of the same parts as in Figures 5 and 6 will be omitted.

[0170] In the information processing device 10 of this embodiment, when the control unit 11 executes a target program in response to the start of the screen transition creation program P1 (S21), the control unit 11 generates a GUI screen to be displayed by the execution of the target program, and displays it on the display units 15a and 15b (S61). The control unit 11 photographs and captures the GUI screen displayed on the capture display unit 15a with the camera 17a (S22). Note that the control unit 11 may capture the GUI screen using a screen capture function.

[0171] If there is no screen identical to the captured GUI screen (S24: NO), the control unit 11 executes the processes of S25 to S26, and specifies the arrangement position of each object recognized on the GUI screen (S62).

[0172] Here, the control unit 11 calculates the coordinate values ​​of the upper left and lower right pixels of the area of ​​each object in the display area of ​​the GUI screen based on the captured image data of the GUI screen. Then, the control unit 11 registers each object in the screen transition DB 12b (S27). In this embodiment, the control unit 11 stores the object name and arrangement position of each object in the screen transition DB 12b in association with the name of the GUI screen before the transition.

[0173] If the control unit 11 determines that there is no screen identical to the captured GUI screen (S24: NO), the control unit 11 may execute the following process. The control unit 11 calculates the matching rate (similarity) between the captured GUI screen and a GUI screen already registered as a pre-transition screen in the screen transition DB 12b. If the calculated matching rate is equal to or greater than a predetermined value, the control unit 11 may determine that the current GUI screen is a GUI screen with the same function as the registered GUI screen, and when registering the current GUI screen in the screen transition DB 12b in S25, the control unit 11 may add information indicating that the current GUI screen has the same function as the already registered GUI screen to the screen transition DB 12b and register the current GUI screen in the screen transition DB 12b.

[0174] For example, if the target program is a program related to a navigation system, the "navigation function" and the "audio function" may be designed with different background colors or icon color themes. In addition to the background color, the icon shapes and colors may be different. If the target program is a "multifunction device control program," the "printer function" and the "copy function" may be designed with different background colors or icon color themes.

[0175] In such a target program, the functions contained in the target program may be roughly divided, the GUI screens may be grouped by function, and then verification may be performed for each group later. For example, among objects A, B, and C on the HOME screen shown in Figure 7, the screen transitioned to in response to operations on objects A and B may be grouped as a screen related to "function α (actually a navigation function)," and the screen transitioned to in response to operations on object C may be grouped as a screen related to "function β (actually an audio function)." This makes it possible to perform verification by function, such as giving priority to verifying "function β" in later verification.

[0176] After processing S29, the control unit 11 of this embodiment drives the robot arm 18b based on the placement position of the identified object (S63), and performs an operation on the identified object with the finger unit 18a on the GUI screen being displayed on the operation display unit 15b (S31). Here, the control unit 11 obtains the position of the identified object from the screen transition DB 12b, drives the robot arm 18b using the robot arm driving unit 18 based on the position of the object, moves the finger unit 18a to a position facing the object in the GUI screen displayed on the operation display unit 15b, and further drives the robot arm 18b using the robot arm driving unit 18 to perform a tap operation with the finger unit 18a on the object in the GUI screen.

[0177] If the GUI screen displayed on the capture display unit 15a and the GUI screen displayed on the operation display unit 15b are different sizes, a process may be performed to convert the position of an object calculated from the captured GUI screen to the position of an object in the GUI screen displayed on the operation display unit 15b based on the ratio of screen sizes. This makes it possible to operate the same object in the GUI screen displayed on the operation display unit 15b as the object in the captured GUI screen.

[0178] The control unit 11 generates a GUI screen to be displayed next by operating an object in the GUI screen, and displays the GUI screen on the display units 15a and 15b (S64).Then, the control unit 11 captures the GUI screen displayed on the capture display unit 15a (S32), and stores the captured GUI screen in the screen transition DB 12b as a post-transition screen in association with the GUI screen before the transition and the object that was the object of the operation (S33).

[0179] Furthermore, when the control unit 11 determines that there is an object with zero operation counts in the lower-layer GUI screen to be transitioned from the current GUI screen (S34: YES), it drives the robot arm 18b based on the position of the object for transitioning to the post-transition GUI screen (S65), and performs a tap operation on the object with the finger unit 18a on the GUI screen being displayed on the operation display unit 15b, thereby transitioning to the post-transition GUI screen (S35).

[0180] Here too, the control unit 11 acquires the position of the object for transitioning to the GUI screen after transition from the screen transition DB 12b, drives the robot arm 18b based on the position of the object, moves the finger unit 18a to a position facing the object in the GUI screen displayed on the operation display unit 15b, and further performs a tap operation on the object with the finger unit 18a. This allows an arbitrary object in the current GUI screen to be operated, and transition to an arbitrary GUI screen can be made.

[0181] Furthermore, when the control unit 11 determines that there is a GUI screen before the transition of the current GUI screen (S36: YES), it drives the robot arm 18b based on the position of the Return object (S66), and performs a tap operation on the Return object with the finger unit 18a on the GUI screen being displayed on the operation display unit 15b, thereby transitioning to the GUI screen before the transition (S37).

[0182] Here too, the control unit 11 acquires the arrangement position of the Return object from the screen transition DB 12b, drives the robot arm 18b based on the arrangement position of the Return object, moves the hand and fingers 18a to a position facing the Return object in the GUI screen displayed on the operation display unit 15b, and further performs a tap operation on the Return object with the hand and fingers 18a. This makes it possible to operate the Return object in the current GUI screen and return to the GUI screen before the transition of the current GUI screen.

[0183] By the above-described processing, even if the processing command associated with an object in the GUI screen is unknown, it is possible to actually display the GUI screen and physically manipulate the object in the GUI screen, thereby executing the processing of the target program when the object is manipulated.

[0184] Therefore, even for a GUI program for a device that uses an OS in which processing commands associated with each object are not made public, it is possible to automatically create the screen transition DB 12b by executing the GUI program.

[0185] In the present embodiment, the configuration in which the finger unit 18a performs a tap operation on an object on a GUI screen has been described as an example, but the present invention is not limited to this configuration, and the type of operation may differ for each object. For example, the type of operation to be performed on each object is registered in association with each object, and the control unit 11 specifies the type of operation when performing an operation on each object, and causes the robot arm 18b to perform an action corresponding to the specified type of operation, thereby performing each type of operation on each object.

[0186] The information processing device 10 of this embodiment can create a correct screen transition DB 12b based on, for example, the first released target program through the above-mentioned process, and can create an updated screen transition DB 12b based on the latest target program. The information processing device 10 of this embodiment can also execute the process shown in Fig. 9, and can perform operation verification for the latest target program based on the correct screen transition DB 12b and the updated screen transition DB 12b.

[0187] In the present embodiment, the configuration in which the capture display unit 15a and the operation display unit 15b are provided separately has been described as an example, but a single display unit 15 may be used. That is, a configuration in which the camera 17a captures a GUI screen displayed on the display unit 15 and the finger unit 18a performs an operation on the GUI screen displayed on the display unit 15 may also be used.

[0188] (Embodiment 3) In the above-mentioned first and second embodiments, the verification result (OK / NG) is determined based on the result of comparison with the correct screen transition data (GUI screen after transition).

[0189] In this embodiment, an information processing device that judges the verification result based on the output state of the sound signal in addition to the state of the GUI screen after the transition will be described. Therefore, in this embodiment, even if the GUI screen after the transition matches the correct screen (the similarity is equal to or greater than a threshold), if the output state of the sound signal before and after the screen transition is not the correct state, the verification result is judged as NG.

[0190] The information processing device 10 of this embodiment has the same configuration as the information processing device 10 of the first embodiment shown in FIG. 1, so a description of the configuration will be omitted.

[0191] Fig. 15 is an explanatory diagram showing an example of a record layout of the screen transition DB 12b of the third embodiment. In addition to the configuration of the screen transition DB 12b of the first embodiment shown in Fig. 2B, the screen transition DB 12b of this embodiment includes a sound information sequence that stores information indicating the output state of a sound signal in a GUI screen before the transition, and a sound information sequence that stores information indicating the output state of a sound signal in a GUI screen after the transition. The sound information includes information on the presence or absence of an output sound, the volume, the time from the screen transition to the output of a sound signal, etc.

[0192] Figures 16 and 17 are flowcharts showing an example of a creation process procedure of the screen transition DB 12b of the third embodiment. The process shown in Figures 16 and 17 is obtained by adding S71 between S22 and S23, adding S72 instead of S25, adding S73 between S32 and S33, and adding S74 after S35 in the process shown in Figures 5 and 6. Explanations of the same processes as those in Figures 5 and 6 will be omitted.

[0193] In the information processing device 10 of this embodiment, when the control unit 11 executes a target program in response to the start of the screen transition creation program P1 (S21), it captures the GUI screen to be displayed by the execution of the target program (S22) and also captures the sound to be output to obtain sound information (S71).

[0194] For example, the control unit 11 acquires the presence or absence of an output sound and its volume. The control unit 11 may acquire the type of sound, such as whether the output sound is voice or music, and if the sound is voice, may acquire the output content by a voice recognition function. The control unit 11 may also record the output sound and acquire the recording data.

[0195] When the control unit 11 determines that there is no screen identical to the captured GUI screen (current GUI screen) (S24: NO), it registers the current GUI screen as a pre-transition screen together with the sound information in the screen transition DB 12b (S72). This makes it possible to register sound information indicating the state of the output sound while the GUI screen is being displayed, in addition to the GUI screen that has become the display target by executing the target program.

[0196] The control unit 11 also captures the GUI screen that becomes the display target by operating the object specified in S29 (S32), and captures the output sound to obtain sound information (S73).The control unit 11 then stores the current GUI screen as a post-transition screen together with the sound information in the screen transition DB 12b in association with the GUI screen before the transition, the object that became the operation target, and the number of times the object was operated (S33).

[0197] Furthermore, if the control unit 11 determines that there is an object with zero operation counts in the lower-level GUI screen to which the current GUI screen is to be transitioned (S34: YES), it transitions to the post-transition GUI screen (S35) and captures the output sound on the post-transition GUI screen to obtain sound information (S74).

[0198] In this case, the control unit 11 also stores the current GUI screen and sound information as a post-transition screen in the screen transition DB 12b in association with the GUI screen before the transition, the object that was the operation target, and the number of times the object was operated (S33). By capturing the output sound at the same time as capturing the current GUI screen through the above-described process, not only the screen transition caused by object operations but also the transition state of the output sound can be captured.

[0199] The information processing device 10 of this embodiment can create a correct screen transition DB 12b based on, for example, the first released target program through the above-mentioned process, and can create an updated screen transition DB 12b based on the latest target program. The information processing device 10 of this embodiment can also execute the process shown in Fig. 9, and can perform operation verification for the latest target program based on the correct screen transition DB 12b and the updated screen transition DB 12b.

[0200] In this embodiment, in the processing of FIG. 9, the processing of FIG. 16 and FIG. 17 is performed in S42, and the screen transition data read out in S43 includes sound information while the GUI screen before the transition is displayed and sound information while the GUI screen after the transition is displayed.

[0201] 9, in S44, the control unit 11 of this embodiment reads out the GUI screen after the correct transition and the sound information (the correct sound information) being displayed on this GUI screen. Then, in S45, the control unit 11 compares the GUI screen after the transition and the sound information included in the screen transition data read out in S43 with the GUI screen after the correct transition and the sound information, and judges the verification result. For example, the control unit 11 judges whether the presence or absence of output sound matches in the sound information to be compared, and if there is output sound, judges whether the volume matches (the difference in volume is less than a threshold value).

[0202] If both pieces of sound information being compared have no output sound, the control unit 11 determines that the sound information after the transition is correct (verification result: OK), and if one has an output sound and the other has no output sound, it determines that the sound information after the transition is not correct (verification result: NG). In addition, when the sound information being compared both have output sound, the control unit 11 compares the volume of the sound information being compared, and if the volumes match, it determines that the sound information after the transition is correct sound information (verification result: OK), and if the volumes do not match, it determines that the sound information after the transition is not correct sound information (verification result: NG).

[0203] In addition, when the sound information to be compared includes output contents, the control unit 11 may determine the verification result based on whether the output contents match. For example, when the output contents in the sound information to be compared include the same keyword, the control unit 11 may determine that the sound information after the transition is correct sound information (verification result: OK), and when the keywords included in the output contents are different, the control unit 11 may determine that the sound information after the transition is not correct sound information (verification result: NG).

[0204] By the above-mentioned process, in this embodiment, it is possible to verify the occurrence of a "No sound" state in which sound that should be output is not output, as a state in which the verification result is NG. As a result, even if the GUI screen after the transition matches the correct screen (the similarity is equal to or greater than a threshold), if the output sound information does not match the correct sound information, the verification result can be determined as NG. In this way, by verifying the appropriateness of the transition state resulting from the execution of the target program based on not only the transition state of the GUI screen but also the output state of the sound, it is possible to realize a more accurate operation verification process.

[0205] In the above-mentioned first to third embodiments, the screen transition data is created by operating the objects that can be operated on the current GUI screen. Furthermore, a priority order is set for each object, and the screen transition data can be created efficiently by operating the objects in the order according to the priority order.

[0206] In addition, for example, an operation condition may be set for each object, and when the operation condition is satisfied, each object may be operated to create screen transition data. For example, by setting simultaneous operation of multiple objects, prohibition of operation for a predetermined time from display of a GUI screen, etc. as an operation condition for each object, it is possible to appropriately control the range of GUI screens (verification range) for which screen transition data is to be created. Therefore, even in the operation verification of a GUI program with a complex screen configuration, it is possible to efficiently create screen transition data in order of GUI screens with high priority, thereby making it possible to speed up and streamline operation verification.

[0207] Fig. 18 is an explanatory diagram showing a modified example of the object DB 12a. Different processing commands may be assigned to objects displayed on the GUI screen depending on the type of GUI screen (type of function). Fig. 18B shows an example of a GUI screen when listening to AM radio and an example of a GUI screen when music is being played from a recording medium such as a CD or an SD card. The GUI screen when listening to AM radio and the GUI screen when music is being played have the same configuration, and objects B1 to B4 are provided as operable buttons.

[0208] However, for example, object B3 is assigned a processing command of TUNE UP (processing of increasing the number of channels and searching for a listenable channel) on the AM radio GUI screen, and a processing command of TRACK UP (processing of playing the next music) on the music playback GUI screen. Therefore, when an object in a GUI screen is detected, the control unit 11 can appropriately identify the processing command associated with the object in each GUI screen by determining the type of GUI screen and specifying the processing command according to the determined type. The type of GUI screen may be determined, for example, by extracting keywords in the GUI screen using OCR or the like.

[0209] When the object DB 12a in Fig. 18A is used, for example, in S30 in Fig. 6, the control unit 11 can extract a keyword included in the current GUI screen, and specify a processing command associated with each object in the GUI screen including the keyword from the object DB 12a based on the extracted keyword and object name. In such a configuration, the function assigned to each object for the same object can be switched for each GUI screen. Therefore, since multiple processing commands can be assigned to one object, the types of objects used can be reduced, and a GUI screen with high operability can be provided.

[0210] (Embodiment 4) In the above-described first to third embodiments, when it is determined that either the target program or the execution environment has been updated, the control unit 11 creates a screen transition DB 12b for the updated target program, and compares it with a correct screen transition DB 12b created in advance to determine whether there is a problem with the screen transition data in the updated screen transition DB 12b.

[0211] In this embodiment, a process is described in which pre-update screen transition data, which is screen transition data before either the target program or the execution environment is updated, is compared with post-update screen transition data, which is screen transition data after either the target program or the execution environment is updated, and if there are differences in the transition of GUI screens, the differences are displayed in a manner different from the normal transition in a transition diagram based on the pre-update transition data or a screen transition diagram based on the post-update image transition data.

[0212] FIG. 19 is an example of a transition diagram based on pre-update screen transition data. The transition diagram is displayed on the display unit 15 of the information processing device 10. In the figure, the rectangle in the upper left is the first GUI screen when the test begins. The numbers displayed in each rectangle indicate the transition order. Although not shown, it is desirable to display thumbnail images of the GUI screens in the rectangles. The arrows connecting each rectangle indicate the transition between GUI screens. For example, it indicates a transition from the GUI screen indicated by rectangle "1" to the GUI screen indicated by rectangle "2".

[0213] In the transition diagram shown in Fig. 19, the GUI screen indicated by rectangle "1" contains objects that transition to the GUI screens indicated by rectangle "2," rectangle "6," and rectangle "8." By selecting each object contained in the GUI screen indicated by rectangle "1," arrows are displayed indicating the transition from rectangle "1" to rectangle "2," rectangle "6," and rectangle "8," respectively.

[0214] In the transition diagram shown in Fig. 19, since the GUI screen indicated by rectangle "3" does not include any other objects to transition to, the control unit 11 returns to the GUI screen indicated by rectangle "2", selects another object, and transitions to the GUI screen indicated by rectangle "4". Then, with the transition to the GUI screen indicated by rectangle "5", all transitions from the GUI screen indicated by rectangle "2" are completed, and the control unit 11 returns to the GUI screen indicated by rectangle "1", and selects an object to transition to the GUI screen indicated by rectangle "6". The control unit 11 generates pre-update screen transition data by repeating such processing.

[0215] As described in the above embodiment, the determination of whether a transition has occurred to a new GUI screen or to an already captured GUI screen is made based on whether the degree of similarity between the transitioned GUI screen and the captured GUI screen is less than a predetermined value.

[0216] FIG. 20 is an example of a transition diagram based on updated screen transition data. FIG. 21 is an example of a transition diagram based on updated image transition data, displaying differences from pre-update image transition data. Both are displayed on display unit 15. The rectangles and arrows shown in the transition diagrams of FIGS. 21 and 22 are as explained in the transition diagram of FIG. 19, and only the display of differences will be explained. Arrows and rectangles shown with dashed lines indicate transitions and GUI screens that are included in the transition diagram before the update, but not included in the transition diagram after the update. Arrows shown with thick lines indicate transitions that are not included in the transition diagram before the update, but are included in the transition diagram after the update.

[0217] As shown in Figure 21, by displaying the differences before and after the update in a transition diagram, the person in charge of verification can confirm the differences in screen transitions and conduct testing using a more efficient test scenario that focuses on the differences.

[0218] Note that Figure 21 shows an example of displaying the differences between the pre-update image transition data and the transition diagram based on the post-update image transition data, but it is also possible to display the differences between the pre-update image transition data and the post-update image transition data in a transition diagram based on the pre-update image transition data.

[0219] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. The claims may also be written in a format in which a multiple claim cites at least one other multiple claim (multi-multi claim).

[0220] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0221] 10. Information processing device 11 Control section 12 Storage section 13. Communications Department 14 Input section 15 Display section 17 Camera connection part 18 Robot arm drive unit M Learning Model 12a Object DB 12b Screen transition DB

Claims

1. an execution step of executing a target program accompanied by displaying a GUI (Graphical User Interface) screen; a detection step of detecting a type of an operable object included in the GUI screen displayed by the execution of the target program; an acquisition step of acquiring data regarding a transition of a screen to be displayed when the detected object is operated; a creating step of creating screen transition data of the target program by associating the GUI screen before the transition, a type of an object in the GUI screen, and data related to a screen transition when the object is operated; a comparison step of comparing pre-update image transition data, which is the screen transition data created before the target program or the execution environment of the target program is updated, with post-update image transition data, which is the screen transition data created after the target program or the execution environment of the target program is updated; a display step of displaying, on a display unit, a screen transition diagram based on the pre-update image transition data or a screen transition diagram based on the post-update image transition data, the difference in a manner different from a normal transition, when a difference is found in the transition of the GUI screen as a result of the comparison; A program that causes a computer to execute a process including the steps of:

2. The comparing step calculates a degree of similarity between the GUI screen included in the pre-update image transition data and the GUI screen included in the corresponding post-update image transition data, and compares whether the calculated degree of similarity is less than a predetermined value; In the display step, when the calculated similarity is less than a predetermined value, the GUI screen is determined to have the difference, and is displayed on the display unit in a manner different from a GUI screen that does not have the difference in the screen transition diagram. The program according to claim 1.

3. The comparing step calculates a degree of similarity between a sound signal output on the GUI screen included in the pre-update image transition data and a sound signal output on the GUI screen included in the corresponding post-update image transition data, and compares whether the calculated degree of similarity is less than a predetermined value; In the display step, when the calculated similarity is less than a predetermined value, the GUI screen is determined to have the difference, and is displayed on the display unit in a manner different from a GUI screen that does not have the difference in the screen transition diagram. The program according to claim 1.

4. A control unit that executes a target program accompanied by displaying a GUI screen, The control unit is A process of detecting a type of an operable object included in the GUI screen displayed by the execution of the target program; A process of acquiring data regarding a screen transition that is displayed when the detected object is operated; a process of creating screen transition data for the target program by associating the GUI screen before the transition, a type of an object in the GUI screen, and data regarding a screen transition when the object is operated; A process of comparing pre-update image transition data, which is the screen transition data created before the target program or the execution environment of the target program is updated, with post-update image transition data, which is the screen transition data created after the target program or the execution environment of the target program is updated; If there is a difference in the transition of the GUI screen as a result of the comparison, a process is executed in which the difference is displayed on a display unit in a screen transition diagram based on the pre-update image transition data or a screen transition diagram based on the post-update image transition data in a manner different from a normal transition. Information processing device.

Images