Identification device, identification method, and identification program

The identification device and method address the challenge of accurately identifying screens and components in virtualized environments by combining object access and image recognition, creating a sample screen model to ensure precise identification without altering the server environment.

JP7582470B2Active Publication Date: 2024-11-13NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023526838
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-11
Publication Date
2024-11-13
Estimated Expiration
2041-06-11

AI Technical Summary

Technical Problem

Conventional methods struggle to accurately identify application screens and screen components in virtualized environments due to the inability to access screen component objects and susceptibility to differences in look and feel, display magnification, and variations in screen implementation content, leading to incorrect identification of screen components and display strings.

Method used

An identification device and method that utilizes both screen images and screen component objects to identify screen data, employing a combined object access and image recognition approach to create a sample screen model, allowing accurate identification in virtualized environments without requiring changes to the server environment.

Benefits of technology

Enables precise identification of application screens and screen components in virtualized environments, overcoming issues of object access and image recognition limitations, ensuring correct identification even with variations in screen implementation and display conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582470000009
    Figure 0007582470000009
  • Figure 0007582470000010
    Figure 0007582470000010
  • Figure 0007582470000011
    Figure 0007582470000011
Patent Text Reader

Abstract

This identification device (10) comprises: a first identification unit (151a) for identifying first screen data that includes an image of a screen in an application and information relating to screen component objects constituting the screen, and outputting a first identification result associated with sample screen data that is to be referenced; a derivation unit (152a) for deriving a sample screen model used for identification in a virtualized environment, on the basis of the sample screen data and the first identification result; and a second identification unit (153a) for identifying second screen data that includes the image of the screen and does not include information relating to the screen component objects, and outputting a second identification result associated with the sample screen model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an identification device, an identification method, and an identification program. [Background technology]

[0002] (1. Screen data of the application to be operated) Conventionally, in terminal work, a worker (hereinafter, also referred to as "user") refers to values ​​displayed in text boxes, list boxes, buttons, labels, etc. (hereinafter, referred to as "screen components") that constitute the screen of the target application running on the terminal, and performs operations such as inputting values ​​and selecting values ​​on the screen components. For this reason, some programs that aim to automate and support terminal work (hereinafter, referred to as "automated operation agents") and some programs that aim to grasp and analyze the actual work status (hereinafter, referred to as "work analysis tools") acquire and use the following "screen data" or parts of it:

[0003] Here, “screen data” refers to, as shown in FIG. 24, “screen image” (see FIG. 24(1)), “screen attributes (title, class name, coordinate values ​​of the drawing area, name of the displayed application to be operated, etc.)” (see FIG. 24(2)), and “information on screen component objects” (see FIG. 24(3)).

[0004] An automatic operation agent is a program that records and saves as a scenario the operations performed by the worker on the terminal and additional information that the user wants to refer to along with the screen display of the target application, and opens and plays back the scenario, allowing the same operations to be executed repeatedly later, or displays additional information according to the display screen.In the following, automatic operation agents and work analysis tools will be collectively referred to as "automatic operation agents, etc."

[0005] Screen images and screen attributes can be obtained through an interface provided by the OS (Operating System). Screen component objects (hereinafter simply referred to as "objects") are data prepared in the computer's memory that represent the display contents and status of each screen component as the variable values ​​of the target application so that the OS or the target application can easily control the behavior of the screen components when they are operated and draw the screen images. The information can be obtained through User Interface Automation (hereinafter referred to as "UIA"), Microsoft Active Accessibility (hereinafter referred to as "MSAA"), or an interface provided by the target application. The object information includes information that can be used by each object alone (hereinafter referred to as "attributes"), such as the type of screen component, the display / hide state, the display string, and the coordinate values ​​of the drawing area, as well as information that is internally held by the target application and that represents the relationships between objects, such as the containment relationship and ownership relationship (hereinafter referred to as "screen structure") (see Figure 24 (3-1) (3-2)).

[0006] Even if the functions provided to the worker are the same (hereinafter referred to as "equivalent") on screens displayed at different times on different terminals, the values ​​of some object attributes may differ, or the presence or absence of objects themselves may differ, depending on the case being displayed and the status of the work being performed. For example, if the number of details included in a case is different, the number of rows in the list displaying them may change, and depending on the status of the work being performed, error messages may be displayed / hidden, and the screen structure may also change. Furthermore, even if the screens are equivalent, the information of objects, including the screen structure, may change (hereinafter referred to as "differences in screen implementation content") due to differences in the platform (e.g., Microsoft Foundation Class, .NET Framework, Java (registered trademark) swing, etc.) of the GUI (Graphical User Interface) used by the application to be operated to display the screen, as well as differences in its version (hereinafter referred to as "differences in screen implementation method").

[0007] The screen image reflects the presence or absence of an object and its information. Depending on the displayed project or the status of the work being performed, the values ​​of some of the object's attributes may differ, or the presence or absence of an object itself may differ, and when the screen structure changes, the screen image also changes. Furthermore, compared to object information, the screen image is more susceptible to the effects of differences in the settings of the customization functions of the OS or the application being operated, differences in the number of colors depending on the display environment, and differences in options depending on the communication conditions when remotely logging in to a remote desktop (hereinafter referred to as "differences in look and feel"). These variations in the screen image include changes in the position, size, and color of the area that each screen component occupies in the screen image, and the font type and size of the text displayed on the screen components.

[0008] (2. Identifying Screens and Screen Components) When the automated operation agent is configured, it acquires sample screen data from a specific terminal and uses it to specify the screen components that are the targets of display value acquisition and manipulation (hereafter referred to as "targets of control"). When executing automation or support processing, it acquires screen data (hereafter referred to as "target screen data") from the screen displayed at that time on any terminal, including terminals other than the specific terminal for which the operation was configured, and compares it with the sample screen data, or with the equivalence judgment criteria for screens and screen components obtained by processing it or creating it by hand using it as reference. This identifies screen components from the screen data at that time that are equivalent to the target screen components in the sample screen data, and makes them the targets of display value acquisition and manipulation.

[0009] Furthermore, the work analysis tool acquires screen data and information on operations at the timing when a worker operates a screen component on each terminal, and collects it as an operation log. To enable humans to grasp and analyze patterns and trends from the large amount of collected operation logs, screen data acquired at different times and different terminals is classified so that screens with the same screen components or screen components to be operated are grouped together, and the data is used to derive screen operation flows, tally up the number of operations performed and operation times, etc. One possible method for this classification is to sample some screen data from the large amount of operation logs to use them as sample screen data, and then compare the screen data from the remaining operation logs with the sample screen data to determine where to classify them.

[0010] Alternatively, there may be cases where it is necessary to mask the portions of the screen data in the operation log acquired by the work analysis tool where confidential information is displayed. One method for doing this is to first use a sample of the screen data in which confidential information is displayed as sample screen data, and specify the screen components to be masked in the sample screen data. One possible method is to compare the remaining screen data in the operation log with the sample screen data, identify screen components equivalent to the screen components to be masked in the sample screen data, and perform masking.

[0011] In the following, "identification" refers to determining whether sample screen data and target screen data acquired at different times and on different terminals have equivalent screens and screen components that provide the worker with the same functions, and identifying those equivalent to the screen components of the sample screen data from among the screen components of the target screen data (see Fig. 25). Note that the dashed arrows in Fig. 25 indicate the correspondence between the sample and target screen components in the identification. Also, the above arrows do not represent all the correspondences, but only a portion of them.

[0012] Additionally, the process of preparing sample screen data and various other data other than the target screen data used in identification prior to identification is described as "operation setting for identification." In the case of using an automated operation agent, this corresponds to the process of acquiring sample screen data on a specific terminal and using it to specify the screen components to be controlled. In the case of using a work analysis tool, this corresponds to the process of sampling some screen data from the large amount of acquired operation logs and preparing sample screen data.

[0013] Note that the identification of screens based on screen attributes and the identification of screen components based on screen component information are complementary. For example, even if the screen titles are the same, the screen components contained therein may be completely different, so it is not possible to determine whether the screens are equivalent by comparing screen attributes alone. By identifying the screen components, it is also possible to check whether the screen data to be processed contains screen components equivalent to the screen components to be controlled in the sample screen data, and by taking this result into consideration, it is possible to determine whether the screen data are equivalent. Conversely, if the screen titles are different, it is possible to determine that the screens are not equivalent without having to identify multiple screen components, which helps reduce the amount of calculations.

[0014] (3. Screen Component Identification Technology) (3-1. Object Access Method) The object access method does not use screen images for either the sample screen data or the screen data to be processed, but uses information on screen component objects. Note that a terminal that cannot access the objects of the screen components of the desktop or the main application to be operated, such as a remote desktop, is referred to as a "virtualized environment," and a terminal that can access them is referred to as a "non-virtualized environment."

[0015] (3-2. Image recognition method) The image recognition method uses the image of the screen, rather than information on screen component objects, for both the sample screen data and the processing target screen data.

[0016] (3-3. Object access and image recognition combined method) The object access / image recognition combined method is a method in which both the object access method and the screen recognition method are executed simultaneously or by switching between them (see, for example, Patent Document 1).

[0017] (3-4.Equivalence judgment condition method) The equivalence judgment condition method is a method that uses conditions for judging the equivalence of screen components instead of sample screen data. The methods are further divided depending on whether the screen component object information or the screen image is used as the screen data to be processed.

[0018] (4. Technology for obtaining display strings) (4-1. Object access method) The object access method does not use the screen image as the screen data to be processed, but uses information about screen component objects. It makes it easy to obtain the display string by using UIA, MSAA, or an interface provided by the target application.

[0019] (4-2. Optical character recognition method) The optical character recognition method does not use information about screen component objects as screen data to be processed, but uses an image of the screen, and OCR (Optical Character Recognition) technology can be used. [Prior art documents] [Patent documents]

[0020] [Patent Document 1] JP 2015-005245 A Summary of the Invention [Problem to be solved by the invention]

[0021] However, in the above-mentioned conventional technology, it is not possible to accurately identify the screen and screen components of an application and obtain display character strings in a virtualized environment without taking the time and effort of setting up the operation of the identification and the reading setting of the display character string. This is because the above-mentioned conventional technology has the following problems.

[0022] (1.Issues regarding identification of screen components) (1-1. Issues regarding object access methods) In the object access method, only the screen image is transferred in the virtualized environment, and the objects cannot be accessed from an automated operation agent or the like, so that the screen components cannot be identified.

[0023] (1-2. Issues regarding image recognition methods) In the image recognition method, the image on the screen is more susceptible to the effects of differences in look and feel than in the object access method, and the occurrence of such variations makes it difficult to correctly identify the screen components. Also, in the image recognition method, the image on the screen is significantly affected by the display magnification of the screen, and the occurrence of such variations makes it difficult to correctly identify the screen components.

[0024] (1-3. Issues regarding object access and image recognition combined methods) In the combined object access and image recognition method, the two methods can be used in a mutually complementary manner, and furthermore, adaptively and evolutionarily based on this premise, but this is limited to cases where it is possible to access not only the images on the screen, but also the objects; when using a virtualization environment that does not meet these conditions, it becomes impossible to correctly identify the screen components.

[0025] (1-4. Issues regarding the equivalence criteria method) In the equivalence judgment condition method, when using information on screen component objects, an "arrangement pattern" that expresses the conditions of the relative arrangement relationship between screen components on a sample screen (two-dimensional plane) is used as a condition for judging the equivalence of the screen components to be controlled, but this is based on the premise that information on screen component objects is used as the screen data to be processed, and screen components cannot be identified in a virtualized environment. Also, with the above method, currently, a person must create an arrangement pattern for each screen and screen component while anticipating fluctuations that may occur on the screen to be processed.

[0026] Furthermore, in the equivalence determination condition method, when a screen image is used, a "screen model" is prepared as a graph in which the nodes are fragments of the screen image or displayed character strings or their regular expressions corresponding to the screen components, and the links are their adjacent relationships on the sample screen (two-dimensional plane), and matching is performed with the image of the screen to be processed or character strings obtained from a partial area of ​​it using OCR technology; however, just as with the above layout patterns, currently the screen model must be created manually for each screen and screen component, while anticipating fluctuations that may occur on the screen to be processed.

[0027] (2. Issues regarding obtaining display strings) (2-1. Issues regarding object access methods) With the object access method, the display string can be easily obtained by using UIA, MSAA, or an interface provided by the target application itself; however, since the method uses information from screen component objects, the display string cannot be obtained in a virtualized environment.

[0028] (2-2. Issues regarding optical character recognition methods) In optical character recognition, the display character strings of multiple screen components drawn on the screen image are recognized without any prerequisites regarding the font type and size after the display magnification is reflected, so many display character strings containing errors are obtained. [Means for solving the problem]

[0029] In order to solve the above-mentioned problems and achieve the object, the identification device of the present invention is characterized by comprising a first identification unit that identifies first screen data including an image of an application screen and information regarding screen component objects which are objects of elements that constitute the screen, and outputs a first identification result that corresponds to sample screen data which is the screen data to be referenced, and a second identification unit that identifies second screen data that includes an image of the screen and does not include information regarding the screen component objects, and outputs a second identification result that corresponds to the sample screen data.

[0030] Furthermore, the identification method of the present invention is an identification method executed by an identification device, and is characterized in that it includes a step of identifying first screen data including an image of an application screen and information regarding screen component objects which are objects of elements that constitute the screen, and outputting a first identification result associated with sample screen data which is the screen data to be referenced, and a step of identifying second screen data including an image of the screen and not including information regarding the screen component objects, and outputting a second identification result associated with the sample screen data.

[0031] In addition, the identification program of the present invention is characterized in that it causes a computer to execute the steps of identifying first screen data including an image of an application screen and information regarding screen component objects which are objects of elements that make up the screen, and outputting a first identification result that corresponds to sample screen data which is the screen data to be referenced, and identifying second screen data that includes an image of the screen but does not include information regarding the screen component objects, and outputting a second identification result that corresponds to the sample screen data. Effect of the Invention

[0032] In the present invention, in a virtualized environment, it is possible to accurately identify the screen and screen components of an application and obtain display strings without the hassle of setting up the identification operation or the reading settings for the display string. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an identification system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the identification device etc. according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of data stored in a screen data storage unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of data stored in the processing result storage unit according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of data stored in the identification information storage unit according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of data stored in the first classification result storage unit according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of data stored in the identification example storage unit according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of data stored in the screen model storage unit according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of data stored in the screen model storage unit according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of data stored in the drawing area storage unit according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of data stored in the layout relationship storage unit according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of data stored in the second classification result storage unit according to the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a process for estimating the font type and size of a display character string according to the first embodiment. [Figure 14] FIG. 14 is a diagram showing an example in which a difference in the number of characters in a display character string affects the relative positional relationship of character string drawing areas. [Figure 15] FIG. 15 is a diagram showing an example of a matching process when estimating the font type and size of a display character string according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of processing for identifying a character string drawing area according to the first embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of the overall flow of processing according to the first embodiment. [Figure 18] FIG. 18 is a flowchart showing an example of the flow of a sample screen model derivation process according to the first embodiment. [Figure 19] FIG. 19 is a flowchart showing an example of the flow of the identification process of the second screen data according to the first embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of the flow of the second character string acquisition process according to the first embodiment. [Figure 21] FIG. 21 is a flowchart showing an example of the flow of a process for estimating the font type and size of a character string to be displayed when the character string drawing area is unknown according to the first embodiment. [Figure 22] FIG. 22 is a flowchart showing an example of the flow of a process for limiting component model candidates for matching based on whether or not a character string drawing area can be specified according to the first embodiment. [Figure 23] FIG. 23 is a diagram illustrating a computer that executes a program. [Figure 24] FIG. 24 is a diagram showing an example of the screen data. [Diagram 25] FIG. 25 is a diagram showing an example of the process of identifying screen components. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Hereinafter, an embodiment of an identification device, an identification method, and an identification program according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0035] [First embodiment] Below, we will explain the configuration of the identification system 100 related to the first embodiment (or, as appropriate, the present embodiment), a comparison between the conventional technology and this embodiment, the configuration of the identification device 10, etc., details of each process, and the flow of each process, and finally, the effects of this embodiment will be explained.

[0036] [Configuration of the identification system 100] The configuration of an identification system (or "this system" as appropriate) 100 according to this embodiment will be described in detail with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an identification system according to a first embodiment. Below, an example of the overall configuration of this system 100 will be shown, and then the processing of this system 100 will be described.

[0037] (1. Example of overall system configuration) This system 100 includes an identification device 10 and an automatic operation agent device 20 that executes functions such as an automatic operation agent. The identification device 10 and the automatic operation agent device 20 are arranged in the same terminal and are communicatively connected by an API (Application Programming Interface) between the devices or an inter-process communication means provided by the OS of the terminal, or are communicatively connected by wire or wirelessly via a predetermined communication network not shown. Note that the identification system 100 shown in FIG. 1 may include a plurality of identification devices 10 and a plurality of automatic operation agent devices 20.

[0038] 1, the automatic operation agent device 20 is configured independently of the identification device 10, but it may be configured to be integrated into the identification device 10. In other words, the identification device 10 may be a separate device that operates in cooperation with the automatic operation agent device 20, or may be realized as a part of the automatic operation agent device 20.

[0039] In addition, in this system 100, the data acquired by the automatic operation agent device 20 involves screen data with object information (hereinafter also referred to as "first screen data") 30 and screen data without object information (hereinafter also referred to as "second screen data") 40. Here, the first screen data 30 is screen data that includes, in addition to the image of the screen, information on screen component objects that are the attributes of objects and the screen structure (hereinafter also referred to as "information on screen component objects"). Also, the second screen data 40 is screen data that includes the image of the screen but does not include information on the screen component objects.

[0040] (2. Overall system processing) In the above-mentioned system, an example will be described in which a screen / screen component is identified in a non-virtualized environment, a display string is obtained, the screen data used at that time is recorded, and the recorded screen data is referred to in a virtualized environment to identify a screen / screen component and obtain a display string.

[0041] First, the automatic operation agent device 20 acquires first screen data 30 in a non-virtualized environment (step S1), and transmits the acquired first screen data 30 to the identification device 10 (step S2). Next, the identification device 10 identifies a screen and screen components from the first screen data 30 transmitted by the automatic operation agent device 20, acquires a display character string (step S3), and transmits the acquired display character string and information required to identify the screen components to be controlled to the automatic operation agent device 20 (step S4). Hereinafter, the processing executed in the above steps S1 to S4 will also be referred to as the "object information use mode."

[0042] The identification device 10 also creates a sample screen model to be used in a non-virtualized environment from the first screen data 30 (step S5). Hereinafter, the process executed in step S5 above will also be referred to as a "sample screen modeling mode."

[0043] On the other hand, the automatic operation agent device 20 acquires the second screen data 40 in the non-virtualized environment (step S6), and transmits the acquired second screen data 40 to the identification device 10 (step S7). Then, the identification device 10 uses the sample screen model created in the process of step S5 above to identify the screen and screen components from the second screen data 40 transmitted by the automatic operation agent device 20, acquires a display character string (step S8), and transmits the acquired display character string and information required for identifying the screen components to be controlled to the automatic operation agent device 20 (step S9). Hereinafter, the processes executed in steps S6 to S9 above will also be referred to as "object information non-use mode".

[0044] In the identification system 100 according to this embodiment, the actual screen of the application to be operated is used in a non-virtualized environment to set up and confirm the identification operation in the same manner as in the above-mentioned object access method. At that time, the sample screen data used in the identification operation setting and the identification example screen data determined to be equivalent to the sample screen data when confirming the identification operation setting are acquired and stored, including the information on the screen component objects. The information on these objects is used to identify the screen components and read the displayed character strings in the virtualized environment. Therefore, the present system 100 enables correct identification processing even in a virtualized environment without being affected by the execution environment or operational restrictions.

[0045] [Comparison between conventional classification processing and classification processing according to this embodiment] Here, a conventional identification process that is generally performed will be described as a reference technology. Below, the conventional screen component identification process and the conventional display character string acquisition process will be described, and then the screen component identification process and the display character string acquisition process according to this embodiment will be described.

[0046] (1. Conventional Screen Component Identification Processing) (1-1. Object access method) As described above, the object access method is a method in which neither the sample screen data nor the screen data to be processed uses screen image data, but information on screen component objects. For example, a method (conventional identification method A) is known in which the screen structures of a sample and a screen to be processed are compared on the premise that screen component objects will be accessed, to determine the equivalence of the screen and screen components. Conventional identification method A does not necessarily require a human to create conditions for determining the equivalence of screen components.

[0047] When a virtualized desktop or application, such as a remote desktop, is the target of operation, normally only the screen image is transferred to the terminal that the worker directly operates, and an automatic operation agent, etc. operating within the terminal that the worker directly operates cannot access objects of the screen components of the target application.In response to this, a method is known (conventional identification method B) that allows an automatic operation agent, etc. operating within the thin client to access objects of the screen components of the target application operating within the server by installing a plug-in on the server on which the target application is operating (hereinafter referred to as the "thin client server") and on the terminal to which only the screen image is transferred (hereinafter referred to as the "thin client client").

[0048] However, in the object access method, in a virtualized environment, usually only the screen image is transferred, and the object cannot be accessed from an automated operation agent, etc. Therefore, the screen components cannot be identified by only the object access method including the conventional identification method A (Problem A).

[0049] In addition, the conventional identification method B can be said to solve the problem A by enabling the automatic operation agent or the like operating in the client to transparently access the screen component objects of the target application running on the thin client server. However, this method requires changes to the server environment, such as the need to install a plug-in not only on the thin client but also on the server. This means that the behavior of the target application running on the server, including performance, may be affected. In addition, the impact is constant throughout the entire time period when the terminal work is performed. Therefore, the organization that provides and operates the target application is required to investigate whether there is an impact and take measures such as increasing server resources. This is a major obstacle when using the automatic operation agent or the like as a means of improving business efficiency that can be promoted by the organization that performs the terminal work (problem B).

[0050] In addition, since both conventional identification methods A and B are premised on object access, differences in the screen implementation contents of the target application make it impossible to correctly identify screen components (problem C).

[0051] (1-2. Image recognition method) As described above, the image recognition method uses the image of the screen, rather than using information on screen component objects, for both the sample screen data and the screen data to be processed. Note that the image of the screen used as the sample screen data may also be divided into regions rather than used as the entire screen image. For example, a method (conventional identification method C) is known as "image region identification processing" in the "adaptive operator emulation method" that determines the equivalence of screen components by matching a template image of the screen component, i.e., a fragment of a sample screen image, with the image of the screen to be processed.

[0052] Conventional identification method C can be said to solve problems A to C by using screen images for both sample screen data and target screen data. However, screen images are more susceptible to the effects of differences in look and feel than the object access method, and the resulting fluctuations make it difficult to correctly identify screen components (problem D).

[0053] In addition, the image on the screen is greatly affected by the display magnification of the screen, and the occurrence of fluctuations makes it impossible to correctly identify the screen components (problem E). Specifically, first, the size of the image area corresponding to the same part of the screen changes. In addition, even if either the image of the sample screen that has already been acquired or the image of the screen to be processed is enlarged so that the size of the image area corresponding to the same part of the screen becomes the same, they will not be the same because the image resolution is finite. This is not just because the smoothness of the lines is different, but depending on the characters displayed, some lines are omitted and the topology is different. Since the conventional identification method C does not compare the displayed character strings but rather the images of the screen that are the result of drawing them, it is theoretically impossible to completely avoid the failure to identify the screen components due to the image fluctuations described above, even if Scale-Invariant Feature Transform (SIFT) features that are not affected by the display magnification are used. In addition, as a result of problems E and E, as a problem in the operation of an automatic operation agent, etc., some of the identification operation settings must be redone, such as re-acquiring the template image.

[0054] Furthermore, in conventional identification method C, in the operation settings for identification, a person must determine and specify "what feature displayed on the screen should be used as a landmark to identify the screen component to be controlled" while assuming possible fluctuations that may occur on the screen to be processed, using the sample screen as a clue. In particular, when the screen component to be controlled itself does not have a landmark feature, it is necessary to find one or more screen components in the vicinity that can serve as a landmark, and prepare a condition for judging the equivalence of the screen components as a condition for the relative positional relationship with them, which is difficult (issue C).

[0055] (1-3. Object access and image recognition combined method) As mentioned above, the object access and image recognition combined method is a method in which both the object access method and the screen recognition method are executed simultaneously or switched between them. For example, a method (conventional identification method D) is known in which a screen is represented as a tree structure in which the screen components are nodes and their containment relationships are parent-child relationships, using at least one information resource from among screen component objects obtained through HTML (Hyper Text Markup Language) files or UIA, and the screen is identified as a screen component to be controlled.

[0056] Also, as a method for solving the problems when an automatic operation agent is executed on a plurality of terminals or GUI platforms with different settings, or when it is applied to an operation target application that has been partially changed due to a version upgrade, a method is known in which both identification processing by the object access method and identification processing by the screen recognition method are executed simultaneously, and the validity is verified by comparing the results. Also, a method is known in which identification by the object access method and automatic operation using the same are performed, while template images of screen components required for identification by the image recognition method are recorded, or conversely, identification by the image recognition method and automatic operation using the same are performed, while information on screen component objects required for identification by the object access method are recorded (conventional identification method E).

[0057] Conventional identification method D can be said to realize identification of screen components as a unified method by abstracting and treating information resources such as screen component objects and images. However, when a screen image or a fragment thereof is used as sample screen data, the screen component is identified by comparing the screen image as the screen data to be processed, and when information on screen component objects is used as sample screen data, the screen component is identified by comparing the information on screen component objects as the screen data to be processed. Therefore, problems A to E of the object access method and the image recognition method are not solved by this method. Also, when an image is used as an information resource, it is unclear how the "information resource acquisition program" creates a tree structure from the screen image, with the images of the screen components as nodes and their inclusion relationships as parent-child relationships, and problem F cannot be said to be solved either.

[0058] Conventional identification method E combines identification using an object access method and identification using an image recognition method for the same screen, and records scenarios (including information necessary for identification) corresponding to both methods through example operations on a terminal in which the application to be operated is not virtualized.On a terminal in which the application to be operated or the desktop is virtualized, only identification using the image recognition method is performed, making it possible to use the method without changing the scenario, and thus solving problems A and B.

[0059] Furthermore, by using identification by the object access method and identification by the image recognition method in a mutually complementary manner, even if the screen components cannot be identified by the object access method due to differences in the screen implementation contents between the sample screen data and the screen data to be processed, the screen components can be identified by the image recognition method as long as the differences in look and feel and screen display magnification are sufficiently small. Conversely, even if the screen configuration cannot be identified by the image recognition method due to differences in look and feel and screen display magnification, the screen components can be identified by the object access method as long as there are no differences in the screen implementation contents.

[0060] Furthermore, as described above, when identifying screen components using one method, the template image or object information of the screen components required for identifying the screen components using the other method can be recorded in parallel and made available for subsequent identification, enabling adaptive and evolutionary use. This allows for a cycle to be established in which, when changes to the screen implementation content and changes to the look and feel or display magnification of the screen occur sequentially over a period of time during the long-term operation of an automated operation agent or work analysis tool, the information required for identification using the method affected by the changes is updated to fit the screen after the changes. This can be said to solve problems C to D under certain conditions.

[0061] However, the use of both methods in a mutually complementary manner, and furthermore, the use of both methods in an adaptive and evolutionary manner based on that premise, is possible only when access to objects is possible, not just images on the screen, and problems e to e are not solved when using a virtualization environment that does not meet these conditions. This will be explained in more detail below.

[0062] When performing the identification process in a virtualized environment, the object cannot be accessed, and therefore the object access method cannot be used for supplementation. In addition, there may be operational restrictions, such as using a virtualized environment for routine work by workers, and using a non-virtualized environment for temporary purposes such as verifying the application to be operated, and only when permission is obtained from the organization that provides and operates the application to be operated. When using an automatic operation agent or the like as a means of improving work efficiency that can be promoted by the organization that performs terminal work, under such operational conditions, it is difficult to use a non-virtualized environment that can use both methods every time differences occur in the look and feel and display magnification caused mainly by differences in the environment on the terminal side, such as differences in the number of colors depending on the display environment and differences in options depending on the communication conditions when remotely logging in to a remote desktop. As a result, the information required for identification cannot be updated.

[0063] Furthermore, if the difference in look and feel is due to differences in options depending on the communication conditions when remotely logging in via remote desktop, even if a non-virtualized environment can be used, it is difficult to reproduce the same look and feel of a virtualized environment, and both methods cannot be used in an adaptive and evolutionary manner.

[0064] As a result, it is essentially the same as using the screen recognition method alone, and as a problem in the operation of an automated operation agent, etc., some of the recognition operation settings must be redone, such as re-acquiring the template image, as with the image recognition method alone. Even if the use is limited to a non-virtualized environment, if a difference in the screen implementation content and at least one of the look and feel or the display magnification of the screen occur simultaneously, neither method can correctly identify the screen components, and problems C to E are not solved. Furthermore, problem F regarding the specification of the conditions for determining the equivalence of the screen components, such as the area in the sample screen image used as the template image and the conditions of the relative positional relationship with them, which is necessary for the image recognition method, in the recognition operation settings, is not solved.

[0065] (1-4. Equivalence judgment condition method) As described above, the equivalence determination condition method is a method that uses a condition for determining equivalence of screen components instead of sample screen data. Methods are further divided depending on whether information of screen component objects or an image of a screen is used as the screen data to be processed. For example, under the premise that information of screen component objects is used as the screen data to be processed, a method (conventional identification method F) is known in which an "arrangement pattern" that represents the condition of the relative arrangement relationship between screen components on a sample screen (two-dimensional plane) is prepared as a condition for determining equivalence of the screen components to be controlled, and screen components that satisfy the arrangement pattern are searched for on the screen to be processed to identify the screen components (conventional identification method F).

[0066] In addition, assuming that a screen image is used as the screen data to be processed, a "screen model" is prepared as a graph in which fragments of the screen image corresponding to the screen components and regular expressions of displayed character strings are used as nodes, and their adjacent relationships on a sample screen (two-dimensional plane) are used as links. A method (conventional identification method G) is known in which the state of the screen is identified by matching the image of the screen to be processed and character strings obtained from a partial area of ​​the screen using optical character recognition (OCR) technology.

[0067] Conventional identification method F can be said to solve problems C to E by using an "arrangement pattern" that represents the conditions of the relative arrangement relationship between screen components on a sample screen (two-dimensional plane) as a condition for judging the equivalence of the screen components to be controlled. However, it is premised on using the information of screen component objects as the screen data to be processed, and problems A to B are not solved.

[0068] In addition, a method for automatically creating an arrangement pattern to be used for identifying a target screen from the relative arrangement relationship on a sample screen (two-dimensional plane) between screen components specified during the identification operation setting, and in particular, a method for determining to what extent the relative arrangement relationship on the sample screen should be reproduced on the target screen and reflected in the relative arrangement relationship conditions, is unclear. Therefore, in the current situation, a person must create the pattern while assuming the fluctuations that may occur on the target screen for each screen and screen component, and this problem remains unsolved.

[0069] Conventional identification method G can be said to solve problems A to C by preparing a "screen model" as a graph in which fragments of screen images corresponding to screen components, display character strings or their regular expressions are nodes, and their adjacent relationships on a sample screen (two-dimensional plane) are links, and matching with the image of the screen to be processed and character strings obtained from a partial area of ​​the screen using OCR technology. However, as with the layout patterns in conventional identification method F, currently the screen model needs to be created manually for each screen and screen component while anticipating fluctuations that may occur on the screen to be processed, and problem F is not solved.

[0070] Furthermore, character strings obtained from a partial area of ​​the image of the screen to be processed using OCR technology contain errors derived from the current OCR technology, and although it is expected that the effects of differences in look and feel and display magnification can be reduced, this leads to failures in matching with the display character string in the screen model or its regular expression. Therefore, it cannot be said that problems e to e have been fully solved. There is a possibility that matching failures can be reduced by using regular expressions of the display character string in the screen model that take into account errors in the character string obtained using OCR technology, but this will make it even more difficult to create the screen model, which will be counterproductive from the perspective of problem k.

[0071] (1-5. Screen Component Identification Process According to the Present Embodiment) The following describes the identification process of screen components according to this embodiment. First, the identification process of screen components according to this embodiment enables identification of screen components in a virtualized environment, contributing to the solution of problem A. Second, the identification process of screen components according to this embodiment does not require environmental changes on the server side of the thin client, thereby contributing to the solution of problem B. Third, the identification process of screen components according to this embodiment can correctly identify screen components even in a virtualized environment in which objects are inaccessible or in a non-virtualized environment in which objects are accessible, even in a situation in which differences in screen implementation contents occur, and even in a case in which there is a variation in the image of the sample and the screen to be processed due to differences in look and feel or display magnification, thereby contributing to the expansion of conditions for solving problems C to E. Fourth, the identification process of screen components according to this embodiment does not necessarily require manual creation of judgment conditions for the equivalence of the screen components to be controlled for each screen or screen component. Even when the operation settings for identification created in a non-virtualized environment are used to identify screen components in a virtualized environment, there is no need to manually change them, thereby contributing to the solution of problem F.

[0072] Furthermore, in the screen component identification process of this embodiment, the relative positional relationship between areas in the image of the screen to be processed where character strings are drawn (hereinafter referred to as "character string drawing areas") is compared with the relative positional relationship between character string drawing areas in the screen images of the sample screen data and the identified example screen data, and by using the relative positional relationship between screen components, it is possible to more accurately determine the relative positional relationship between areas where display character strings can be drawn, even with a smaller number of identified example screen data, and it becomes possible to more accurately identify the screen components.

[0073] (2. Conventional display string acquisition process) (2-1. Object access method) As mentioned above, the object access method does not use the screen image as the screen data to be processed, but uses information on screen component objects. In the above method, the character strings displayed in the screen components are often stored as attributes of the object, so they can be easily acquired by using UIA, MSAA, or an interface provided by the target application (conventional character string acquisition method A). However, the object access method has the same issues as issues A and B related to identifying screen components.

[0074] (2-2. Optical character recognition method) The optical character recognition method uses the image of the screen as the screen data to be processed, rather than the information of the screen component objects. With this method, character strings displayed on the screen components are also acquired from the image of the screen. One possible method for this is to use OCR technology. In the narrow sense, OCR technology refers to reading character strings from an image that is cropped to only the character string drawing area, but in the following explanation, the term "OCR technology" will also include image processing technology such as detecting the character string drawing area that is performed as preprocessing. One possible method for using OCR technology is to acquire the displayed character strings of multiple screen components all with the same settings, such as applying OCR technology to the entire screen image (conventional character string acquisition method B).

[0075] Another possible method is to use image processing technology to divide the screen image into screen components or character string regions, and then acquire the displayed character string with settings that reflect the conditions of the character string in each region, such as the number of characters, character type, font type and size, etc. (conventional character string acquisition method C).

[0076] Conventional character string acquisition methods B and C do not require information on the screen component object to be processed, and therefore can be said to solve problems A and B. However, when OCR technology acquires display character strings of multiple screen components drawn on a screen image with the same settings (conventional character string acquisition method B), many characters are acquired correctly, but errors are also included (problem G). For example, characters that are different in type but similar in shape, such as the number zero "0", the uppercase or lowercase alphabet "O" or "o", the number one "1" and the lowercase alphabet "l", the kanji "kuchi" (mouth) and the katakana "ro" (ro), are likely to be confused. In addition, even if limited to type, recognition without preconditions regarding the type and size of the font after reflecting the display magnification is also considered to be a cause of errors.

[0077] Conventional character string acquisition method C, when the screen of a business system is the recognition target, utilizes the fact that there are certain conditions for the displayed character string for each screen component, and specifies the number of characters, character type, font type and size, etc. (hereinafter referred to as "reading settings") for each area to be recognized so that these conditions can be used during recognition, thereby solving problem K. However, even if the font type and size in the image of the sample screen are specified as the reading settings, if there is a difference in look and feel or display magnification between the sample screen and the screen to be processed, the font type and size in the image to be processed will differ from those of the sample, and using them as is will not improve the recognition accuracy (problem K). In addition, it is time-consuming to manually specify the reading settings for each area of ​​the screen component to be the target of acquiring the displayed character string (problem I).

[0078] (2-3. Display character string acquisition process according to this embodiment) The following describes the process of acquiring display character strings according to this embodiment. First, the process of acquiring display character strings according to this embodiment enables acquisition of display character strings of screen components in a virtualized environment, and contributes to solving problem A. Second, the process of acquiring display character strings according to this embodiment does not require a change in the server-side environment of the thin client, and therefore contributes to solving problem B. Third, the process of acquiring display character strings according to this embodiment can use the reading settings related to font type and size to improve the recognition accuracy of OCR technology even if the look and feel or display magnification is different between the sample screen and the screen to be processed, and therefore contributes to solving problem K. Fourth, the process of acquiring display character strings according to this embodiment does not necessarily require manual setting of reading settings for each screen component, and therefore contributes to solving problems K and K.

[0079] (3. Overview of Classification Processing According to the Present Embodiment) (3-1. Prerequisites for this embodiment) First, the prerequisites for this embodiment will be described below. Many of the screens of applications targeted for operation by an automatic operation agent or the like are information input forms or information reference forms for business systems, and on these equivalent screens, although the displayed contents of each item and the number of items vary depending on the business situation or case, a certain regularity is maintained in the following respects.

[0080] First, even if there are differences in the screen implementation contents, look and feel, or display magnification, there is no difference in the relative positional relationship between equivalent screen components in an equivalent screen. Second, some strings displayed on equivalent screens are always the same regardless of the business situation or case, such as the item names in information input forms and information reference forms. Third, even if there are differences in the screen implementation contents, look and feel, or display magnification, there is no difference in the displayed strings as described above. Fourth, the font type and size of the strings displayed on the screen are not changed by the worker on a character or item basis, and are changed uniformly even if there are differences in the screen implementation contents, look and feel, or display magnification. In other words, if there are differences in the screen implementation contents, look and feel, or display magnification between screen components that use the same font type to draw the displayed string, the displayed string will still be drawn in the same font type even if the type is different from before the difference occurred. The font size ratio is also maintained.

[0081] Even if daily operations are performed in a virtualized environment, there are non-virtualized environments, such as a business system verification environment, in which the same target application can be used. Temporary use of a verification environment is different from changes to the commercial server side environment used in daily operations via a virtualized environment, and does not require measures such as investigation of the presence or absence of impact by the organization providing and operating the target application or strengthening server resources, making it a low barrier to implementation. In fact, when introducing an automated operation agent itself or deploying a scenario to multiple organizations, it is common to check the operation in advance in a verification environment in order to prevent overloading or erroneous operations in the commercial environment of the business system.

[0082] (3-2. Basic Concept of the Present Embodiment) The basic concept of this embodiment will be described below. In this embodiment, the actual screen of the application to be operated in a non-virtualized environment is used to set up and confirm the identification operation in the same manner as in the conventional object access method. At that time, the sample screen data used for the identification operation setting and the identification example screen data determined to be equivalent to the sample screen data when confirming the identification operation setting are acquired and stored, including information on the screen component objects. The information on these objects is used to identify the screen components in the virtualized environment and to read the displayed character strings.

[0083] Specifically, in order to solve the above problems A and B, in this embodiment, only the image of the screen is used as the processing target screen data acquired in the virtualized environment to identify the screen components. At that time, in order to expand the conditions for solving the above problems C to E, the character string drawn in the processing target screen image is compared with the display character string acquired as object information of the sample screen data or the identified example screen data, and their relative positional relationship. In other words, instead of the screen structure that is affected by the difference in the screen implementation content, the relative positional relationship on a two-dimensional plane that is not affected by the difference is used. Also, instead of the image that is affected by the difference in look and feel and display magnification, the display character string is used.

[0084] In addition, in order to solve the above-mentioned problem (c), the system identifies those pieces of identified example screen data that are equivalent to each sample screen data, and further identifies screen component objects (hereinafter referred to as "common objects") that exist and are displayed in common in the equivalent screen data. It then obtains the relative positional relationships that always hold between them and objects that always have the same displayed character strings, and uses these to compare with the character strings drawn in the image of the screen to be processed and their relative positional relationships.

[0085] Furthermore, in order to solve the above-mentioned problems K and KE, regularities such as the type of variation in the displayed character string of the common object, the number of characters, and the type of characters are obtained from the object information of the sample screen data and the equivalent identification example screen data, and these are used as reading settings when reading the character string drawing area in the processing target screen associated with the common object using OCR technology.

[0086] Furthermore, in order to solve the above-mentioned problems and to more reliably read drawn character strings in the image on the screen to be processed and identify areas in which known character strings are drawn, such as the display character string of an object whose display character string is always the same, the type and size of the font used when the display character string is drawn on the image on the screen to be processed is estimated.

[0087] [Configuration of the Identification Device 10, etc.] Next, the functional configuration of each device of the system 100 shown in Fig. 1 will be described. Here, the configuration of the identification device 10 and the like according to this embodiment will be described in detail, particularly with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the identification device and the like according to this embodiment. Below, the data and functional units stored in the storage unit as the configuration of the identification device 10 will be described, and the configuration of the automatic operation agent device 20 will be described.

[0088] (1. Configuration of Identification Device 10) (1-1. Overall Configuration of Identification Device 10) The identification device 10 has an input unit 11, an output unit 12, a communication unit 13, a storage unit 14, and a control unit 15. The input unit 11 is responsible for inputting various types of information to the identification device 10. For example, the input unit 11 is realized by a mouse, a keyboard, or the like, and accepts input of setting information, etc. to the identification device 10. The output unit 12 is responsible for outputting various types of information from the identification device 10. For example, the output unit 12 is realized by a display, etc., and outputs setting information, etc. stored in the identification device 10.

[0089] The communication unit 13 is responsible for data communication with other devices. For example, the communication unit 13 performs data communication with each communication device. The communication unit 13 can also perform data communication with an operator's terminal (not shown). In the above example, the communication unit 13 receives the first screen data 30 and the second screen data 40 from the automatic operation agent device 20. The communication unit 13 also stores the received first screen data 30 and second screen data 40 in the screen data storage unit 14a described below.

[0090] (1-2. Configuration of the storage unit 14 of the identification device 10) The storage unit 14 stores various information referenced when the control unit 15 operates and various information created as a result of the operation of the control unit 15. Here, the storage unit 14 can be realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Note that, in the example of Fig. 2, the storage unit 14 is installed inside the identification device 10, but it may be installed outside the identification device 10, or multiple storage units may be installed.

[0091] The storage unit 14 includes a screen data storage unit 14a, a processing result storage unit 14b, an identification information storage unit 14c, a first identification result storage unit 14d, an identification example storage unit 14e, a screen model storage unit 14f, a drawing area storage unit 14g, a layout relationship storage unit 14h, and a second identification result storage unit 14i. Examples of data stored in each storage unit will be described below with reference to Figs. 3 to 12.

[0092] (1-2-1. Screen data storage unit 14a) An example of data stored in the screen data storage unit 14a will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of data stored in the screen data storage unit according to the first embodiment. The screen data storage unit 14a stores "screen data to be processed." Here, the screen data to be processed is screen data acquired by the automatic operation agent device 20 from a screen displayed on an arbitrary terminal.

[0093] As shown in Fig. 3, the screen data to be processed includes "screen data ID", "screen component object information" including "screen component object attributes" and "screen structure", "screen image", "screen attributes", etc. Note that in Fig. 3, the "screen component drawing area", which is one of the "screen component object attributes", is written as [xxx,yyy,zzz,www], but even if the same characters are used, this does not mean that it is the same value like a mathematical algebraic expression, and each is an individual numerical value according to its respective drawing area, as in the example of screen data in Fig. 24.

[0094] (1-2-2. Processing result storage unit 14b) An example of data stored in the processing result storage unit 14b will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of data stored in the processing result storage unit according to the first embodiment. The processing result storage unit 14b stores "screen data processing results." Here, the screen data processing results are data processed by the control unit 22 of the automatic operation agent device 20, and are data indicating the results of associating the screen components to be controlled with the sample screen model, and the display character strings acquired from the screen data to be processed.

[0095] As shown in Figure 4, the screen data processing results include the "screen data ID to be processed," "sample screen model ID," "results of matching of controlled screen components and acquisition of display strings," etc. Here, "results of matching of controlled screen components and acquisition of display strings (1)" is data acquired in the mode using object information, and is an example where the screen data to be processed contains object information. On the other hand, "results of matching of controlled screen components and acquisition of display strings (2)" is data acquired in the mode not using object information, or is an example where the screen data to be processed does not contain object information.

[0096] In Fig. 4, the "character string drawing area," which is one piece of data, is written as [xxx,yyy,zzz,www]. However, even though the same characters are used, this does not mean that the value is the same like a mathematical algebraic expression. Rather, as in the example screen data in Fig. 24, each is an individual numerical value corresponding to the respective drawing area.

[0097] (1-2-3.Identification information storage unit 14c) An example of data stored in the identification information storage unit 14c will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of data stored in the identification information storage unit according to the first embodiment. The identification information storage unit 14c stores "identification information." Here, the identification information is a collection of sample screen data to be referred to during identification processing.

[0098] As shown in FIG. 5, the identification information includes the "sample screen data ID," "screen component object information" including the "screen component object attributes" and "screen structure," "screen image," and "screen attributes." The identification information also includes the above screen data for each of the multiple sample screen data. Note that in FIG. 5, the "screen component drawing area," which is one of the "screen component object attributes," is written as [xxx,yyy,zzz,www], but even if the same characters are used, this does not mean that the value is the same as in a mathematical algebraic expression, and each is an individual numerical value corresponding to the respective drawing area, as in the example of screen data in FIG. 24.

[0099] (1-2-4. First identification result storage unit 14d) An example of data stored in the first classification result storage unit 14d will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of data stored in the first classification result storage unit according to the first embodiment. The first classification result storage unit 14d stores a "first classification result." Here, the first classification result is data output by a first classification unit 151a of the classification device 10 described below, and is a collection of data indicating the correspondence between screen components of the processing target screen data and screen components of the sample screen data that have been determined to be equivalent. The first classification result may also be data acquired from another device via the communication unit 13.

[0100] As shown in FIG. 6, the first identification result includes identification results such as "sample screen data ID," "equivalence judgment result," and "screen component matching method." Here, the "screen component matching method" includes the screen component ID of the processing target screen data and the screen component ID of the corresponding sample screen data. The first identification result also includes the above identification result for each of the multiple sample screen data. Based on the first identification result, the "result of matching of control target screen components and display string acquisition (1)" to be stored in the processing result storage unit 14b is created.

[0101] (1-2-5.Identification case storage unit 14e) An example of data stored in the recognition case storage unit 14e will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of data stored in the recognition case storage unit according to the first embodiment. The recognition case storage unit 14e stores "recognition cases." Here, the recognition case is data output by a first recognition unit 151a of the recognition device 10 described below, and is data that accumulates recognition results for each processing target screen data that has been subjected to a recognition process in the past. The recognition case may also be data acquired from another device via the communication unit 13.

[0102] As shown in FIG. 7, the identification example includes "screen data to be processed" and "sample screen data ID" for each "screen data ID to be processed." Here, the "sample screen data ID" included in the identification example is not limited to those determined to be equivalent, and includes all sample screen data IDs stored in the identification information storage unit 14c. The identification example also includes identification results such as "equivalence determination result" and "screen component association method." That is, the identification example includes not only the screen data to be processed obtained by the most recent identification process (e.g., screen data to be processed ID: 20200204203243), but also the screen data to be processed obtained by the past identification process (e.g., screen data to be processed ID: 20200202101721).

[0103] (1-2-6. Screen model storage unit 14f) An example of data stored in the screen model storage unit 14f will be described with reference to Figs. 8 and 9. Figs. 8 and 9 are diagrams showing an example of data stored in the screen model storage unit according to the first embodiment. The screen model storage unit 14f stores a "sample screen model". Here, the sample screen model is data derived by a derivation unit 152a of the identification device 10 described later, and is data used in the identification process in the virtualized environment. The sample screen model may also be data acquired from another device via the communication unit 13.

[0104] 8 and 9, the sample screen model includes "sample screen data ID", "set of target identification example screen data IDs", as well as "screen component model information" including "screen component model attributes", "relative layout relationship of screen component models (horizontal direction)", and "relative layout relationship of screen component models (vertical direction)". In addition, the "screen component model attributes" also include "set of display character strings", "font type", "font size", etc. for each screen component.

[0105] (1-2-7.Drawing area storage section 14g) An example of data stored in the drawing area storage unit 14g will be described with reference to FIG. 10. FIG. 10 is a diagram showing an example of data stored in the drawing area storage unit according to the first embodiment. The drawing area storage unit 14g stores a "character string drawing area." Here, the character string drawing area is data identified by a second identification unit 153a of the identification device 10 described below, and is data indicating a drawing area of ​​a character string read using OCR technology. The character string drawing area may also be data acquired from another device via the communication unit 13.

[0106] As shown in Fig. 10, the character string drawing area includes a "read character string," a "character string drawing area," a "fixed value match flag," etc. for each "character string drawing area ID." The character string drawing area is data that is deleted or added as appropriate by processing such as OCR technology or image template matching.

[0107] In Fig. 10, the "character string drawing area," which is one piece of data, is written as [xxx,yyy,zzz,www]. However, even if the same characters are used, this does not mean that the value is the same like a mathematical algebraic expression. Rather, each will be an individual numerical value corresponding to the respective drawing area, as in the example screen data in Fig. 24.

[0108] (1-2-8. Layout relationship storage unit 14h) An example of data stored in the layout relationship storage unit 14h will be described with reference to FIG. 11. FIG. 11 is a diagram showing an example of data stored in the layout relationship storage unit according to the first embodiment. The layout relationship storage unit 14h stores "character string drawing area layout relationship." Here, the character string drawing area layout relationship is data determined by a second identification unit 153a of the identification device 10 described below, and is data indicating the relative layout relationship between any two character string drawing areas. In addition, the character string drawing area layout relationship may be data acquired from another device via the communication unit 13.

[0109] 11, the character string drawing area layout relationship includes "relative layout relationship of the character string drawing areas (horizontal direction)" and "relative layout relationship of the character string drawing areas (vertical direction)." Furthermore, the character string drawing area layout relationship is data determined from the character string drawing areas stored in the above-mentioned drawing area storage unit 14g.

[0110] (1-2-9. Second Classification Result Storage Unit 14i) An example of data stored in the second classification result storage unit 14i will be described with reference to FIG. 12. FIG. 12 is a diagram showing an example of data stored in the second classification result storage unit according to the first embodiment. The second classification result storage unit 14i stores a "second classification result." Here, the second classification result is data output by a second classification unit 153a of the classification device 10 described later, and is data indicating a correspondence between a character string drawing area of ​​the screen data to be processed and a screen component of the sample screen model. The second classification result may also be data acquired from another device via the communication unit 13.

[0111] As shown in Fig. 12, the second identification result includes identification results such as "sample screen data ID" and "association method." Here, the "association method" includes the character string drawing area ID of the processing target screen data and the screen component ID of the associated sample screen model. Based on the second identification result, the "result of association of control target screen components and display character string acquisition (2)" to be stored in the processing result storage unit 14b is created.

[0112] (1-3. Configuration of the control unit 15 of the identification device 10) The control unit 15 controls the entire identification device 10. The control unit 15 has a first identification unit 151a and a first acquisition unit 151b as a first screen data control unit 151, a derivation unit 152a as a screen model control unit 152, and a second identification unit 153a and a second acquisition unit 153b as a second screen data control unit 153. Here, the control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0113] (1-3-1. First Screen Data Control Unit 151: First Identification Unit 151a) The first identification unit 151a identifies the first screen data 30 including an image of the application screen and information about screen component objects that are objects of elements that configure the screen, and outputs a first identification result that corresponds to the sample screen data that is the screen data to be referenced. For example, the first identification unit 151a identifies the first screen data 30 including a first character string and a drawing area of ​​a screen component object that has the first character string as an attribute by determining equivalence using information about the screen component object, and outputs a first identification result that corresponds to the screen component object for each sample screen data determined to have equivalence. Here, the first character string is a display character string included in the first screen data 30, and includes a character string that is displayed as a screen image as well as a character string that is in a hidden state.

[0114] To explain the details of the process, first, the first identification unit 151a acquires the first screen data to be processed from the screen data storage unit 14a, and acquires the identification information of the sample screen data from the identification information storage unit 14c. Next, the first identification unit 151a judges the equivalence between the first screen data and the sample screen data using information on the screen component object included in the identification information (attributes of the screen component object such as the type of object, the display character string, and the drawing area of ​​the screen component, and the screen structure). Then, the first identification unit 151a outputs a first identification result in which the screen component ID in the first screen data to be processed corresponds to the screen component ID of the sample screen data for each sample screen data ID judged to have equivalence.

[0115] The first classification unit 151a also stores the output first classification result in the first classification result storage unit 14d. Furthermore, the first classification unit 151a stores the output first classification result as a classification case in the classification case storage unit 14e.

[0116] (1-3-2. First Screen Data Control Unit 151: First Acquisition Unit 151b) Based on the first identification result, the first acquisition unit 151b acquires a first character string included in the first screen data 30. For example, the first acquisition unit 151b uses the first identification result to acquire the first character string from the processing target screen data associated with the screen component object.

[0117] To explain the details of the process, first, the first acquisition unit 151b selects one result that is determined to be equivalent from the first identification results stored in the first identification result storage unit 14d and that is based on a preset index such as the highest priority given to the sample screen data in advance, the best evaluation value of the association method, etc. Next, the first acquisition unit 151b acquires the sample screen data determined to be equivalent to the processing target screen data in the selected first identification result from the identification information storage unit 14c, and identifies the object to be controlled and the object to be displayed, which is a part of the object, in the sample screen data.

[0118] Furthermore, the first acquisition unit 151b uses the result of associating the object in the sample screen data with the object in the processing target screen data, which is included in the selected first identification result, to identify the object to be controlled and the object to be the display string acquisition target in the processing target screen data, and stores them in the processing result storage unit 14b. Furthermore, the first acquisition unit 151b acquires a display string from the information of the object in the processing target screen data for each of the objects identified as the display string acquisition target in the processing target screen data, and reflects the result in the processing result storage unit 14b. Note that, if there is no identification result determined to be equivalent to the processing target screen data, the first acquisition unit 151b does not store anything in the processing result storage unit 14b.

[0119] (1-3-3. Screen model control unit 152: Derivation unit 152a) The derivation unit 152a derives a sample screen model to be used for identification in a virtualized environment based on the sample screen data and the first identification result. For example, the derivation unit 152a derives a sample screen model including a relative positional relationship for each drawing area of ​​the first character string using an identification example including a plurality of first identification results.

[0120] To explain the details of the process, first, the derivation unit 152a acquires the identification information of the sample screen data from the identification information storage unit 14c, and acquires the identification case from the identification case storage unit 14e. Next, the derivation unit 152a determines the relative arrangement relationship for each drawing area of ​​the screen component objects included in the identification case for each sample screen data, and outputs it as a sample screen model. In addition, the derivation unit 152a stores the output sample screen model in the screen model storage unit 14f. The flow of the sample screen model derivation process by the derivation unit 152a will be described later in [Flow of Each Process] (2. Flow of Sample Screen Model Derivation Process).

[0121] Furthermore, prior to identifying the second screen data, the derivation unit 152a identifies common objects that are included in common to multiple first screen data among the screen component objects of the sample screen data from the sample screen data and the first identification result, determines the relative positional relationship of these common objects for each drawing area, and derives a sample screen model including them.

[0122] Furthermore, the derivation unit 152a identifies fixed value objects that are included in common in a plurality of first screen data 30 and have the same character string among the screen component objects of the sample screen data included in the identification example, and derives a sample screen model. Here, the derivation unit 152a identifies, among the screen component objects of the sample screen data, those that always exist and are displayed on an equivalent screen, and whose displayed character string is always the same, as fixed value objects, and outputs them as the sample screen model.

[0123] Furthermore, the derivation unit 152a uses the sample screen model to derive a sample screen model that further includes at least one of the type of variation, the number of characters, the type of characters, the type of font, and the size of the first character string. At this time, the derivation unit 152a generates a check image of the first character string included in the sample screen model, and derives the sample screen model that further includes the type and size of the font of the first character string by matching the check image with an image of the screen included in the sample screen data. Details of the font estimation process by the derivation unit 152a will be described later in [Details of each process] (4-2. Font estimation process when the character string drawing area is known).

[0124] (1-3-4. Second Screen Data Control Unit 153: Second Identification Unit 153a) The second identification unit 153a identifies the second screen data 40, which includes an image of the screen but does not include information on the screen component object, and outputs a second identification result associated with the sample screen model. For example, the second identification unit 153a uses optical character recognition processing to identify the second character string and the drawing area of ​​the second character string from the image of the screen, determines the relative positional relationship for each drawing area of ​​the second character string, identifies the second screen data 40 based on the drawing area of ​​the second character string and the relative positional relationship for each drawing area of ​​the second character string, and outputs a second identification result associated with the screen component object for each sample screen model. Here, the second character string is a display character string included in the second screen data 40. In addition, the second identification unit 153a identifies the second image data 40 by estimating the font type or size of the second character string using the font type or size of the character string derived by the derivation unit 152a, and outputs the second identification result.

[0125] At this time, the second identification unit 153a identifies the second screen data 40 by determining equivalence using constraints based on the existence of a drawing area for the second string, the strings drawn in the drawing area for the second string, and the relative positional relationship of each drawing area for the second string, and a predetermined evaluation function, and outputs a second identification result that corresponds to a screen component object for each sample screen model.

[0126] Explaining the details of the process, the second identification unit 153a applies the following character string drawing area identification process, character string drawing area layout relationship determination process, and second screen data matching process in order to each of the sample screen models stored in the screen model storage unit 14f (hereinafter referred to as the "selected sample screen model").

[0127] (1-3-4-1. Character string drawing area specification process) First, the second identification unit 153a obtains a display string of a screen component model (hereinafter, referred to as a "fixed value screen component model") in which the same display string is always drawn in an equivalent screen image from the selected sample screen model. Here, the fixed value screen component model is a screen component model in which the "occurrence rate" is 1, the "number of blank characters" is 0, and the "type of variation of display string" is a "fixed value" in the information of the screen component model. In addition, the second identification unit 153a identifies the display string and the drawing area of ​​the display string from the image of the screen to be processed using optical character recognition processing. Then, the second identification unit 153a stores the drawing area of ​​the identified display string (second string) in the drawing area storage unit 14g. Details of the string drawing area identification processing by the second identification unit 153a will be described later in [Details of each processing] (3-1. String drawing area identification processing).

[0128] (1-3-4-2. Character string drawing area layout relationship determination process) Next, the second identification unit 153a determines a relative positional relationship for each combination of character string drawing areas stored in the drawing area storage unit 14g, and stores the determined positional relationship in the positional relationship storage unit 14h. Details of the character string drawing area positional relationship determination process performed by the second identification unit 153a will be described later in [Details of Each Process] (3-2. Character string drawing area positional relationship determination process).

[0129] (1-3-4-3. Corresponding process of second screen data) Then, the second identification unit 153a obtains a correspondence method between the screen component model of the sample screen model and the character string drawing area of ​​the image of the screen to be processed, and its evaluation value, so as to satisfy the constraint conditions, based on the relative arrangement relationship between the character string drawing area in which the display character string of the fixed value screen component model stored in the drawing area storage unit 14g is drawn in the sample screen model and the image of the screen to be processed, and the character string drawing area stored in the arrangement relationship storage unit 14h. At this time, if a correspondence method that satisfies the constraint conditions is obtained and a correspondence method has not been obtained for other sample screen models, or if the evaluation value of the correspondence method obtained for other sample screen models is better than that of the correspondence method obtained for other sample screen models, the second identification unit 153a updates the second identification result storage unit 14i with the newly obtained correspondence method (second identification result). Details of the correspondence process of the second screen data by the second identification unit 153a using the constraint conditions and the evaluation function will be described later in [Details of each process] (3-3. Correspondence process of second screen data).

[0130] (1-3-5. Second Screen Data Control Unit 153: Second Acquisition Unit 153b) The second acquiring unit 153b acquires a second character string included in the second screen data based on the second identification result. For example, the second acquiring unit 153b acquires the second character string from the processing target screen data associated with the screen component object by using the second identification result.

[0131] Furthermore, the second acquisition unit 153b acquires a second character string included in the second screen data based on the second identification result and at least one of the type of variation, the number of characters, the type of character, the type of font, and the size of the first character string included in the sample screen model.

[0132] The details of the process are as follows: first, when the second identification result storage unit 14i stores the identification result, the second acquisition unit 153b acquires the sample screen model determined to be equivalent to the screen data to be processed from the screen model storage unit 14f, and identifies the screen component model of the control target and the screen component model of the display string acquisition target, which is a part of the control target, in the sample screen model. The second acquisition unit 153b also uses the correspondence result between the screen component model in the sample screen model and the character string drawing area in the image of the screen to be processed, which is included in the second identification result, to identify the character string drawing area of ​​the control target and the character string drawing area of ​​the display string acquisition target in the image of the screen to be processed, and stores the result in the processing result storage unit 14b. Furthermore, the second acquisition unit 153b acquires the display string for each of the character string drawing areas of the display string acquisition target in the image of the screen to be processed, and reflects the result in the processing result storage unit 14b. Details of the second character string acquisition process by the second acquisition unit 153b will be described later in [Details of Each Process] (3-4. Display String Acquisition Process).

[0133] (2. Configuration of the Automatic Operation Agent Device 20) The automatic operation agent device 20 has a communication unit 21 that controls the transmission and reception of various data between other devices, and a control unit 22 that controls the entire automatic operation agent device 20.

[0134] (2-1. Communications Department 21) The communication unit 21 transmits the first screen data 30 and the second screen data 40 acquired by the control unit 22 to the identification device 10. The communication unit 21 also receives from the identification device 10 the first character string and the second character string as well as information required for identifying the screen component to be controlled, such as the screen component ID at the time of execution and the drawing area.

[0135] (2-2. Control unit 22) The control unit 22 acquires first screen data 30 including an image of the application screen and information on screen component objects which are objects of elements that configure the screen. Similarly, the control unit 22 acquires second screen data 40 including an image of the screen but not information on the screen component objects. Furthermore, the control unit 22 uses the screen data processing results stored in the processing result storage unit 14b to execute controls such as operations on the screen components to be controlled that were specified during prior operation settings using the sample screen data and processes using the acquired display character strings.

[0136] [Details of each process] The details of each process according to this embodiment will be described using Fig. 13 to Fig. 16, formulas, etc. In the following, an overview of the font estimation process for a character string displayed in an image will be described as a process common to a plurality of processes, and then the sample screen model control process, the second screen data control process, and the font estimation process for a character string displayed in an image will be described in detail.

[0137] In the following explanation, the horizontal (up and down) and vertical (left and right) layout relationships are treated as relative layout relationships for all combinations of screen components that always exist in an equivalent screen, and the constraint conditions and evaluation functions are also set accordingly. However, other relative layout relationships, constraint conditions, and evaluation functions may be used, for example, by limiting the combinations to only those of adjacent screen components, or by considering not only up and down and left and right but also relative distances, as long as they can be automatically derived from information on the screen component objects of the sample screen and an equivalent identified example screen.

[0138] (1. Overview of font estimation process for displayed character strings in images) An outline of font estimation processing for a display character string in an image will be described with reference to FIG. 13. FIG. 13 is a diagram showing an example of processing for estimating the type and size of a font for a display character string according to the first embodiment. In the following, processing for estimating the type and size of a font used when a display character string is drawn in an image on a processing target screen will be described in order to more reliably read a drawn character string in an image on a processing target screen and specify an area in which a known character string is drawn, such as a display character string of an object whose display character string is always the same. Details of the font estimation processing for a display character string in an image will be described later in (5. Font Estimation Processing for Display Character String in an Image).

[0139] First, in sample screen data that has acquired both the screen image and object information, the drawing area of ​​the screen components and their display strings can be determined from the object information, and the font type and size in the sample screen image for each object are obtained by utilizing this. That is, in Fig. 13, in the screen image of the sample screen data, the font type of the display string for "Order details" is specified as "Meiryo" and the font size as "c0 pt" using the object information (see Fig. 13(1-1)). Similarly, the font type of the display string for "Contract type" is specified as "Meiryo" and the font size as "d0 pt" (see Fig. 13(1-2)).

[0140] Secondly, for objects whose display string and drawing area are relatively easily identifiable, the display string and drawing area are obtained from the image on the screen to be processed, and the font type and size are also obtained, and differences in the font type and size in the sample screen image are examined. That is, in Fig. 13, the font type of the display string of "Order details", an object that is relatively easily identifiable, is specified as "MS P Mincho" and the font size as "c1 pt" (see Fig. 13(2)).

[0141] Thirdly, the above two types of processing results are combined to infer the font type and size. That is, in Fig. 13, the font type changes from "Meiryo" to "MS P Mincho" and the font size changes to "c1 / c0 times" between the sample screen data and the image of the processing target screen, so it is inferred that the font type of the display character string of "Contract type" in the image of the processing target screen is "MS P Mincho" and the font size is "d0 x c1 / c0 pt" (see Fig. 13(3)).

[0142] (2. Sample screen model control process) Prior to the classification process, the sample screen model control process for deriving a sample screen model as intermediate data will be described in detail below. The process will be described in the following order: classification example acquisition process, sample screen model initialization process, sample screen model update process, display character string regularity derivation process, font derivation process for the sample screen image, and fixed-value screen component model matching success rate and matching evaluation value calculation process.

[0143] (2-1. Acquisition process of identification cases) For the selected sample screen data, the classification device 10 acquires, from among the classification example screen data and classification results stored in the classification example storage unit 14e, those determined to be equivalent to the sample screen data.

[0144] (2-2. Initialization process of sample screen model) For the selected sample screen data, the identification device 10 initializes a sample screen model corresponding to the selected sample screen data (hereinafter referred to as the "selected sample screen model") based on the object information of the sample screen data itself as follows.

[0145] (2-2-1. Initialization process of sample screen model ID and target identification example screen data ID set) The value of the "sample screen data ID" of the selected sample screen data is used as the sample screen model ID. Also, the "target identification example screen data ID set" is set to empty.

[0146] (2-2-2. Initialization of screen component model attributes) First, prepare one screen component model for each object in the selected sample screen data. Also, initialize the attributes of each screen component model as follows:

[0147] First, the attributes of the screen component model, such as "runtime screen component ID," "type," "control target," and "display string acquisition target," inherit the values ​​of the object as is.

[0148] Secondly, the "occurrence count" attribute is initialized to 1 for the modeled objects for which a display string may be drawn if their "visible / hidden state" is "visible", and to 0 otherwise. The possibility of a display string being drawn may be determined based on conditions related to the type of screen component etc. that are prepared in advance according to the means of acquiring information on the screen component object. For example, even if a window holds a display string as the window title, it is not considered if it is known that the display string will not be drawn on the screen.

[0149] Third, the attribute "null string count" is initialized to 1 if the "visibility state" of the modeled object is "visible" and the "display string" is a null string, and to 0 otherwise.

[0150] Fourth, the "display string set" attribute is initialized with the string of the "display string" of the object being modeled.

[0151] On the other hand, the attributes of the screen component model, namely “appearance rate,” “type of change in display string,” “number of characters in display string,” “type of characters in display string,” “font type,” and “font size,” are left unset at the time of initialization, as they will be set in subsequent processing.

[0152] (2-2-3. Initialization process of relative position relationship of screen component model) The identification device 10 identifies two arbitrary screen component models u in the sample screen model, on which a display character string may be drawn. i Tou j The relative positional relationship between the above objects is examined using the drawing areas of the objects in the sample screen data that are the subject of modeling.

[0153] As a result, u i Tou j r represents the horizontal arrangement of h (i,j) and r h Determine the value of (j,i). Note that "0" means "undefined."

[0154] u i Ga u j To the left of u i The right end of u j If it is to the left of the left end of r h (i,j)=1,r h Let (j,i) = -1.

[0155] u i Ga u j To the right of u i The left end of u j If it is to the right of the right end of r h (i,j)=-1,r h Let (j,i) = 1.

[0156] In other cases, r h (i,j)=0,r h Let (j,i) = 0.

[0157] Similarly, u i Tou j r represents the vertical arrangement of l (i,j) and r l Determine the value of (j,i).

[0158] u i Ga u jAbove, i.e., u i The lower end of u j If it is above the upper end of r h (i,j)=1,r h Let (j,i) = -1.

[0159] u i Ga u j , i.e., u i The top of u j If it is below the lower end of r h (i,j)=-1,r h Let (j,i) = 1.

[0160] In other cases, r h (i,j)=0,r h Let (j,i) = 0.

[0161] In addition, the information of the screen component objects is in an inclusive relationship, h (i,j)=0,r l For the screen component for which (i,j)=0, if the layout relationship between the areas in which the display string can be drawn can be further limited by conditions related to the type of screen component, etc., which are prepared in advance according to the means of acquiring information on the screen component object, etc., then r h (i,j) and r l A value reflecting this may be used as the value of (i, j).

[0162] (2-3. Updating sample screen model based on identified cases) (2-3-1. Selection of identification cases and determination of whether or not to reflect in the model) The classification device 10 selects the acquired object-attached screen data classification cases in order (hereinafter referred to as "selected classification example screen data" and "selected classification result"). If the "screen data ID" of the selected classification example screen data is not included in the "target classification example screen data ID set" of the selected sample screen model, the classification device 10 uses the selected classification example screen data and the classification result to update the sample screen model through subsequent processing. Note that in the classification result, the object of the selected classification example screen data that is associated with the object in the sample screen data that is the modeling target is referred to as the "model reflection target object."

[0163] (2-3-2. Update process of the data ID set for the object identification example screen) The classification device 10 adds the value of the "screen data ID" of the selected classification example screen data to the "target classification example screen data ID set" of the selected sample screen model.

[0164] (2-3-3. Updating attributes of screen component model) When a model reflection target object exists, the identification device 10 updates the attributes of each screen component model as follows.

[0165] First, for the attribute "occurrence count", if the "display / hide state" of the model reflection target object is "display", add 1. However, if there are multiple model reflection target objects in the same identification example screen data due to the presence of a repetitive structure such as a list, add 1 a maximum of once per identification example screen data, not for each model reflection target object.

[0166] Secondly, for the attribute "display string set", if the string in the "display string" of the model reflection target object is not included, it is added.

[0167] Third, for the attribute "null string count", if the "visibility state" of the model reflection target object is "visible" and the "display string" is a null string, add 1.

[0168] (2-3-4. Updating the relative positional relationship of screen component models) The identification device 10 identifies two arbitrary screen component models u i Tou j For each of the model reflection target objects u', f(i) andu' f(j) If the object is associated with the model object, the drawing area of ​​the object to be modeled is checked to check their relative positional relationship.

[0169] As a result, u' f(i) andu' f(j) r', which represents the horizontal arrangement of h (i,j) and r' h Determine the value of (j,i).

[0170] u' f(i) u' f(j) to the left of u', i.e. f(i) The right end of u' f(j) If it is to the left of the left end of r' h (i,j)=1, r' h Let (j,i) = -1.

[0171] u' f(i) u' f(j) to the right of u', i.e. f(i) The left end of u' f(j) If it is to the right of the right end of r' h (i,j)=-1, r' h Let (j,i) = 1.

[0172] In all other cases, r' h (i,j)=0, r' h Let (j,i) = 0.

[0173] Similarly, u' f(i) andu' f(j) r' represents the vertical arrangement of l (i,j) and r' lDetermine the value of (j,i).

[0174] u' f(i) u' f(j) above, i.e., u' f(i) The lower end of u' f(j) If it is above the upper end of r' l (i,j)=1, r' l Let (j,i) = -1.

[0175] u' f(i) u' f(j) u', i.e., u' f(i) The top of u' f(j) If it is below the lower end of r' l (i,j)=-1, r' l Let (j,i) = 1.

[0176] In all other cases, r' l (i,j)=0, r' l Let (j,i) = 0.

[0177] As with the initialization of the relative layout relationship of screen components in the sample screen model, the screen component objects are in an inclusive relationship, and r' h (i,j)=0, r' l For the screen components where (i,j)=0, if the layout relationship between the areas in which the display character string can be drawn can be further limited by taking into account the type of screen component, etc., then r' h (i,j) and r' l A value reflecting this may be used as the value of (i, j).

[0178] Then, any two screen component models u in the sample screen model i Tou j r represents the relative positional relationship of h (i,j), r h (j,i),r l (i,j), r lThe value of (j,i) is updated as follows. In other words, if the left-right and up-down relationships always hold, they are maintained, and if not, they are set to "indeterminate."

[0179] (Update horizontal relative positioning) r h (i,j) ≠ r' h In the case of (i,j) (in this case r h (j,i) ≠ r' h (j,i) is r h (i,j)=0,r h Let (j,i) = 0. On the other hand, in cases other than the above, no update is performed.

[0180] (Update relative vertical alignment) r l (i,j) ≠ r' l In the case of (i,j) (in this case r l (j,i) ≠ r' l (j,i) is r l (i,j)=0,r l Let (j,i) = 0. On the other hand, in cases other than the above, no update is performed.

[0181] As described above, when determining the relative positional relationship that always holds between any two screen component models in the sample screen model, the "drawing areas of the objects themselves" of the sample screen data or equivalent identification example screen data are compared. On the other hand, when determining the relative positional relationship, the positional relationship may also be examined between the "drawing areas of the display character strings" in the image.

[0182] Using Fig. 14, we will explain a case where the layout relationship of drawing areas changes depending on the number of characters in the display string of screen components. Fig. 14 is a diagram showing an example where the difference in the number of characters in the display string affects the relative layout relationship of the character string drawing areas. If a sufficient number of variations are not covered as identification example screen data, the layout relationship that should be "indefinite" will be erroneously reflected in the sample screen model as a "left-right" or "top-bottom" relationship (see Fig. 14(1)). On the other hand, such a problem can be avoided by checking the layout relationship between the drawing areas of the objects themselves (see Fig. 14(2)).

[0183] In other words, the process of determining the character string drawing area positional relationship uses the relative positional relationship between screen components as a comparison target for the relative positional relationship between character string drawing areas in the screen images of the sample screen data and the identified example screen data, compared to when the relative positional relationship between character string drawing areas in the screen images of the sample screen data and the identified example screen data.By using the relative positional relationship between screen components as a comparison target for the relative positional relationship between character string drawing areas in the screen image of the target screen, the process of determining the character string drawing area positional relationship can more accurately determine the relative positional relationship between areas where display character strings can be drawn, even with a smaller number of identified example screen data, and makes it possible to more accurately identify the screen components.

[0184] (2-3-5. Calculation process of appearance ratio of screen component model) When the classification device 10 has completed the above process for all acquired object-attached screen data classification cases, it divides the “number of occurrences” of each screen component model by the number of elements in the “set of target classification case screen data IDs” plus 1, and sets the resulting value as the “occurrence ratio.”

[0185] (2-4. Deriving the regularity of displayed character strings) The identification device 10 first determines the "type of display string variation" for screen component models in the sample screen model that have one or more model reflection target objects in all identified example screen data equivalent to the selected sample screen data (hereinafter referred to as "common screen component models") as follows: A common screen component model is one whose "appearance rate" is 1 in the screen component model information.

[0186] If the number of elements in the "display string set" is 1, set it to a "fixed value."

[0187] If the number of elements in the "display character string set" is greater than 1 and is equal to or less than a predetermined threshold, it is set to a "category value."

[0188] If the number of elements in the "display character string set" is greater than a predetermined threshold, it is set to an "arbitrary value."

[0189] Furthermore, the identification device 10 performs the following for those for which the "type of variation in display character string" is "arbitrary value."

[0190] (Number of characters in the display string) The lengths of the strings contained in the "display string set" are checked, and if they are all the same, that length is set.

[0191] (Character type of display string) It is checked whether the characters in the strings included in the "display string set" include alphabets (uppercase and lowercase), numbers, hiragana, katakana, kanji, etc., and the type of characters included is set.

[0192] (2-5. Font derivation process for sample screen image) The process of deriving the type and size of the font used to draw the image of the sample screen for the display character string of the object that is the subject of modeling of each common screen component model will be described.

[0193] Depending on the interface provided by the application to be operated, it may be possible to obtain information such as the overall screen display magnification, font type, and font size when the overall screen display magnification is 100%. In this case, by using this interface, it is possible to obtain the font type and size used to draw the image on the sample screen. The font size is obtained by multiplying the font size when the overall screen display magnification is 100% by the overall screen display magnification.

[0194] On the other hand, depending on the interface provided by the target application, the font type and size may not be obtained correctly. In that case, they are estimated from the sample screen image as follows. First, it is known that the displayed string is drawn within the drawing area of ​​the object, and that no other strings are drawn. Therefore, by applying OCR technology to the drawing area of ​​the object in the sample screen image, it is possible to identify the drawing area of ​​the displayed string of the object. After that, the font type and size are obtained for this string drawing area by performing font estimation for when the string drawing area is known, as described below.

[0195] (2-6. Calculation of matching success rate and matching evaluation value of fixed value screen component model) The process of calculating a matching success rate and a matching evaluation value for each fixed-value screen component model will be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of a matching process when estimating the font type and size of a display character string according to the first embodiment.

[0196] First, the identification device 10 creates an image (hereinafter referred to as an "image for matching suitability investigation") in which a display string (the only element of a display string set) of a fixed-value screen component model is drawn while changing the font type and size within candidates designated prior to the implementation of the present invention. This image can be created, for example, by using the functions of the OS on which the program according to this embodiment runs to display the screen of the program according to this embodiment on a display, drawing the display string thereon in a specific font type and size, and capturing an image of the drawn area (see Fig. 15 (1) and (2)).

[0197] Next, the classification device 10 uses image processing techniques such as image template matching that use features such as SIFT features that are not easily affected by differences in the size of what is drawn in the image (in this case, characters) to match the image of the sample screen with each image for the matching suitability survey (see Figure 15 (3)).

[0198] Based on the matching result, the classification device 10 judges whether the matching is successful, that is, whether the area identified as a result of the matching is included in the drawing area of ​​the object that was the subject of modeling of the fixed-value screen component model. Furthermore, if the matching is successful, the classification device 10 checks the similarity output by the image processing technology and sets it as a matching evaluation value (see FIG. 15(4)).

[0199] Then, the classification device 10 calculates the matching success rate and matching evaluation value of the object from the matching success rate and matching evaluation value for all the matching suitability investigation images. Here, the matching success rate is the rate of the matching success rate of the matching suitability investigation images for which the matching success rate is "successful", and the matching evaluation value is the minimum, average, median, or the like of the matching evaluation value of the matching success rate of the matching suitability investigation images for which the matching success rate is "successful".

[0200] In the above, it has been described that each matching suitability investigation image is matched only with the sample screen image, but the identification example screen data obtained in the preceding process and determined to be equivalent to the sample screen data may be included in the matching target. In this case, the identification device 10 determines whether or not the matching is successful based on whether or not the identification example screen data, which corresponds to the fixed value screen component model, is included in the drawing area of ​​the model reflection target object in the identification example screen data.

[0201] In Fig. 15, the "Screen component drawing area" is written as one of the "Attributes of screen component model" in the sample screen data as [xxx,yyy,zzz,www]. However, even though the same characters are used, this does not mean that the value is the same like a mathematical algebraic expression. Rather, like the example screen data in Fig. 24, each is an individual numerical value corresponding to the respective drawing area.

[0202] (3. Second screen data control processing) The second screen data control process, which compares the target screen data, which does not include object information, with the information of the screen component objects of the sample screen and an equivalent identification example screen, and acquires the display string, is described in detail below. The process of identifying the string drawing area, the process of determining the layout relationship of the string drawing area, the process of associating the second screen data, and the process of acquiring the display string are described in this order.

[0203] (3-1. Character string drawing area specification process) The process of identifying a character string drawing area in the second screen data control process will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of the process of identifying a character string drawing area according to the first embodiment. The identification device 10 identifies an area in the image of the screen to be processed where a character string is drawn, and among that, an area where the display character string of the fixed value screen component model in the selected sample screen model is drawn, as follows.

[0204] First, the identification device 10 uses OCR technology on the entire image of the screen to be processed to obtain the area determined to have a character string drawn and the character string read from the image of that area (hereinafter referred to as the "read character string"), and stores them in association with each other in the drawing area storage unit (see FIG. 16(1)). Note that in FIG. 16, even if the same characters are used in the parts written as [xxx, yyy, zzz, www] as the obtained "character string drawing area", this does not mean that this part has the same value like a mathematical algebraic expression, and each has an individual numerical value corresponding to the respective drawing area, as in the example of screen data in FIG. 24.

[0205] Next, the identification device 10 checks and classifies each of the fixed-value screen component models in the selected sample screen model as to whether the display string is included in the multiple set of read strings (hereinafter referred to as a "detected fixed-value screen component model") or not (hereinafter referred to as an "undetected fixed-value screen component model"). The identification device 10 also sets a fixed value match flag in the string drawing area associated with the read string that matches the display string of the fixed-value screen component model (see FIG. 16(2)).

[0206] If there are multiple fixed-value screen component models with the same display character string in the sample screen model, the identification device 10 classifies them taking into consideration whether the display character string is included in the multiple set of the read character string for each of the multiple models. For example, if the multiple set of the read character string does not include all of the multiple models, the identification device 10 temporarily treats all of them as undetected fixed-value screen component models, and does not set the fixed value match flag in the character string drawing area.

[0207] Then, the identification device 10 performs one or both of the following processes for each undetected fixed-value screen component model to identify the drawing area of ​​the displayed character string and to correct the read character string and the character string drawing area stored in the character string drawing area storage unit.

[0208] (3-1-1. Detecting the drawing area of ​​a displayed character string using optical character verification technology) The identification device 10 identifies the drawing area of ​​the display character string of the undetected fixed-value screen component model in the image of the screen to be processed using optical character verification (OCV) technology (see FIG. 16(3-1)). At this time, the character string drawing area for which the fixed value match flag is set at this point is excluded from the scanning target by the OCV technology. At this time, for the undetected fixed-value screen component model for which the drawing area has been identified, the identification device 10 corrects the read character string and the character string drawing area by the method described below using the display character string and its drawing area. The identification device 10 also reclassifies the model into a detected fixed-value screen component model.

[0209] (3-1-2. Detecting the drawing area of ​​the displayed character string by comparing with a fixed value template image) First, the identification device 10 performs font estimation when the character string drawing area is unknown, as described below, by using the fixed-value screen component models in the selected sample screen model as candidate screen component models for matching and the undetected fixed-value screen component models as screen component models to be subjected to font estimation (see FIG. 16 (3-2-1)). At this time, those classified as detected fixed-value screen component models are treated as screen component models with a known drawing area for display character strings.

[0210] Next, the identification device 10 generates an estimation result for the font type and size of the estimation result if they are not “unknown,” and if they are “unknown,” generates an image (hereinafter referred to as a “fixed value template image”) in which the character string is drawn using the font type and size of the undetected fixed-value screen component model in the sample screen model (see Figure 16 (3-2-2)).

[0211] The classification device 10 then uses image processing techniques such as image template matching to match the image of the screen to be processed with a fixed-value template image, thereby identifying the drawing area of ​​the display character string of the undetected fixed-value screen component model (see FIG. 16 (3-2-3)). At this time, the classification device 10 excludes the character string drawing area for which the fixed value match flag is set at this point from the scanning target for matching. At this time, for the undetected fixed-value screen component model for which the drawing area has been identified, the classification device 10 corrects the read character string and the character string drawing area by a method described below, using the display character string and its drawing area. The classification device 10 also reclassifies the model into a detected fixed-value screen component model.

[0212] (3-1-3. Correction of read character string and character string drawing area) The identification device 10 identifies, from among the character string drawing areas stored in the drawing area storage unit 14g, those that overlap with the drawing area of ​​the display character string of the undetected fixed-value screen component model for which the drawing area has been identified, and deletes the character string drawing area and its read character string. The identification device 10 also associates the display character string of the undetected fixed-value screen component model for which the drawing area has been identified and its drawing area as a read character string and a character string drawing area, and additionally stores them in the character string drawing area storage unit, and sets a fixed value match flag for the character string drawing area (see FIG. 16(4)).

[0213] In Figure 16(4), the character string drawing area [20,50,150,100] with character string drawing area ID = 3 overlaps with a drawing area identified by OCV technology, so the information for character string drawing area ID = 3 has been deleted. Also, the character string drawing area [20,70,150,100] with character string drawing area ID = 6 overlaps with a drawing area identified by image template matching, etc., so the information for character string drawing area ID = 6 has been deleted. On the other hand, information for character string drawing area IDs = 31, 103, 106, etc. has been added as areas identified by OCV technology, image template matching, etc.

[0214] (3-2. Character String Drawing Area Layout Relationship Determination Process) The identification device 10 identifies two arbitrary character string drawing areas v i and v j The relative positional relationships of the combinations of are examined.

[0215] As a result, the identification device 10 calculates v i and v j s represents the horizontal arrangement of h (i,j) and s h Determine the value of (j,i).

[0216] v i V j To the left of v i The right end of v j If it is to the left of the left edge of s h(i,j)=1,s h Let (j,i) = -1.

[0217] v i V j to the right of v i The left end of v j If it is to the right of the right edge of s h (i,j)=-1,s h Let (j,i) = 1.

[0218] In other cases, s h (i,j)=0,s h Let (j,i) = 0.

[0219] Similarly, v i and v j s represents the vertical arrangement of l (i,j) and s l Determine the value of (j,i).

[0220] v i V j Above, i.e., v i The lower end of v j If it is above the upper end of s l (i,j)=1,s l Let (j,i) = -1.

[0221] v i V j , i.e., v i The upper end of v j If it is below the lower end of s l (i,j)=-1,s l Let (j,i) = 1.

[0222] In other cases, s l (i,j)=0,s l Let (j,i) = 0.

[0223] (3-3. Corresponding process of second screen data) As a process for deriving the correspondence between screen component models and character string drawing areas, we will explain a method for reducing the problem to a constraint satisfaction problem as a method for matching between the screen component models in the sample screen model and the character string drawing areas in the image of the screen to be processed and for calculating the evaluation value.

[0224] The classification device 10 dynamically creates a constraint satisfaction problem, which will be described below, based on the selected sample screen model and the results of identifying character string drawing areas and deriving character string drawing area layout relationships for the image of the screen to be processed, and obtains a solution and an evaluation value using a constraint satisfaction problem solving method. Moreover, the classification device 10 stores the result in the object information-free screen data classification result storage unit only if the evaluation value is better than the results obtained up to that point. Note that, as a constraint satisfaction problem solving method, a method of pruning the search space or a method of obtaining an approximate solution instead of an exact solution may be used.

[0225] (3-3-1. Definition of symbols) The set of common screen component models and their number of elements in the sample screen model are denoted as U and |U|, and the set of character string drawing areas and their number of elements in the image of the screen to be processed are denoted as V and |V|.

[0226] Among the common screen component models in the sample screen model, those with a null string count of 0 (hereafter referred to as "screen component models that require a string drawing area") are called U disp Furthermore, let U be the set of fixed value screen component models. fix Let us assume that.

[0227] Fixed value screen component model u in sample screen model i Change the display string to p i , the character string drawing area v in the image on the target screen i’ Read the string of q i’ Let us assume that.

[0228] Common screen component model in sample screen model i ∈U and the character string drawing area v i’An integer variable x that determines whether to map V and V, taking the value 1 if there is a map and 0 if there is no map. i,i’ It is expressed as:

[0229] Common screen component model in sample screen model i An integer variable y ∈ U that takes the value 1 when it corresponds to at least one character string drawing area in the image of the processing target screen, and takes the value 0 when it does not correspond to at least one character string drawing area in the image of the processing target screen. i It is expressed as:

[0230] (3-3-2. Formulation as a constraint satisfaction problem) The method of associating the common screen component model in the sample screen model with the character string drawing area in the image of the target screen is as follows: 1,1 ,x 1,2 ,…,x 2,1 ,x 2,2 ,…,x |U|,|V| can be expressed as a method of assigning a value of 0 or 1 to each of the elements. However, in order to associate the common screen component models in the sample screen model with the character string drawing areas in the image of the screen to be processed, there are conditions that must be met in order for the sample screen and the screen to be processed, or their screen components, to be equivalent, and these become the constraints in the constraint satisfaction problem.

[0231] Furthermore, when there are multiple methods for associating a common screen component model with a character string drawing area while satisfying the constraint conditions for the same sample screen model and the same processing target screen data, or when there are multiple sample screen models capable of associating a common screen component model with a character string drawing area while satisfying the constraint conditions for the same processing target screen, there is an index for which association method should be selected, and this is the evaluation function in the constraint satisfaction problem. The constraint conditions and evaluation function are each explained below.

[0232] (3-3-2-1. Constraints) In order for the sample screen model and the screen data to be processed to be equivalent, at least all of the following conditions must be met.

[0233] (Constraint condition 1) Even if there are differences in the screen implementation, look and feel, or display magnification, there will be no difference in the layout of the screen components, and the text drawing area is within the range of the drawing area of ​​the screen component. Under this premise, the relative positional relationship of any combination of text drawing areas must conform to the relative positional relationship between the common screen component models that are associated with each of them. Note that "conforms" means that i’ and v J’ However, each i Tou j When two common screen component models u i Tou j If there is a horizontal or vertical relationship between the i’ and v J’ This means that the same relationship holds between

[0234] That is, for any u i ,u j ∈U and v i’ ,v J’ For ∈V, the condition is that both the following formula (1) holds as the horizontal arrangement relationship and the following formula (2) holds as the vertical arrangement relationship.

[0235]

number

[0236]

number

[0237] (Constraint condition 2) If the sample screen model and the screen data to be processed are equivalent, then for common screen component models in which strings are always displayed in the sample screen model, a string drawing area should also exist in the image of the screen to be processed. However, when detecting string drawing areas using OCR technology, a displayed string in one screen component model may end up being split into multiple string drawing areas. Therefore, a screen component model that requires a string drawing area must be associated with at least one string drawing area in the image of the screen to be processed.

[0238] That is, for any u i ∈U disp The condition is that the following formula (3) is satisfied.

[0239]

number

[0240] (Constraint condition 3) Among the screen component models that require a character string drawing area, the fixed-value screen component model in particular specifies the area in which the display character string is drawn in advance when specifying the character string drawing area, and does not require consideration of cases in which the character string drawing area is divided when it is detected by OCR technology. Therefore, the fixed-value screen component model needs to correspond to exactly one character string drawing area in the image of the screen to be processed.

[0241] That is, for any u i ∈U fix For this, instead of constraint 2, the condition is that the following equation (4) is satisfied.

[0242]

number

[0243] (Constraint condition 4) On the screen of an application targeted for operation by an automatic operation agent or the like, display character strings of different screen components are drawn separately from the viewpoint of ensuring human visibility, and therefore display character strings of multiple screen component models are not detected as one character string drawing area. In other words, one character string drawing area is not associated with two or more common screen component models. Also, the screen to be processed may contain screen components other than the common screen component models, or a screen component equivalent to a common screen component model may not have a display character string. Therefore, some character string drawing areas may not be associated with any common screen component model. Therefore, a maximum of one common screen component model is associated with a character string drawing area in the image of the screen to be processed.

[0244] That is, for any v i’ For ∈V, the condition is that the following equation (5) holds.

[0245]

number

[0246] (Constraint condition 5) For the fixed-value screen component model, in addition to constraint 3, the read character string for the character string drawing area in the image of the screen to be processed corresponding to it must match the displayed character string.

[0247] That is, for any u i ∈U fix The condition for this is that the following formula (6) is satisfied.

[0248]

number

[0249] (Constraint condition 6) As is clear from the definition of the variables, x i,1 ,…x i,|V|At least one of y is 1, i is equal to 1.

[0250] That is, for any u i For ∈U, the condition is that the following equation (7) holds.

[0251]

number

[0252] (3-3-2-2. Evaluation function) When all constraints are satisfied, the higher the percentage of character string drawing areas in the image of the target screen that can be matched with common screen component models in the sample screen model, the better the matching method.Similarly, the higher the percentage of common screen component models in the sample screen model that can be matched with character string drawing areas in the image of the target screen, the better the matching method.

[0253] Therefore, for example, Φ in the following equation (8) is set as the evaluation function, where α is a predetermined weighting parameter.

[0254]

number

[0255] (3-4. Display string acquisition process) As a process for identifying a screen data control target without object information and acquiring a display string, a method for acquiring a display string for each of the character string drawing areas (hereinafter referred to as the "selected character string drawing area") in the image of the processing target screen from which the display string is to be acquired will be described in detail.

[0256] If the "type of variation of displayed string" of the common screen component model corresponding to the selected string drawing area (hereinafter referred to as "screen component model from which the selected string is to be acquired") is "fixed value," the identification device 10 regards the displayed string as the acquisition result.

[0257] In other cases, the identification device 10 first performs font estimation when the character string drawing area is unknown, as described below, using fixed-value screen component models in the sample screen model associated with the screen data to be processed as candidate screen component models for matching and the screen component model from which the selected character string is to be obtained as the screen component model from which the font is to be estimated. Note that, at this time, all of the models classified as fixed-value screen component models are treated as screen component models with a known drawing area for the displayed character string.

[0258] Next, the identification device 10 reflects the following in the reading settings in addition to the font estimation result according to the "type of variation in displayed character string" of the screen component model of the selected character string acquisition target.

[0259] When the screen component model of the selected character string acquisition target is a "category value," the character strings in the "display character string set" are reflected in the read setting as character string candidates.

[0260] If the screen component model for the selected character string acquisition target is "arbitrary value," the setting of "number of characters in the displayed character string" or "type of characters in the displayed character string" will be reflected in the read settings.

[0261] Then, the identification device 10 reads the image in the selected character string drawing area using OCR technology based on the reading settings reflected above, and the result is regarded as the obtained display character string.

[0262] (4. Font estimation process for character strings displayed in images) A detailed description will be given of the font estimation process for a character string displayed in an image, which is executed by the classification device 10. The following describes the font estimation process when the character string drawing area is unknown, and then the font estimation process when the character string drawing area is known.

[0263] (4-1. Font estimation process when the character string drawing area is unknown) When the drawing area of ​​the display character string in the screen image is unknown, the identification device 10 selects, from among fixed-value screen component models that satisfy either of the following: (1) the display character string and its drawing area are known, and (2) the drawing area can be identified using a publicly known image processing technique such as image template matching that is the same as that used to calculate the matching success rate and matching evaluation value of the fixed-value screen component model, if the sample screen model contains a fixed-value screen component model with the same font type as the screen component model to be subjected to font estimation, that fixed-value screen component model will be given priority; if not, any fixed-value screen component model will be used as the screen component model for matching.

[0264] Next, the identification device 10 obtains the display character string of the screen component model for comparison and its drawing area in the image of the screen to be processed, and performs font estimation when the character string drawing area is known, thereby obtaining the type and size of the font used to draw the image of the screen to be processed. After that, the identification device 10 infers the type and size of the font used when the display character string of the screen component model for font estimation is drawn in the image of the screen to be processed from the relationship between the type and size of the font in the sample screen model of the screen component model for comparison and the screen component model for font estimation.

[0265] The process flow for estimating the font type and size of a display string in an image on a screen to be processed for a screen component model that is the target of font estimation, i.e., a fixed-value screen component model for generating a fixed-value template image or a screen component model for obtaining a display string, will be described later in [Processing Flow] (5. Font estimation process flow when the string drawing area is unknown) (6. Process flow for limiting model candidates based on whether the display string drawing area can be identified).

[0266] (4-2. Font estimation process when the character string drawing area is known) When the identification device 10 knows the display character string and its drawing area in the screen image, it generates an image in which the display character string is drawn while changing the font type and size within the specified candidates, and matches this with the drawing area in the screen image using image processing techniques such as image template matching, determines the font type and size that provide the best match, and uses these as the estimated result.

[0267] [Process flow] The flow of each process according to this embodiment will be described in detail with reference to Fig. 17 to Fig. 22. Below, the flow of the entire identification process, the flow of the sample screen model derivation process, the flow of the second screen data identification process, the flow of the second screen data acquisition process, the flow of the font estimation process when the character string drawing area is unknown, and the flow of the model candidate limitation process depending on whether the displayed character string drawing area can be specified will be described in that order.

[0268] (1. Overall flow of the identification process) The overall flow of the identification process according to this embodiment will be described in detail with reference to FIG. 17. FIG. 17 is a flowchart showing an example of the overall flow of the process according to the first embodiment. The following describes the object information use mode executed by the first screen data control unit 151 of the identification device 10, the sample screen modeling mode executed by the screen model control unit 152, and the object information non-use mode executed by the first screen data control unit 153 in that order. Note that the following steps S101 to S105 can also be executed in a different order. Also, some of the following steps S101 to S105 may be omitted.

[0269] (1-1. Object information use mode) The object information use mode performs identification processing using the object access method in a non-virtualized environment, while storing the identification result examples in preparation for subsequent use in the sample screen modeling mode, and includes the processing of the following steps S101 and S102.

[0270] First, the first identification unit 151a of the identification device 10 identifies a screen and screen components from the first screen data (step S101). Next, the first acquisition unit 151b of the identification device 10 acquires a first display character string (step S102).

[0271] (1-2. Sample screen modeling mode) The sample screen modeling mode creates a sample screen model that is required when used in an object information non-use mode in an arbitrary environment, and includes the process of step S103 below. The derivation unit 152a of the identification device 10 creates a sample screen model (step S103).

[0272] (1-3. Object information not used mode) The object information non-use mode executes the process of identification without using information of screen component objects by using a sample screen model in any environment, and includes the processes of steps S104 and S105 below.

[0273] First, the second identification unit 153a of the identification device 10 identifies the screen and the screen components from the second screen data (step S104). Next, the second acquisition unit 153b of the identification device 10 acquires the second display character string (step S105).

[0274] The above object information use mode and object information non-use mode may be explicitly specified by the user, or may be automatically switched depending on the environment in which it is implemented. The above sample screen modeling mode may be explicitly specified by the user, or may be temporarily switched to or used in parallel when used in other modes.

[0275] (2. Sample screen model derivation process flow) The flow of the sample screen model derivation process according to this embodiment will be described in detail with reference to Fig. 18. Fig. 18 is a flowchart showing an example of the flow of the sample screen model derivation process according to the first embodiment.

[0276] First, when there is unreflected sample screen data (step S201: Yes), the derivation unit 152a of the classification device 10 selects one of the sample screen data not reflected in the model from the classification information storage unit 14c (step S202), and acquires the classification example screen data and the classification results from the classification example storage unit 14e that are determined to be equivalent to the sample screen data (step S203). On the other hand, when there is no unreflected sample screen data (step S201: No), the derivation unit 152a ends the process.

[0277] Next, if there is a predetermined number or more of equivalent identification example screen data (step S204: Yes), the derivation unit 152a creates and initializes a sample screen model corresponding to the selected sample screen data (step S206).On the other hand, if there is not a predetermined number or more of equivalent identification example screen data (step S204: No), the derivation unit 152a processes the selected sample screen data as having been reflected (step S205) and proceeds to the process of step S201.

[0278] Next, if there is unreflected identification example screen data (step S207: Yes), the derivation unit 152a selects one of the identification example screen data and the identification result that is not reflected in the model (step S208), updates the sample screen model using the selected identification example screen data and the identification result (step S209), processes the selected identification example screen data as reflected (step S210), and proceeds to the processing of step S207.

[0279] On the other hand, if there is no unreflected recognition example screen data (step S207: No), the derivation unit 152a derives the regularity of the displayed character string for the sample screen model (step S211), derives the font in the sample screen image (step S212), calculates the matching success rate and matching evaluation value of the fixed value screen component model (step S213), processes the selected sample screen data as reflected (step S205), and proceeds to the processing of step S201.

[0280] It should be noted that the above steps S201 to S213 may be executed in a different order and at different timings. Also, among the above steps S201 to S213, some of the processes may be omitted.

[0281] Furthermore, the derivation unit 152a starts processing in the following cases, but is not limited to the following cases: for example, a user explicitly instructs the start of execution; when classification is performed on screen data to be processed that includes object information and a certain number or more of classification results are newly added to the classification example storage unit 14e; when an attempt is made to determine the equivalence of screen data to be processed that does not include object information with certain sample screen data and a sample screen model corresponding to the sample screen data does not exist in the screen model storage unit 14f; and when, after the creation of the sample screen model corresponding to the sample screen data, a certain number or more of classification results that have been determined to be equivalent to the sample screen data have been newly added to the classification example storage unit 14e.

[0282] (3. Flow of the second screen data identification process) The flow of the identification process of the second screen data according to the present embodiment will be described in detail with reference to Fig. 19. Fig. 19 is a flowchart showing an example of the flow of the identification process of the second screen data according to the first embodiment.

[0283] First, when an uncompared sample screen model exists (step S301: Yes), the second identification unit 153a of the identification device 10 selects one uncompared sample screen model from the screen model storage unit 14f (step S302). On the other hand, when an uncompared sample screen model does not exist (step S301: No), the second identification unit 153a ends the process.

[0284] Next, the second identification unit 153a identifies the character string drawing areas in the image of the screen to be processed and, within that, the areas in which the display character string of the fixed value screen component model in the selected sample screen model is drawn (step S303), derives the relative positional relationship between the character string drawing areas in the image of the screen to be processed (step S304), associates the screen component model in the selected sample screen model with the character string drawing areas in the image of the screen to be processed (step S305), processes the selected sample screen model as having been compared (step S306), and proceeds to the processing of step S301.

[0285] It should be noted that the above steps S301 to S306 may be executed in a different order and at different timings.Furthermore, among the above steps S301 to S306, some of the processes may be omitted.

[0286] (4. Flow of the second screen data acquisition process) The flow of the process of acquiring second screen data according to this embodiment will be described in detail with reference to Fig. 20. Fig. 20 is a flowchart showing an example of the flow of the process of acquiring a second character string according to the first embodiment.

[0287] First, when the classification result is stored in the second classification result storage unit 14i (step S401: Yes), the second acquisition unit 153b of the classification device 10 acquires the sample screen model determined to be equivalent from the screen model storage unit 14e (step S402). On the other hand, when the classification result is not stored in the second classification result storage unit 14i (step S401: No), the second acquisition unit 153b ends the process.

[0288] Next, if there is an unprocessed control screen component model in the sample screen model (step S403: Yes), the second acquisition unit 153b selects one unprocessed control screen component model in the sample screen model (step S404), and identifies a character string drawing area in the image of the screen to be processed that is associated with the control screen component model from the second identification result and the character string drawing area in the drawing area storage unit 14g (step S405).On the other hand, if there is no unprocessed control screen component model in the sample screen model (step S403: No), the process ends.

[0289] Next, if the second acquisition unit 153b can identify the character string drawing area (step S406: Yes), it stores the character string drawing area identified in the process of step S405 in the process result storage unit (step S407), and proceeds to the process of step S408. On the other hand, if the second acquisition unit 153b cannot identify the character string drawing area (step S406: No), it determines that the character string drawing area was not identified as the character string drawing area because the screen component is not displayed on the screen to be processed or the displayed character string is a null character string, processes the character string drawing area as "unknown" and the displayed character string as a null character string, stores them in the process result storage unit 14b (step S410), processes the selected model as processed (step S411), and proceeds to the process of step S403.

[0290] Then, if the selected control target screen component model is the display string acquisition target (step S408: Yes), the second acquisition unit 153b acquires the display string from the string drawing area in the image of the screen to be processed that is associated with the selected control target screen component model, stores the display string in the processing result memory unit 14b (step S409), processes the selected model as having been processed (step S411), and proceeds to processing of step S403.

[0291] It should be noted that the above steps S401 to S411 may be executed in a different order and at different timings. Also, among the above steps S401 to S411, some of the processes may be omitted.

[0292] (5. Flow of font estimation process when character string drawing area is unknown) The flow of font estimation processing when the character string drawing area according to this embodiment is unknown will be described in detail with reference to Fig. 21. Fig. 21 is a flowchart showing an example of the flow of processing for estimating the font type and size of a displayed character string when the character string drawing area according to the first embodiment is unknown.

[0293] First, the identification device 10 processes the font type and size of the font estimation result as "undetermined" (step S501), and eliminates from the sample screen model the candidate screen component models for comparison whose font type differs from that of the screen component model to be subjected to font estimation (step S502).

[0294] Next, when there is one or more screen component model candidates for comparison (step S503: Yes), the identification device 10 identifies the drawing area of ​​the display character string in the image of the screen to be processed for the screen component model candidates for comparison, and excludes those that could not be identified from the candidates (step S504). Note that the flow of the process of limiting the screen component model candidates for comparison will be described later with reference to FIG. 22. On the other hand, when there is one or more screen component model candidates for comparison (step S503: No), the identification device 10 proceeds to the process of step S511.

[0295] Next, after the processing of step S504, if there is one or more candidate screen component model for comparison (step S505: Yes), the identification device 10 selects one of the candidates screen component model for comparison as a screen component model for comparison, performs font estimation when the character string drawing area candidate is known, and obtains the font type and size of the screen component model for comparison (step S506).On the other hand, after the processing of step S504, if there is one or more candidate screen component model for comparison (step S505: No), the identification device 10 proceeds to the processing of step S511.

[0296] Furthermore, if the font type is not "unknown" (step S507: Yes), the identification device 10 processes the font type of the comparison screen component model as the font type of the font estimation result (step S508), calculates the ratio of the font sizes in the comparison screen component model and the screen component model to be the target of font estimation in the sample screen model (step S509), processes the font size of the font estimation result as a size reflecting the calculated ratio in the size of the comparison screen component model in the image of the target screen (step S510), and ends the process. On the other hand, if the font type is "unknown" (step S507: No), the identification device 10 proceeds to the process of step S509.

[0297] Furthermore, after the processing of step S503 or S505, if the font type is not "unknown" (step S511: Yes), the identification device 10 performs processing to return the comparison screen component model candidates to their initial state before exclusion (step S512), processes the font type of the font estimation result as "unknown" (step S513), and excludes from the comparison screen component model candidates those whose font type is the same as that of the screen component model (step S514), and proceeds to the processing of step S503. On the other hand, after the processing of step S503 or S505, if the font type is "unknown" (step S511: No), the identification device 10 processes the font size of the font estimation result as "unknown" (step S515), and ends the processing.

[0298] It should be noted that the above steps S501 to S515 may be executed in a different order and at different timings. Also, among the above steps S501 to S515, some of the processes may be omitted.

[0299] (6. Flow of model candidate limitation process based on whether the display character string drawing area can be identified) The flow of font estimation processing when the character string drawing area according to this embodiment is unknown will be described in detail with reference to Fig. 22. Fig. 22 is a flowchart showing an example of the flow of processing for limiting component model candidates for matching based on whether the character string drawing area can be specified according to the first embodiment.

[0300] First, the identification device 10 extracts candidates for screen component models to be used for matching or those whose drawing areas of their display strings are known from the image of the screen to be processed (step S601). If one or more are extracted (step S602: Yes), the identification device 10 excludes those that were not extracted from the candidates for screen component models to be used for matching (step S603), and terminates the process.

[0301] On the other hand, if the identification device 10 does not extract one or more candidate screen component models for matching or those with known drawing areas for their displayed character strings in the image of the screen to be processed (step S602: No), it excludes from the candidate screen component models for matching those whose matching success rate or matching evaluation value is less than a threshold value (step S604).

[0302] Next, if there is one or more candidate screen component model for matching (step S605: Yes), the identification device 10 generates an image (hereinafter referred to as a "candidate matching template image") in which the display string of the candidate screen component model for matching is drawn using the font type and size of the sample screen model (step S606), and uses image processing technology such as image template matching, which is the same as that used when calculating the "matching success rate" and "matching evaluation value" of the fixed-value screen component model in deriving the sample screen model, to match the image of the screen to be processed with the candidate matching template image, and identify the drawing area of ​​the display string of the candidate screen component model for matching (step S607), and excludes those for which the drawing area of ​​the display string cannot be identified from the candidate screen component model for matching (step S608), and ends the processing.

[0303] On the other hand, if there is not one or more candidate screen component model for matching (step S605: No), the classification device 10 ends the process.

[0304] It should be noted that the above steps S601 to S608 may be executed in a different order and at different timings.Furthermore, among the above steps S601 to S608, some of the processes may be omitted.

[0305] In the above explanation, in order to make it easier to understand the configuration and processing contents of the device of the present invention in accordance with the actual usage scene, a "sample screen model" and a "sample screen component model" are created as intermediate data and used for comparison prior to identifying the image of the screen to be processed. However, essentially, the image of the screen to be processed is compared with the information of the screen component objects of the sample screen and an equivalent identified example screen, and it is not limited to whether intermediate data is created as a sample screen model or the existence of the derivation unit 152a that does so.

[0306] [Advantages of the first embodiment] First, in the identification process according to the present embodiment described above, first screen data 30 including an image of an application screen and information on screen component objects which are objects of elements that make up the screen is identified, a first identification result associated with sample screen data which is the screen data to be referenced is output, and second screen data 40 including an image of the screen but not information on the screen component objects is identified, and a second identification result associated with the sample screen data is output. Therefore, in this process, in a virtualized environment, it is possible to accurately identify the screen and screen components of an application and obtain display strings without the trouble of setting up the operation of identification or the reading setting of the display strings.

[0307] Secondly, in the identification process according to the present embodiment described above, prior to identifying the second screen data, a derivation unit identifies common objects that are included in common in a plurality of first screen data from among the screen component objects of the sample screen data based on the sample screen data and the first identification result, obtains the relative positional relationship of the common objects for each drawing area, and derives a sample screen model including them. Therefore, in this process, the screen and screen components of an application can be accurately identified in a virtualized environment, and can be used for an automated operation agent or a work analysis tool.

[0308] Thirdly, in the identification process according to the present embodiment described above, the first screen data 30 including the first character string and the drawing area of ​​the screen component object having the first character string as an attribute is identified by determining the equivalence using information on the screen component object, a first identification result in which the screen component object is associated with each sample screen data determined to have equivalence is output, a sample screen model including the relative positional relationship for each drawing area of ​​the first character string is derived using an identification example including a plurality of first identification results, an optical character recognition process is used for the second screen data to identify the second character string and the drawing area of ​​the second character string from the image of the screen, a relative positional relationship for each drawing area of ​​the second character string is determined, the second image data 40 is identified based on the drawing area of ​​the second character string and the relative positional relationship for each drawing area of ​​the second character string, and a second identification result in which the screen component object is associated with each sample screen model is output. Therefore, in this process, the screen and screen components of the application can be accurately identified in a non-virtualized environment and a virtualized environment, and can be used for an automatic operation agent or a work analysis tool.

[0309] Fourth, in the identification process according to the present embodiment described above, among the screen component objects of the sample screen data included in the identification example, fixed value objects that are included in multiple first screen data and have the same character string are identified, and a sample screen model is derived. Therefore, in this process, the screens and screen components of the application can be accurately and effectively identified in both non-virtualized and virtualized environments, and used in an automated operation agent or a work analysis tool.

[0310] Fifth, in the identification process according to the present embodiment described above, a sample screen model is used to derive a sample screen model further including at least one of the type of variation, number of characters, type of character, type of font, and size of the first character string, and at least one of the type of variation, type of character, type of font, and size of the character string is used to identify the second character string and the drawing area of ​​the second character string from the screen image using optical character recognition processing on the second screen data. Therefore, in this process, the screen and screen components of the application can be identified accurately and efficiently in a non-virtualized environment and a virtualized environment, and can be used for an automated operation agent or a work analysis tool.

[0311] Sixth, in the identification process according to the present embodiment described above, a second character string included in the second screen data is obtained based on the second identification result and at least one of the type of variation, the number of characters, the type of character, the type of font, and the size of the character string included in the sample screen model. Therefore, in this process, the screen and screen components of the application can be identified accurately and efficiently in a non-virtualized environment and a virtualized environment, and can be used more effectively in an automated operation agent or a work analysis tool.

[0312] Seventh, in the identification process according to the present embodiment described above, the second image data 40 is identified by determining equivalence using constraint conditions related to the second character string and a predetermined evaluation function, and a second identification result associated with a screen component object for each sample screen model is output. Therefore, in this process, the screen and screen components of an application can be accurately identified by effectively using the drawing area of ​​the character string in both non-virtualized and virtualized environments, and can be used in an automated operation agent or a work analysis tool.

[0313] [System configuration, etc.] Each component of each device shown in the figures according to the above embodiment is a functional concept, and does not necessarily have to be physically configured as shown in the figures. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figures, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0314] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0315] 〔program〕 It is also possible to create a program in which the processing executed by the identification device 10 described in the above embodiment is written in a language executable by a computer. In this case, the same effects as those of the above embodiment can be obtained by having the computer execute the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to realize the same processing as those of the above embodiment.

[0316] Fig. 23 is a diagram showing a computer that executes a program. As shown in Fig. 23, a computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and these components are connected by a bus 1080.

[0317] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as exemplified in FIG. 23. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090, as exemplified in FIG. 23. The disk drive interface 1040 is connected to a disk drive 1100, as exemplified in FIG. 23. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, as exemplified in FIG. 23. The video adapter 1060 is connected to a display 1130, as exemplified in FIG. 23.

[0318] 23, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored, for example, in the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

[0319] Furthermore, the various data described in the above embodiment are stored as program data, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes various processing procedures.

[0320] Note that the program module 1093 and program data 1094 relating to the program are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 relating to the program may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)) and read by the CPU 1020 via the network interface 1070.

[0321] The above-described embodiments and their modifications are included in the technology disclosed in this application, as well as in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0322] 10 Identification device 11 Input section 12 Output section 13, 21 Communications Department 14 Storage section 14a Screen data storage section 14b Processing result storage unit 14c Identification information storage unit 14d First identification result storage unit 14e Identification case storage unit 14f Screen model storage section 14g drawing area storage section 14h Layout relationship memory section 14i Second classification result storage unit 15, 22 Control section 151 First screen data control section 151a 1st identification part 151b 1st acquisition part 152 Screen model control section 152a Derivation part 153 Second screen data control section 153a Second identification part 153b 2nd acquisition part 20 Autonomous Agent Device 30 Screen data with object information (first screen data) 40 Screen data without object information (second screen data) 100 Identification System

Claims

1. a first identification unit that identifies first screen data including an image of an application screen and information on screen component objects that are objects of elements that configure the screen, and outputs a first identification result that associates the first screen data with sample screen data that is reference screen data; a second identification unit that identifies second screen data including an image of the screen and not including information related to the screen component object based on the sample screen data and the first identification result, and outputs a second identification result associated with the sample screen data; An identification device comprising:

2. a derivation unit which, prior to identifying the second screen data, identifies common objects included in common to a plurality of the first screen data from the screen component objects of the sample screen data based on the sample screen data and the first identification result, obtains relative layout relationships of the common objects for each drawing area, and derives a sample screen model including them; The identification device according to claim 1 , further comprising:

3. the first identification unit identifies the first screen data including a first character string and a drawing area of ​​a screen component object having the first character string as an attribute by determining equivalence using information related to the screen component object, and outputs the first identification result in which the screen component object is associated with each of the sample screen data determined to have the equivalence; the second identification unit identifies a second character string and a drawing area of ​​the second character string from an image of the screen using an optical character recognition process on the second screen data, determines a relative positional relationship for each drawing area of ​​the second character string, identifies the second screen data based on the drawing area of ​​the second character string and the relative positional relationship for each drawing area of ​​the second character string, and outputs the second identification result associated with the screen component object for each sample screen model.

3. The identification device according to claim 2.

4. the derivation unit identifies a fixed value object that is included in common in a plurality of the first screen data and has an identical character string among the screen component objects of the sample screen data included in the first identification result, and derives the sample screen model.

4. The identification device according to claim 3.

5. The derivation unit derives the sample screen model using the sample screen model, the sample screen model further including at least one of a type of variation, a number of characters, a type of character, a type of font, and a size of the first character string; the second identification unit uses at least one of a type of variation, a type of character, a type of font, and a size of the first character string to identify the second character string and a drawing area of ​​the second character string from an image of the screen by using an optical character recognition process on the second screen data; 4. The identification device according to claim 3.

6. a second acquisition unit that acquires a second character string included in the second screen data based on the second identification result and at least one of a type of variation, a number of characters, a type of character, a type of font, and a size of the first character string included in a sample screen model; The identification device according to claim 5, further comprising:

7. the second identification unit identifies the second screen data by determining equivalence using a constraint condition related to the second character string and a predetermined evaluation function, and outputs the second identification result.

6. The identification device according to claim 3, wherein the first and second electrodes are arranged in a first direction.

8. 1. A method of identification performed by an identification device, comprising: a step of identifying first screen data including an image of a screen of an application and information on screen component objects which are objects of elements constituting the screen, and outputting a first identification result in which the first identification result is associated with sample screen data which is screen data to be referenced; identifying second screen data including an image of the screen and not including information regarding the screen component object, and outputting a second identification result corresponding to the sample screen data; 23. A method for identifying a subject, comprising:

9. a step of identifying first screen data including an image of a screen of an application and information on screen component objects which are objects of elements constituting the screen, and outputting a first identification result in which the first identification result is associated with sample screen data which is screen data to be referenced; identifying second screen data including an image of the screen and not including information about the screen component object, and outputting a second identification result corresponding to the sample screen data; An identification program characterized by causing a computer to execute the above steps.

Citation Information

Patent Citations

  • Method and device for specifying component to be automatically operated

    JP2013033377A

  • Automatic operation device and method by image recognition, and program

    JP2015005245A

  • Information processing apparatus and information processing program

    JP2019197504A