Object search device and object search method

Abstract

An object search device according to an embodiment of the present invention comprises a storage unit and a search unit. The storage unit stores, for a plurality of user interface (UI) elements for constituting a display screen, a table in which are associated an identifier of each of the plurality of UI elements and reference information regarding each of a plurality of objects of a model composed of the plurality of objects, which respectively correspond to the plurality of UI elements. The search unit acquires an identifier for a target UI element from among the plurality of UI elements, detects reference information corresponding to the acquired identifier from the table, and uses the detected reference information to search in the model for a target object corresponding to the target UI element.

Description

Object search device and object search method

[0001] The present invention relates to an object search device and an object search method.

[0002] Conventionally, business systems that support users' business operations have been provided. A user terminal displays a display screen related to the provided business system. The user uses various functions related to the business system by operating UI (User Interface) elements on the display screen.

[0003] Recently, a technology has been proposed for automatically operating UI elements or monitoring the operation status of UI elements. According to this technology, a user terminal achieves the above-mentioned automatic operation or monitoring by accessing UI elements using an API (Application Programming Interface) installed in its own operating system (OS). An API or library having this function is called an accessibility API (see, for example, Non-Patent Document 1).

[0004] Accessibility APIs often manage multiple UI elements as a tree-structured model consisting of multiple objects. The user terminal searches the model in a predetermined order for an object (target object) that corresponds to a UI element (target UI element) that is the target of automatic operation or monitoring.

[0005] However, the above search method cannot quickly search for a target object from a model, and as a result, the user terminal cannot quickly detect the operation state of a UI element and cannot respond immediately to a user operation on the UI element.

[0006] “Using Caching in UI Automation”, [online], May 10, 2023, Microsoft Learn, [Retrieved February 28, 2024], Internet <URL: https: / / learn.microsoft.com / ja-jp / dotnet / framework / ui-automation / use-caching-in-ui-automation>

[0007] An object of the present invention is to quickly search for a target object from a model.

[0008] An object search device according to an embodiment includes a storage unit and a search unit. The storage unit stores a table that associates identifiers of multiple user interface (UI) elements constituting a display screen with reference information for each of multiple objects in a model including multiple objects corresponding to the multiple UI elements. The search unit acquires the identifier of a target UI element among the multiple UI elements, detects reference information corresponding to the acquired identifier from the table, and searches the model for a target object corresponding to the target UI element using the detected reference information.

[0009] According to the present invention, a target object can be quickly searched for in a model.

[0010] Fig. 1 is a functional configuration diagram of an object search device according to an embodiment. Fig. 2 is a diagram showing a display screen and an object model according to an embodiment. Fig. 3 is a diagram showing a correspondence table according to an embodiment. Fig. 4 is an operation flow diagram of an object search device according to an embodiment. Fig. 5 is a diagram showing state transitions of the correspondence table according to an embodiment. Fig. 6 is a hardware configuration diagram of an object search device according to an embodiment.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Parts with the same reference numerals are considered to operate in the same manner, and redundant description will be omitted as appropriate.

[0012] FIG. 1 is a functional configuration diagram of an object search device 1 according to an embodiment. The object search device 1 is a device that searches for an object. The object search device 1 is communicably connected to an automation / monitoring tool 10. The object search device 1 includes a search unit 11, a determination unit 12, and a storage unit 13. The search unit 11 is communicably connected to the automation / monitoring tool 10, the determination unit 12, and the storage unit 13. The object search device 1 may further include the automation / monitoring tool 10. The object search device 1 may be installed in a user terminal used by a user.

[0013] The automation / monitoring tool 10 is a tool that performs automatic operation or monitoring of an object (target object) corresponding to a UI element (target UI element) that is the target of automatic operation or monitoring. The automation / monitoring tool 10 holds an identifier for the target UI element. The automation / monitoring tool 10 may be software that performs automatic operation or monitoring of a UI element displayed on a display screen, such as robotic process automation (RPA) and UI automation. The target UI element may be at least one UI element out of multiple UI elements that constitute the display screen.

[0014] The search unit 11 is a means for searching various data. The search unit 11 may acquire an identifier of a target UI element from the automation / monitoring tool 10. The search unit 11 may detect reference information corresponding to the acquired identifier from a table. The search unit 11 may use the detected reference information to search for a target object corresponding to the target UI element from a model. The search unit 11 may perform processing according to a determination result of the determination unit 12. When the determination unit 12 determines that the target UI element is not displayed on the display screen, the search unit 11 may delete the identifier of the target UI element and the reference information of the target object from the table.

[0015] The determination unit 12 is a means for making various determinations. The determination unit 12 may determine whether a property value of the target object is valid. Specifically, the determination unit 12 may determine whether the property value is obtainable or not, or may determine whether the property value is an invalid value or not. The determination unit 12 may determine whether the target UI element is displayed on the display screen based on the property value. If the determination unit 12 determines that the target UI element is not displayed on the display screen, the determination unit 12 may transmit this determination result to the search unit 11.

[0016] The storage unit 13 is a means for storing various types of data. First, the storage unit 13 may store a model that manages multiple UI elements for configuring a display screen. The model may exist outside the object search device 1. In this case, the object search device 1 may directly reference information from an external accessibility API. The model may have a tree structure made up of multiple objects corresponding to the multiple UI elements (see FIG. 2). Second, the storage unit 13 may store a table that associates identifiers of the multiple UI elements with reference information (or object references) for each of the multiple objects in the model (see FIG. 3).

[0017] 2A and 2B are diagrams showing a display screen 2 and an object model 300 according to an embodiment. Fig. 2A shows the screen configuration of the display screen 2. Fig. 2B shows the data structure of the object model 300. A plurality of UI elements on the display screen 2 correspond to a plurality of objects in the object model 300. The object model 300 is managed by an accessibility API.

[0018] 2A, the display screen 2 is a screen displayed on a user terminal. The display screen 2 displays windows for a first application and a second application related to a business system. The display screen 2 includes, as UI elements, a first window 21 corresponding to the first application and a second window 22 corresponding to the second application.

[0019] The first window 21 includes, as UI elements, a first title bar 211 and a first content 212. The first content 212 includes, as UI elements, a first text box 212A and a first button 212B. Similarly, the second window 22 includes, as UI elements, a second title bar 221 and a second content 222. The second content 222 includes, as UI elements, a second text box 222A and a second button 222B. In this way, the multiple UI elements on the display screen 2 are configured hierarchically such that one UI element in a higher hierarchical level includes multiple UI elements in a lower hierarchical level.

[0020] The user terminal responds to a user operation by inputting a value into the first text box 212A (or the second text box 222A). When the user selects the first button 212B (or the second button 222B), the user terminal registers the input value. In this way, the user can use various functions related to the business system by operating various UI elements on the display screen 2.

[0021] 2B , object model 300 is a tree-structured model that manages a plurality of objects corresponding to a plurality of UI elements on display screen 2. Object model 300 includes objects 3 that correspond to display screen 2. Objects 3 include object 31 that corresponds to first window 21 and object 32 that corresponds to second window 22.

[0022] The object 31 includes an object 311 corresponding to the first title bar 211 and an object 312 corresponding to the first content 212. The object 312 includes an object 312A corresponding to the first text box 212A and an object 312B corresponding to the first button 212B. Similarly, the object 32 includes an object 321 corresponding to the second title bar 221 and an object 322 corresponding to the second content 222. The object 322 includes an object 322A corresponding to the second text box 222A and an object 322B corresponding to the second button 222B. In this way, the object model 300 is hierarchically structured such that one object in a higher hierarchy includes multiple objects in a lower hierarchy.

[0023] 3 is a diagram showing a correspondence table 400 according to an embodiment. The correspondence table 400 is a table that associates identifiers of multiple UI elements on the display screen 2 with reference information of multiple objects in the object model 300. The correspondence table 400 registers a correspondence between each of the multiple identifiers and the reference information of each of the multiple objects. In its initial state, the correspondence table 400 does not register this correspondence. The correspondence table 400 is stored in the storage unit 13.

[0024] 4 is an operation flow diagram of the object search device 1 according to the embodiment. The object search device 1 and the automation / monitoring tool 10 repeatedly execute the processes of steps S1 to S7C.

[0025] (Step S1) First, the automation / monitoring tool 10 acquires the identifier of the target UI element. Specifically, the automation / monitoring tool 10 acquires the identifier of the target UI element at the timing when it is desired to monitor changes in the operation state of the target UI element on the display screen 2. For example, the automation / monitoring tool 10 acquires the identifier of the target UI element at the timing when the target UI element is operated by a user. The automation / monitoring tool 10 may acquire the identifier of the target UI element by periodic polling. The automation / monitoring tool 10 transmits (or inputs) the acquired identifier to the searching unit 11.

[0026] The identifier of the target UI element is information that uniquely identifies the target UI element. First, if a name that uniquely identifies the target UI element exists in a model (e.g., object model 300) managed by the accessibility API, the name may be used as the identifier. Second, if the name does not exist in the model and the model has a tree structure (i.e., object model 300), a set of multiple numbers that represent the parent-child and sibling relationships in the tree structure may be used as the identifier. Third, if the display screen 2 is a UI screen on a web browser, Xpath (XML Path Language) may be used as the identifier. The model is not limited to a tree structure and may have any structure that can reference objects.

[0027] According to the second case described above, the identifier may be a set of multiple numbers assigned to each object in the object model 300 (see FIG. 2B ). For example, for two objects 31 and 32 branching from object 3, object 31 is assigned the number "1" and object 32 is assigned the number "2." For two objects 311 and 312 branching from object 31, object 311 is assigned the number "1" and object 312 is assigned the number "2." For two objects 312A and 312B branching from object 312, object 312A is assigned the number "1" and object 312B is assigned the number "2." In this case, the set (1, 2, 1) identifies object 312A corresponding to the first text box 212A. Like this set, the identifier may be expressed in the form of a list of numbers indicating branches in each hierarchy, from the highest to the lowest hierarchy in the object model 300.

[0028] (Step S2) Next, the search unit 11 determines whether or not an identifier is registered in the table. Specifically, the search unit 11 determines whether or not the identifier of the target UI element input in step S1 is registered in the correspondence table 400. If the identifier is registered (step S2—YES), the process proceeds to step S3A. If the identifier is not registered (step S2—NO), the process proceeds to step S4B.

[0029] (Step S3A) Next, the search unit 11 detects the reference information of the target object from the table. Specifically, for the identifier of the target UI element determined to be registered in step S2, the search unit 11 detects the reference information of the target object corresponding to the identifier from the correspondence table 400.

[0030] (Step S4A) Subsequently, the search unit 11 searches for the target object using the reference information of the target object. Specifically, the search unit 11 searches for the target object from the object model 300 using the reference information of the target object detected in step S3A.

[0031] (Step S5A) Subsequently, the search unit 11 determines whether or not the target object has been successfully accessed. Specifically, the search unit 11 determines whether or not the target object has been accessed as a result of searching the object model 300 in step S4A. If the access has been successful (step S5A—YES), the process proceeds to step S6A. If the access has not been successful (step S5A—NO), the process proceeds to step S7B.

[0032] (Step S6A) In this case, for the target object that was successfully accessed in step S5A, the determination unit 12 determines whether or not a target UI element corresponding to the target object is displayed on the display screen 2. If the target UI element is displayed (step S6A—YES), the process proceeds to step S7A. If the target UI element is not displayed (step S6A—NO), the process proceeds to step S7B.

[0033] The determination unit 12 may learn in advance a determination condition (or a determination rule) for determining whether a target UI element is displayed (or not displayed). For example, the determination unit 12 may learn in advance a property value of the target object when the target UI element is displayed (or not displayed). The determination unit 12 may learn that the target UI element is not displayed when the "UI element size," which is an example of a property value, is "0." The determination unit 12 can make the above determination based on the determination condition learned in advance.

[0034] In step S7C (described later), the target UI element was displayed on display screen 2, but as a result of a dynamic change in display screen 2, the target UI element may not be displayed (or may be hidden) on display screen 2 even though the target object exists in object model 300. The determination in step S6A assumes a case in which, in automatic operation or monitoring, it is desired to consider that the target object does not exist in object model 300. The determination in step S6A makes it possible to exclude target UI elements that are not displayed on display screen 2 from being targets of automatic operation or monitoring, and to reduce the amount of data managed by correspondence table 400 (see step S7B).

[0035] (Step S7A) In this case, the search unit 11 returns reference information for the target object. Specifically, the search unit 11 returns reference information for the target object detected in step S3A to the automation / monitoring tool 10. The automation / monitoring tool 10 uses the returned reference information to automatically operate or monitor the target UI element. Specifically, the automation / monitoring tool 10 automatically operates or monitors the target UI element determined to be displayed in step S6A. The automation / monitoring tool 10 may automatically click the target UI element or automatically input text into the target UI element. Alternatively, the automation / monitoring tool 10 may acquire a property value of the target object and compare the acquired property value with a previous property value. In this way, the automation / monitoring tool 10 can monitor changes in the operation state of the target UI element. The automation / monitoring tool 10 may monitor whether the target UI element has been clicked (or focused) or whether text has been input into the target UI element. After step S7A, the object search device 1 ends the series of processes.

[0036] (Step S7B) In this case, the search unit 11 deletes the identifier of the target UI element and the reference information of the target object from the table. Specifically, the search unit 11 deletes the identifier of the target UI element input in step S1 and the reference information of the target object detected in step S3A from the correspondence table 400. After step S7B, the process proceeds to step S4B.

[0037] (Step S4B) Next, the search unit 11 searches for the target object using the identifier of the target UI element. Specifically, the search unit 11 searches for the target object from the object model 300 using the identifier provided as input in step S1.

[0038] (Step S5B) Subsequently, the search unit 11 determines whether or not the target object has been successfully accessed. Specifically, the search unit 11 determines whether or not the target object has been accessed as a result of searching the object model 300 in step S4B. If the access has been successful (step S5B-YES), the process proceeds to step S6B. If the access has not been successful (step S5B-NO), the process proceeds to step S6C.

[0039] (Step S6B) In this case, the search unit 11 registers the identifier of the target UI element and the reference information of the target object in a table. Specifically, the search unit 11 registers the identifier of the target UI element input in step S1 and the reference information of the target object successfully accessed in step S5B in the correspondence table 400. That is, the search unit 11 registers in the correspondence table 400 the correspondence between the identifier of the target UI element and the reference information of the target object in the object model 300.

[0040] (Step S7C) Subsequently, the search unit 11 returns the reference information of the target object. Specifically, the search unit 11 returns the reference information of the target object that was successfully accessed in step S5B to the automation / monitoring tool 10. The automation / monitoring tool 10 uses the returned reference information to automatically operate or monitor the target UI element. Specifically, for the target object that was successfully accessed in step S5B, the automation / monitoring tool 10 automatically operates or monitors the target UI element corresponding to the target object. Step S7C is the same as step S7A. After step S7C, the object search device 1 ends the series of processes.

[0041] (Step S6C) In this case, the search unit 11 returns an error. Specifically, the search unit 11 returns an error indicating that access to the target object failed in step S5B to the automation / monitoring tool 10. After step S6C, the object search device 1 ends the series of processes.

[0042] 5A and 5B are diagrams showing state transitions of the correspondence table 400 according to the embodiment. Fig. 5A shows a correspondence table 400A for step S2 during the first operation. Fig. 5B shows a correspondence table 400B for step S6B during the first operation. Fig. 5C shows a correspondence table 400C for step S2 during the second or subsequent operation. Fig. 5D shows a correspondence table 400D for step S7B during the second or subsequent operation.

[0043] Hereinafter, a series of processes in which the object search device 1 and the automation / monitoring tool 10 monitor the first text box 212A on the display screen 2 as a target UI element will be described with reference to FIGS.

[0044] (First Operation) First, the automation / monitoring tool 10 acquires "First Text Box" as the identifier of the target UI element (Step S1). The object search device 1 determines whether the identifier "First Text Box" is registered in the correspondence table 400 (Step S2). Since the data in the correspondence table 400A is empty (or "none"), the object search device 1 determines that the identifier "First Text Box" is not registered in the correspondence table 400A (Step S2 - NO).

[0045] Next, the object search device 1 searches for the target object from the object model 300 using the identifier "first text box" of the target UI element (step S4B). If the object search device 1 successfully accesses the target object (step S5B—YES), it registers the identifier "first text box" of the target UI element and the reference information "OBJ-A" of the target object in the correspondence table 400A (step S6B). This registration causes the correspondence table 400A to transition to the correspondence table 400B. The record in the first row of the correspondence table 400B registers the correspondence between the identifier "first text box" and the reference information "OBJ-A." Finally, the object search device 1 returns the reference information of the target object (step S7C), and the series of processes ends.

[0046] The object search device 1 and the automation / monitoring tool 10 repeat the above series of processes, treating each of the other UI elements on the display screen 2 as the target UI element. As a result, the correspondence table 400B transitions to the correspondence table 400C. In addition to the record in the first row described above, the correspondence table 400C registers the correspondence between the identifier "Second Button" and the reference information "OBJ-B" in the record in the second row. The correspondence table 400C also registers similar correspondences for the records in the third row and beyond. The reference information "OBJ-A" and "OBJ-B" are notations for convenience. The reference information may also be an address in the memory space of the OS where the target object exists. For example, the address is expressed as a 32-bit integer such as "0x00000000."

[0047] (Operations from the second time onward) The automation / monitoring tool 10 again acquires "first text box" as the identifier of the target UI element (step S1). The object search device 1 determines whether the identifier "first text box" is registered in the correspondence table 400 (step S2). The object search device 1 determines that the identifier "first text box" is registered in the record in the first row of the correspondence table 400C (step S2 - YES).

[0048] Next, the object search device 1 detects the reference information "OBJ-A" of the target object corresponding to the identifier "first text box" from the correspondence table 400C (step S3A).The object search device 1 uses the reference information "OBJ-A" to search for the target object (i.e., object 312A) from the object model 300 (step S4A).

[0049] First, if the object search device 1 successfully accesses the object 312A (step S5A-YES) and the target UI element (i.e., the first text box 212A) is displayed on the display screen 2 (step S6A-YES), it returns the reference information of the target object (step S7A) and ends the series of processes.

[0050] Secondly, if the object search device 1 has successfully accessed the object 312A (step S5A—YES) and the first text box 212A is not displayed on the display screen 2 (step S6A—NO), it deletes the identifier "first text box" and the reference information "OBJ-A" from the correspondence table 400C (step S7B). This deletion causes the correspondence table 400C to transition to the correspondence table 400D. The correspondence table 400D does not register the correspondence relationship between the identifier "first text box" and the reference information "OBJ-A". Finally, the object search device 1 executes each process from step S4B onwards and ends the series of processes.

[0051] Third, if the object search device 1 fails to access the object 312A (step S5A-NO), it deletes the identifier "first text box" and the reference information "OBJ-A" from the correspondence table 400C (step S7B). The subsequent processes are the same as those in the second case described above.

[0052] According to the embodiment described above, the object search device 1 stores a table that manages identifiers of multiple UI elements on a display screen and reference information of multiple objects corresponding to the multiple UI elements. When the object search device 1 first detects a target object corresponding to a target UI element from a model, the object search device 1 registers in the table a correspondence between the identifier of the target UI element and the reference information of the target object.

[0053] Therefore, when searching for a target object corresponding to the same target UI element from a model, the object search device 1 can directly access the target object from the reference information registered in the table. As a result, the object search device 1 can quickly search for the target object from the model. The object search device 1 quickly detects the operation state of the target UI element (or changes in the display screen) based on the property value of the target object, so it can respond immediately to a user's operation on the target UI element. The object search device 1 can support RPA and can support user operation navigation.

[0054] 6 is a hardware configuration diagram of an object search device 1 according to an embodiment. The object search device 1 includes, as its components, a CPU 111, a RAM 112, a ROM 113, a storage 114, a display device 115, an input device 116, and a communication device 117. The components are connected to each other via an internal bus so that they can communicate with each other. The object search device 1 may include at least some of the components.

[0055] The CPU 111 is a processor that executes various processes according to programs. The CPU 111 uses a predetermined area of ​​the RAM 112 as a work area. The CPU 111 realizes each unit (e.g., the search unit 11, the determination unit 12) by reading and executing each program stored in the ROM 113 or the storage 114. The CPU 111 is an example of a processing unit.

[0056] The RAM 112 is a memory that stores various data in a rewritable manner. The RAM 112 is an SDRAM (Synchronous Dynamic Random Access Memory) or the like. The ROM 113 is a memory that stores various data in a non-rewritable manner. The storage 114 is various storage media. The storage 114 may be a drive device that writes or reads various data to or from the storage media. The storage 114 may write or read various data to or from the storage media in accordance with control by the CPU 111. The RAM 112, the ROM 113, or the storage 114 is an example of a storage unit (e.g., the storage unit 13).

[0057] The display device 115 is a device that displays various types of data. The display device 115 is an LCD (Liquid Crystal Display) or the like. The display device 115 displays various types of data based on a display signal from the CPU 111. The display device 115 is an example of a display unit.

[0058] The input device 116 is a device that receives various input operations from the user. The input device 116 is a mouse, a keyboard, etc. The input device 116 receives operations input by the user as instruction signals and transmits the received instruction signals to the CPU 111. The input device 116 may also receive input operations from the user for UI elements on the display screen 2. The input device 116 is an example of an input unit.

[0059] The communication device 117 communicates with external devices via a network under the control of the CPU 111. The communication device 117 is an example of a communication unit.

[0060] Each embodiment of the present invention is presented as an example and does not limit the scope of the present invention. Each embodiment can be implemented in various forms without departing from the spirit of the present invention. Each embodiment may be combined with others, and in such cases, combined effects can be obtained. Each embodiment includes multiple components, and various combinations of these multiple components can result in various inventions. Each embodiment or combination of each component is included within the scope of the present invention.

[0061] 1...Object search device, 2...Display screen, 3, 31, 32, 311, 312, 312A, 312B, 321, 322, 322A, 322B...Objects, 10...Automation / monitoring tool, 11...Search unit, 12...Determination unit, 13...Memory unit, 21...First window, 22...Second window, 111...CPU, 112...RAM, 113...ROM, 114...Storage, 115...Display device 116...input device, 117...communication device, 211...first title bar, 212...first content, 212A...first text box, 212B...first button, 221...second title bar, 222...second content, 222A...second text box, 222B...second button, 300...object model, 400, 400A, 400B, 400C, 400D...correspondence table

Claims

1. An object search device comprising: a storage unit that stores a table that associates identifiers of multiple UI (User Interface) elements that constitute a display screen with reference information for each of multiple objects in a model consisting of multiple objects that respectively correspond to the multiple UI elements; and a search unit that, for a target UI element among the multiple UI elements, obtains the identifier of the target UI element, detects reference information corresponding to the obtained identifier from the table, and uses the detected reference information to search for a target object corresponding to the target UI element from the model.

2. The object search device of claim 1, further comprising a determination unit that determines whether the target UI element is displayed on the display screen based on a property value of the target object, and when it is determined that the target UI element is not displayed on the display screen, the search unit deletes the identifier of the target UI element and the reference information corresponding to the identifier of the target UI element from the table.

3. An object search method in which a computer stores a table that associates, for a plurality of UI (User Interface) elements that constitute a display screen, identifiers of the plurality of UI elements with reference information for each of the plurality of objects in a model consisting of the plurality of objects that respectively correspond to the plurality of UI elements; acquires the identifier of a target UI element from the plurality of UI elements; detects reference information corresponding to the acquired identifier from the table; and uses the detected reference information to search for a target object corresponding to the target UI element from the model.

Images