Operation target modification system and operation target modification method

Abstract

An operation target modification system according to an embodiment comprises: a modification necessity determination unit that, when a GUI object to be operated that is a target of automatic operation among GUI objects in a system screen is not found, compares a first information array for the GUI object to be operated with a second information array for the system screen, and determines whether or not the GUI object to be operated needs to be modified on the basis of the degree of matching between the first information array and the second information array; a candidate extraction unit that, if the GUI object to be operated needs to be modified, generates a search information array including less information than the first information array from the first information array, and extracts a candidate information array including the same information array as the search information array from the second information array; and an estimation unit that estimates, as a target of the automatic operation, the GUI object described by the candidate information array, on the basis of the distance between the candidate information array and the first information array.

Description

Operation target correction system and operation target correction method

[0001] The embodiments relate to a system and a method for modifying an operation target.

[0002] Robotic process automation (RPA) is known for automating system operations to improve business efficiency. Software that performs RPA operates and / or monitors pre-registered GUI objects (e.g., buttons or text boxes) displayed on the screen of a system (or a personal computer (PC)) that uses a GUI (graphical user interface) in accordance with control logic described as scenarios and / or rules. Examples of operations include selecting an object and inputting text. Examples of monitoring include checking whether text in an appropriate format has been entered into a certain text box.

[0003] When the system screen is changed, the registered information about the GUI object to be operated may no longer match the current information. In this case, the control logic does not function correctly. To address this issue, there is a technology that uses machine learning to create logic that predicts the GUI object to be operated, preventing software from operating the wrong GUI object to be operated when the system screen is changed.

[0004] However, even if software-based operation errors can be prevented, ultimately, a person must re-register the target GUI object. Therefore, the software is required to automatically determine which registered target GUI object corresponds to which GUI object on the changed system screen, and to correct the information about the target GUI object based on the determination.

[0005] Hideki Koya, Hajime Nakajima, Ken Masuda, and Ikuya Takahashi, "A Study on User Interface Control Methods to Prevent Miscontrol," Proceedings of the IEICE Conference, p.82, Mar. 2013

[0006] There are two issues when modifying a target GUI object using software. The first issue is determining whether or not information about the target GUI object needs to be modified. When a registered target GUI object cannot be found, it is necessary to determine which of the following two cases applies. The first case is when the target GUI object cannot be found due to a change in the design of the system screen, even though the target GUI object exists on the system screen. The second case is when the current system screen is a system screen different from the system screen on which the target GUI object exists, and the target GUI object does not exist on the current system screen. The second case occurs frequently when the target GUI object is a monitoring target.

[0007] If the first problem is not solved, when software automatically modifies a GUI object to be operated, the software may modify it to an incorrect GUI object, or the software may frequently execute a process to find the GUI object to be used for the modification, deteriorating the performance of the automatic operation.

[0008] The second problem is that when it is determined that a GUI object to be operated needs to be modified, accuracy is required in estimating which GUI object to modify. Different GUI objects have different information configurations. That is, each GUI object has multiple properties set thereto and also has various information configurations that depend on the implementation of the application that provides the GUI. Therefore, a method is required for automatically modifying a GUI object to be operated that does not assume that the GUI object has special information for modification.

[0009] The estimation utilizes the degree of agreement between the information configuration of the target GUI object and the information configuration of each of the various GUI objects on the current system screen. Since it is an estimation, there is a possibility that an incorrect GUI object will be estimated as a target for correction. For this reason, the accuracy of the estimation, that is, the degree to which the information configurations of two GUI objects must match before being estimated as the target GUI object, is required to be manually adjustable.

[0010] The present invention has been made in light of the above circumstances, and its object is to provide a means for automatically correcting information about a GUI object that is the target of automatic operation.

[0011] An embodiment of the operation target correction system includes a correction necessity determination unit, a candidate extraction unit, and an estimation unit. When a target GUI object to be automatically operated among GUI objects in a system screen is not found, the correction necessity determination unit compares a first information array for the target GUI object with a second information array for the system screen and determines whether the target GUI object needs to be corrected based on the degree of agreement between the first information array and the second information array. When the target GUI object needs to be corrected, the candidate extraction unit generates a search information array from the first information array that includes less information than the first information array, and extracts a candidate information array from the second information array that includes an information array identical to the search information array. The estimation unit estimates a GUI object described by the candidate information array as a target of the automatic operation based on the distance between the candidate information array and the first information array.

[0012] According to the embodiment, it is possible to provide a means for automatically correcting information about a GUI object that is the target of automatic operation.

[0013] FIG. 1 shows the configuration of a system including an operation target correction system according to the first embodiment. FIG. 2 shows the hardware configuration of an information processing system including an operation target correction system according to the first embodiment. FIG. 3 shows the functional configuration of a user terminal including an operation target correction system according to the first embodiment. FIG. 4 shows the functional configuration of the operation target correction system according to the first embodiment. FIG. 5 shows an example of GUI objects and data handled by the operation target correction system according to the first embodiment. FIG. 6 shows an example of an operation flow of the operation target correction system according to the first embodiment. FIG. 7 shows a partial operation flow of the operation target correction system according to the first embodiment. FIG. 8 schematically shows an example of a partial operation of the operation target correction system according to the first embodiment. FIG. 9 shows a partial operation flow of the operation target correction system according to the first embodiment. FIG. 10 schematically shows an example of a partial operation of the operation target correction system according to the first embodiment. FIG. 11 schematically shows an example of a partial operation of the operation target correction system according to the first embodiment. FIG. 12 shows a partial operation flow of the operation target correction system according to the first embodiment. FIG. 13 schematically shows an example of a partial operation of the operation target correction system according to the first embodiment. FIG. 14 schematically shows an example of a part of the operation of the operation target modification system according to the first embodiment.

[0014] Hereinafter, embodiments will be described with reference to the drawings.

[0015] 1. First Embodiment 1.1. Configuration (Structure) 1.1.1. Configuration of Information Processing System FIG. 1 shows the configuration of a system including an operation target correction system according to the first embodiment. As shown in FIG. 1, the information processing system 100 includes a user terminal 10 and an information processing device 20. The information processing device 20 is a computer for providing a service to a service user. The information processing device 20 receives input data and provides the service to the service user based on the input data. Examples of the information processing device 20 include a server and a database device. The information processing device 20 uses a GUI and receives data provided using the GUI as input data.

[0016] The user terminal 10 is a terminal of a user of a service provided by the information processing device 20. The user terminal 10 can perform RPA when using a service provided by the information processing device 20. The user terminal 10 automatically performs operations on GUI objects provided by the information processing device 20. The user terminal 10 operates and / or monitors GUI objects pre-registered in the user terminal 10 in accordance with control logic described as pre-registered scenarios and / or rules. The user terminal 10 includes an operation target correction system according to the first embodiment, as described below.

[0017] Fig. 2 shows the hardware configuration of a user terminal including the operation target modification system according to the first embodiment. As shown in Fig. 2, the user terminal 10 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage 104, a display device 105, an input device 106, and a communication device 107.

[0018] The CPU 101 is an integrated circuit capable of executing various programs. By executing the programs, the CPU 101 controls the overall operation of the user terminal 10. By the CPU 101 executing the programs stored in the ROM 102 and loaded onto the RAM 103, the user terminal 10 performs various operations and functions as various functional blocks.

[0019] The ROM 102 is a non-volatile memory that stores programs and data such as control data for controlling the user terminal 10. The RAM 103 is a volatile memory that temporarily stores data. The RAM 103 also functions as a work area for the CPU 101.

[0020] The storage 104 is an auxiliary storage device of the user terminal 10. Examples of the storage 104 include a hard disk drive (HDD), a solid state drive (SSD), and a memory card. The storage 104 stores data necessary for executing an automatic operation. The storage 104 may also store a program that causes the CPU 101 to execute the automatic operation.

[0021] The display device 105 is a device for visually presenting information to the user of the user terminal 10. Examples of the display device 105 include an LCD (Liquid Crystal Display) or an EL (Electroluminescence) display. The display device 105 displays a screen for the user to operate the user terminal 10. The display device 105 can display a system screen based on the GUI of the information processing device 20. The system screen includes a plurality of GUI objects.

[0022] The input device 106 is a device that enables a user of the user terminal 10 to input data to the user terminal 10. Examples of the input device 106 include a keyboard, a mouse, a touch panel, or a button switch.

[0023] The communication device 107 is a device that enables the user terminal 10 to communicate with the outside world. The communication device 107 is configured to be able to communicate with the information processing device 20 via wired and / or wireless communication.

[0024] The information processing device 20 and the user terminal 10 may be included in one device. In this case, the information processing device 20 is included in the user terminal 10. The functions of the information processing device 20 are realized by part of the resources provided by the hardware of the user terminal 10.

[0025] 3 shows the functional configuration (functional blocks) of a user terminal including an operation target correction system according to the first embodiment. As shown in FIG. 3, the user terminal 10 includes an operation target correction system 1, an automatic operation execution unit 2, and a storage unit 3. The operation target correction system 1, the automatic operation execution unit 2, and the storage unit 3 are realized by some of the resources provided by the hardware of the user terminal 10.

[0026] The storage unit 3 stores automatic operation target information 31, system screen information 32, and a threshold value 33. The automatic operation target information 31 is information that identifies a GUI object that is a target of automatic operation by the user terminal 10 among GUI objects in the information processing device 20. A GUI object is any type of element that is displayed on a GUI-based screen. Examples of GUI objects include an application, a document, a button, a text box, and a combo box. A GUI object may include further GUI objects.

[0027] The automatic operation target information 31 includes multiple properties of the GUI object. The properties are any kind of information related to the GUI object displayed on the system screen of the user terminal 10, particularly on the display device 105. Examples of the information include the name or identifier of the GUI object, the type of the GUI object, the text displayed on the system screen, the coordinates of the GUI object, and the position of the GUI object relative to the origin. The set of all properties for each GUI object is unique to that GUI object. In one example, the automatic operation target information 31 is registered in advance by the user of the information processing system 100.

[0028] The system screen information 32 describes or is information about one or more system screens displayed on the user terminal 10. In one example, the system screen information is provided by the information processing device 20. The system screen information 32 for a certain system screen includes a set of properties of GUI objects included in the system screen. The set of properties of GUI objects for a certain system screen is unique to the system screen.

[0029] The threshold value 33 is a reference value used in the judgment by the operation object modification system 1. In one example, the threshold value 33 can be input by a user of the operation object modification system 1 using the input device 106. The threshold value 33 includes Th1 and Th2, which will be described later.

[0030] The automatic operation execution unit 2 is a functional block that executes automatic operations on GUI objects provided by the information processing device 20, i.e., displayed on the user terminal 10. The automatic operation execution unit 2 automatically performs operations on the GUI objects using automatic operation target information 31 and system screen information 32. The automatic operation execution unit 2 follows the control logic stored in the storage unit 3.

[0031] The operation target correction system 1 is a functional block that corrects an operation target GUI object. The operation target correction system 1 corrects an operation target GUI object, for example, at the request of the automatic operation execution unit 2. For example, if the automatic operation execution unit 2 cannot find an operation target GUI object specified by the control logic, the automatic operation execution unit 2 requests the operation target correction system 1 to correct the operation target GUI object. Upon receiving the request, the operation target correction system 1 determines whether or not the operation target GUI object needs to be corrected, and corrects the operation target GUI object based on the result of the determination of whether or not it needs to be corrected.

[0032] 4 shows the functional configuration of the operation target correction system 1 according to the first embodiment. As shown in FIG. 4, the operation target correction system 1 includes a correction necessity determination unit 11, a candidate extraction unit 12, and an estimation unit 13.

[0033] The correction necessity determining unit 11 is a functional block that determines whether or not the operation target GUI object needs to be corrected when the operation target GUI object cannot be found.

[0034] The candidate extraction unit 12 is a functional block that extracts candidates for other GUI objects to be operated that can become new targets for automatic operations on the GUI object to be operated (i.e., to be replaced) when the GUI object to be operated needs to be modified.

[0035] The estimation unit 13 is a functional block that determines, from among multiple candidate operation target GUI objects, an operation target GUI object that is estimated to be a new target of automatic operation targeting an operation target GUI object that could not be found.

[0036] FIG. 5 shows an example of GUI objects and data handled by the operation target modification system according to the first embodiment.

[0037] The system screen 41 is a screen displayed on the display device 105 and is configured using a GUI. The system screen 41 has a configuration according to the description in the system screen information 32. For example, the system screen 41 displays, as GUI objects, an application (not shown), a menu (not shown), a unique resource location (URL) bar (not shown), buttons 42, text (or labels) 43, and text boxes 44. The buttons 42 include a "Back" button and a "Register" button. The text 43 includes the character string "Service Name" and the character string "Plan." Each text box 44 is aligned with one piece of text 43. Each text box 44 accepts input of a character string of a feature described by the text aligned with the text box. In one example, the text box 44 surrounded by a dashed line, i.e., the text box 44 for inputting the service name, is the GUI object to be operated. The following description is based on this example.

[0038] The GUI object to be operated or its characteristics are described by an information array. The information array includes one or more property groups. Each property group is a set of one or more properties. The properties are general-purpose properties and do not include properties that need to be specifically defined. The property groups may have a hierarchical structure. An example in which the property groups and system screen information 32 have a hierarchical structure will be described below. Examples of the format of the hierarchical system screen information 32 and information array include formats based on HyperText Markup Language (HTML) and an accessibility application programming interface (API). Examples of the accessibility API include User Interface Automation (UIA) for Windows (registered trademark) applications and Java Access Bridge for Java (registered trademark) applications.

[0039] The properties of the system screen information 32 can be obtained by analyzing the system screen information 32. In one example, if the system screen information 32 is HTML, a tag can be used as one property. In one example, the properties can be obtained by using a UIA such as a UIA and Java Access Bridge.

[0040] In one example, the system screen information 32 describing the system screen 41 has the following structure:

[0041] The system screen information 32 includes an element "browser application" in the first (top) layer. Each element in the system screen information 32 includes a property group, and the property group consists of multiple properties (property 1, property 2, property 3, ...).

[0042] The system screen information 32 includes, at the second level, the elements "menu," "URL bar," and "document," which are included in the element "browser application." The GUI object to be operated is included in the element "document."

[0043] In the third layer, the system screen information 32 includes the elements "Group," "Button," "Button," and "Group" contained in the element "Document." The GUI object to be operated is included in the second element from the top, "Group." The second element from the top, "Group," describes a set of GUI objects for inputting a service name. The element "Button" describes button 42.

[0044] The system screen information 32 includes two "group" elements contained within the "group" element at the fourth level. The GUI object to be operated is included in the first "group" element from the top. The second "group" element from the top describes a set of GUI objects for inputting a plan.

[0045] The system screen information 32 includes elements "label" and "textbox" included in the element "group" at the fifth level. The GUI object to be operated is included in the element "textbox."

[0046] The property group for the GUI object to be operated is assigned an index. Property groups located at higher levels have smaller index numbers. Specifically, the property group for the "Browser Application" element at the first level has an index of 1. The property group for the "Document" element at the second level has an index of 2. The property group for the "Group" element at the third level has an index of 3. The property group for the "Group" element at the fourth level has an index of 4. The property group for the "Text Box" element at the fifth level has an index of 5.

[0047] The length of the information array is equal to the number of properties. In the example of Figure 5, the length of the information array is 5.

[0048] When the information array has a hierarchical structure, it is not necessary for the information array to include information for all hierarchical levels. As long as the information array can uniquely identify the GUI object to be operated, it does not have to include information for one or more hierarchical levels.

[0049] 1.2. Operation 1.2.1. Overall Operation of Operation Target Correction Fig. 6 shows an example of the operation flow of the operation target correction system according to the first embodiment. The flow in Fig. 6 is started in response to a request from the automatic operation execution unit 2, for example, when the automatic operation execution unit 2 cannot find an operation target GUI object that is specified to be operated according to the control logic during the automatic operation process on the current system screen (the system screen specified as the target by the control logic). Some steps will be described in detail later.

[0050] The correction necessity determination unit 11 determines whether or not the GUI object to be operated needs to be corrected (ST1). If correction is not necessary (ST2_No), the flow ends.

[0051] If correction is necessary (ST2_Yes), the candidate extraction unit 12 extracts candidate GUI objects to be automatically operated instead of the operation target GUI object (ST3). In step ST3, one or more candidate GUI objects are extracted.

[0052] The estimation unit 13 estimates a GUI object to be automatically operated from among the candidate GUI objects (ST5). The estimation includes finding a candidate GUI object having a property set highly similar to the property set of the operation target GUI object based on the distance (or difference) between the property set of the operation target GUI object and the property set of the candidate GUI object.

[0053] If there is no candidate GUI object having a property set whose distance from the property set of the operation target GUI object is less than Th1 (ST5_No), the flow ends. If there is a candidate GUI object having a property set whose distance from the property set of the operation target GUI object is less than Th1 (ST5_Yes), the estimation unit 13 modifies (replaces) the operation target GUI object with the found candidate GUI object. Specifically, the modification includes updating the property set of the operation target GUI object in the automatic operation target information 31 with a property set whose distance from the property set of the operation target GUI object is less than Th1. The flow ends when step ST6 ends.

[0054] 1.2.2. Correction Necessity Determination Processing Fig. 7 shows a flow of part of the operation of the operation target correction system according to the first embodiment. Specifically, Fig. 7 shows the flow of correction determination processing by the correction necessity determination unit 11, and shows the flow of step ST1 of the flow in Fig. 6. Fig. 7 relates to the determination of whether a certain operation target GUI object needs to be corrected.

[0055] The correction necessity determination unit 11 sets the variable I to 1 (ST11). The correction necessity determination unit 11 acquires a group of properties of the index I of the operation target GUI object (ST12). The group of properties is acquired from the automatic operation target information 31.

[0056] The correction necessity determination unit 11 determines whether a property group matching the acquired property group is present in the system screen information 32 of the system screen on which the control logic specifies that the target GUI object exists (ST13). Hereinafter, the system screen on which the control logic specifies that the target GUI object exists may be referred to as the current system screen.

[0057] If there is a GUI object in the system screen information 32 that has a property group that matches the acquired property group (ST13_Yes), the correction necessity determination unit 11 determines whether the variable I is equal to the length of the information array of the GUI object to be operated (ST14).

[0058] If the variable I is different from the length of the information array of the GUI object to be operated (ST14_No), the correction necessity determination unit 11 increments the variable I by 1 (ST15). Step ST15 is followed by step ST12. Following this route, processing for the next index occurs.

[0059] If the variable I is equal to the length of the information array of the GUI object to be operated (ST14_Yes), the correction necessity determination unit 11 returns "No" as the result of the correction necessity determination process (ST16). The processing result is received by the candidate extraction unit 12. The flow ends when step ST16 ends.

[0060] If there is no GUI object in the system screen information 32 that has a property set that matches the acquired property set (ST13_No), the correction necessity determination unit 11 determines whether (I-1) / (length of the information array of the GUI object to be operated) exceeds Th2 (ST17). The threshold value is the threshold value 33 stored in the storage unit 3. If (I-1) / (length of the array) does not exceed Th2 (ST17_No), the flow proceeds to step ST16.

[0061] If (I-1) / array length exceeds Th2, this indicates that the properties of the target GUI object match the properties of the system screen information 32 in the upper layer. Therefore, it is presumed that the reason the target GUI object could not be found is not because the current system screen is different from the system screen on which the target GUI object is located. Therefore, it is presumed that the target GUI object needs to be modified to address the fact that the target GUI object could not be found. Based on this, the modification necessity determination unit 11 returns "needed" as the result of the modification necessity determination process (ST18). The processing result is received by the candidate extraction unit 12. The flow ends upon completion of step ST18.

[0062] Fig. 8 schematically illustrates an example of a portion of the operation of the operation target correction system according to the first embodiment. Specifically, Fig. 8 illustrates an example of a correction necessity determination process performed by the correction necessity determination unit 11, and illustrates an example of a result of execution of the flow in Fig. 7 .

[0063] By performing three loops of steps ST12, ST13, ST14, and ST15, an element (GUI object) having a property set matching the property set of index 1, index 2, and index 3 was found in the system screen information 32. On the other hand, an element having a property set matching the property set of index 4 was not found in the system screen information 32. In this case, the determination in step ST17 is as follows: (I-1) ÷ 3 = 3 ÷ 5 = 0.6. Based on the example where Th2 is 0.5, it is determined that correction is necessary. 1.2.3. Candidate Extraction Process FIG. 9 shows a flow of part of the operation of the operation target correction system according to the first embodiment. Specifically, FIG. 9 shows the flow of the candidate extraction process by the candidate extraction unit 12, and illustrates the flow of step ST3 of the flow in FIG. 6.

[0064] The candidate extraction unit 12 generates a search information array using the information array of the operation target GUI object (ST21). The search information array has a higher degree of freedom than the information array of the operation target GUI object. That is, the search information array contains less information than the information contained in the information array of the operation target GUI object, and in one example, contains fewer properties than the properties contained in the information array of the operation target GUI object. Figure 10 schematically illustrates an example of a portion of the operation of the operation target correction system according to the first embodiment. Specifically, Figure 10 illustrates an example of a method for creating a search information array.

[0065] The search information array includes only some of the property groups of the information array of the GUI object to be operated, and / or only some of one or more property groups. In the example of FIG. 10 , the entire property groups of indexes 2, 3, and 4 for intermediate elements in the hierarchical structure of the information array of the GUI object to be operated are not included in the search information array. In the example of FIG. 10 , the search information array includes the property group of index 1 at the top level. Meanwhile, the property group of the search information array created from the property group of index 1 includes only property 1. Also, in the example of FIG. 10 , the search information array includes the property group of index 5 at the bottom level. Meanwhile, the property group of the search information array created from the property group of index 5 includes only property 1.

[0066] Returning to Fig. 9, the candidate extraction unit 12 sets the variable I to 1 (ST22). The candidate extraction unit 12 acquires a property group of the index I of the search information array (ST23).

[0067] The candidate extraction unit 12 determines whether there is a property group matching the acquired property group in the system screen information 32 of the current system screen (ST24). If there is a property group matching the acquired property group, the candidate extraction unit 12 temporarily stores the matching property group and temporarily stores all matching property groups. If there is no property group matching the acquired property group in the system screen information 32 (ST24_No), the flow ends.

[0068] If the system screen information 32 contains a property group that matches the acquired property group (ST24_Yes), the candidate extraction unit 12 determines whether the variable I is equal to the length of the information array of the GUI object to be operated (ST25).

[0069] If the variable I is different from the length of the information array of the GUI object to be operated (No in ST25), the candidate extractor 12 increments the variable I by 1 (ST26). Step ST26 is followed by step ST23. Following this route, processing for the next index occurs.

[0070] If the variable I is equal to the length of the information array of the GUI object to be operated (ST25_Yes), the candidate extraction unit 12 returns all property groups found so far that match the property groups in the search information array (ST18). The return is made to the estimation unit 13. The flow ends when step ST18 ends.

[0071] Fig. 11 schematically illustrates an example of a portion of the operation of the operation target modification system according to the first embodiment. Specifically, Fig. 11 illustrates an example of candidate extraction processing by the candidate extraction unit 12, and illustrates an example of the result of execution of the flow in Fig. 9. The search information array is the same as the example illustrated in Fig. 10.

[0072] The system screen information 32 includes two information arrays that include both property 1 of the property group at index 1 and property 1 of the property group at index 2. The first information array is an information array for the first text box at the fifth level. The second information array is an information array for the second text box at the fifth level. These information arrays that include all of the property groups included in the search information array in the system screen information 32 are extracted as candidate information arrays.

[0073] 1.2.4 Estimation Process Fig. 12 shows a flow of a part of the operation of the operation target modification system according to the first embodiment. Specifically, Fig. 12 shows a flow of the estimation process by the estimation unit 13, and shows the flow of step ST4 of the flow in Fig. 6.

[0074] The estimation unit 13 selects one unselected candidate information sequence from all the candidate information sequences (ST31).

[0075] The estimation unit 13 calculates the distance between the information array of the operation target GUI object to be corrected and the selected candidate information array (ST32). The distance may be calculated by any method. Examples of the distance calculation method include developing each of the information array of the operation target GUI object to be corrected and the candidate information array as sentences and calculating the difference between the sentences, and calculating the distance using a formula.

[0076] FIG. 13 schematically illustrates an example of a portion of the operation of the operation target correction system according to the first embodiment, illustrating an example of distance calculation by comparing sentences. The information array of the operation target GUI object to be corrected and the candidate information array are each converted into a sentence containing all properties linked by commas. If each information array has a hierarchical structure, properties at the same level are arranged in the same row, and properties at different levels are arranged in different rows. Examples of methods for calculating the distance between two sentences include calculation using string edit distance and sentence similarity. Examples of string edit distance include the Levenshtein distance and the Jaro-Winkler distance. Examples of calculation of sentence similarity include calculation using N-gram.

[0077] 14 is a diagram showing an example of a part of the operation of the operation target correction system according to the first embodiment, and shows an example of calculating the distance by comparing sentences. The property β of the index (layer) α of the information array of the operation target GUI object to be corrected is expressed as the element P α，β Similarly, the property β of the index α of the candidate information array is defined as α，β In one example, the distance between the information sequence of the GUI object to be modified and the candidate information sequence is calculated as Equation 1.

[0078]

[0079] Returning to FIG. 12 , the estimation unit 13 determines whether all candidate information sequences have been selected in step ST31 (ST33). If not all candidate information sequences have been selected (ST33_No), the flow proceeds to ST31. If all candidate information sequences have been selected (ST33_Yes), the estimation unit 13, particularly its distance calculation function block, presents the distance for each of all candidate information sequences. The distances are received by the function block of the estimation unit 13 that performs step ST5 of the flow in FIG. 6 .

[0080] 1.3. Advantages (Effects) When the operation target correction system 1 cannot find the operation target GUI object on the current system screen, it retrieves the properties of the information array of the operation target GUI object, layer by layer, starting from the top layer, and searches for properties identical to the retrieved properties in the system screen information 32 of the current system screen. If the search returns no hits, the operation target correction system 1 determines whether the operation target GUI object needs to be corrected based on the layer of the properties for which the identical properties are not found in the system screen information 32. If the properties of the operation target GUI object and the properties in the system screen information 32 are identical across a certain percentage of the layers of the information array of the operation target GUI object, the current system screen is presumed to be a system screen related to the operation target GUI object. Otherwise, the current system screen is presumed to be a system screen unrelated to the operation target GUI object. Therefore, by determining whether correction is necessary, it is possible to eliminate cases in which the operation target GUI object is not found because the current system screen is a system screen unrelated to the operation target GUI object.

[0081] When correction is necessary, the operation target correction system 1 generates a search information array from the properties of the operation target GUI object, which contains less information than the information array of the operation target GUI object, and searches for the search information array within the system screen information 32 of the current system screen. The operation target correction system 1 uses the found information array as a candidate information array and estimates a GUI object corresponding to the operation target GUI object based on the distance between the candidate information array and the information array of the operation target GUI object. The estimation is performed using general properties defined in various formats widely used to describe system screens. Therefore, adding special, dedicated information is not required to correct the operation target GUI object.

[0082] In this manner, the operation target correction system 1 determines whether correction is necessary and estimates the GUI object corresponding to the operation target GUI object. Therefore, a means for automatically correcting information about the GUI object that is the target of automatic operation is provided. When creating an automatic operation scenario, appropriately setting information about the operation target GUI object so that the operation target GUI object can be successfully found places a heavy burden on the creator and requires knowledge and experience. Because the operation target correction system 1 automatically tracks the operation target GUI object, the burden on the creator of the control logic to create a scenario and acquire knowledge and experience is reduced.

[0083] Furthermore, there are cases where the operation target GUI object needs to be modified due to the update of the automatic operation system, etc. In such cases, the modification is performed manually, whereas the operation target modification system 1 reduces the workload involved in updating the control logic.

[0084] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.

[0085] 100...information processing system, 10...user terminal, 20...information processing device, 101...CPU, 102...ROM, 103...RAM, 104...storage, 105...display device, 106...input device, 107...communication device, 1...operation target correction system, 2...automatic operation execution unit, 3...storage unit, 11...correction necessity determination unit, 12...candidate extraction unit, 13...estimation unit, 31...automatic operation target information, 32...system screen information, 33...threshold

Claims

1. An operation target correction system comprising: a correction necessity determination unit that, when a target GUI object that is a target of automatic operation among GUI objects in a system screen cannot be found, compares a first information array for the target GUI object with a second information array for the system screen, and determines whether or not the target GUI object needs to be corrected based on the degree of agreement between the first information array and the second information array; a candidate extraction unit that, when the target GUI object needs to be corrected, generates a search information array from the first information array that contains less information than the first information array, and extracts a candidate information array from the second information array that contains an information array identical to the search information array; and an estimation unit that estimates a GUI object described by the candidate information array as a target of the automatic operation based on the distance between the candidate information array and the first information array.

2. The operation target modification system of claim 1, wherein the first information array includes a plurality of first property groups including one or more properties for the GUI object to be operated, and the second information array includes a plurality of second property groups including one or more properties for the GUI object in the system screen.

3. The operation target correction system according to claim 2, wherein the multiple first property groups in the first information array have a hierarchical structure, and the multiple second property groups in the second information array have a hierarchical structure, and the correction necessity determination unit selects the first property groups one by one from the top layer, determines whether the second property group includes a property group identical to a selected one of the multiple first property groups, and when the second property group does not include a property group identical to a selected one of the multiple first property groups, if a ratio of the number of first property groups selected up to that point to the number of the multiple first property groups exceeds a first threshold value, determines that correction of the operation target GUI object is necessary.

4. A method for correcting an operation target, comprising: when a target GUI object to be automatically operated among GUI objects in a system screen is not found, comparing a first information array for the target GUI object with a second information array for the system screen, and determining whether or not the target GUI object needs to be corrected based on the degree of agreement between the first information array and the second information array; if the target GUI object needs to be corrected, generating a search information array from the first information array that contains less information than the first information array, and extracting a candidate information array from the second information array that contains an information array identical to the search information array; and presuming a GUI object described by the candidate information array as the target of the automatic operation based on the distance between the candidate information array and the first information array.

Images