Processing device, processing method, and processing program

Abstract

A processing device (10) includes: an object registration unit (151) that specifies an object of automatic operation displayed on a system screen, and registers and manages position information and an alias of the specified object of automatic operation as object information; a control description unit (152) that uses the object information to describe automatic operation control in a low-code and / or no-code description, describes execution conditions, if any, and converts the description into a prescribed script; an object monitoring unit (153) that monitors information about an application on the desktop, determines whether an execution condition set by the control description unit (152) is satisfied, and outputs a corresponding script if the execution condition is determined to be satisfied; and an execution unit (154) that interprets the script converted by the control description unit (152), replaces the alias with object information about the object of automatic operation, and sequentially processes the script to automatically operate the system screen.

Description

[Technical Field]

[0001] The present invention relates to a processing device, a processing method, and a processing program. [Background technology]

[0002] Robotic Process Automation (RPA) is making progress in improving business efficiency by automating system operations. In RPA, automated operation procedures are written in the form of scenarios and rules. Many of these procedures are developed using low-code / no-code development, making it easy for even non-programmers to write the procedures.

[0003] There is a method for describing the procedure for automatic operations in the form of a scenario using a flowchart (Non-Patent Document 1). In the method described in Non-Patent Document 1, actions such as "click" and "set string" are defined as blocks in a flowchart, and these are connected to describe the control. In this method, information on the target of automatic operation is contained in each action block.

[0004] There is also a method for describing automatic operation procedures in a dedicated script language or various programming languages ​​(Non-Patent Document 2). In the method described in Non-Patent Document 2, control is described using the syntax or API (Application Programming Interface) / commands of each language, and target information is hard-coded as arguments to the API / commands. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Fumihiro Yokose, Hiroyuki Tanaka, Takahiro Toyoda, Kentaro Hotta, Akira Inoue, and Satoru Sugimoto, "Overview and Application Examples of UMS: Terminal Operation Automation Software," IEICE Technical Report, ICM2011-73, pp.149-154, 2012. [Non-patent document 2] Software Freedom Conservancy, “The Selenium Browser Automation Project,” [online], [Retrieved March 28, 2022], Internet<URL: https: / / www.selenium.dev / documentation / > Summary of the Invention [Problem to be solved by the invention]

[0006] In this way, in existing RPA, the targets of automated operation that are susceptible to environmental factors such as the system screen, and the control over how to handle them, are described as a single entity in the form of scenarios and rules.

[0007] For this reason, with existing RPA, when there was a change in the environment, such as when a system screen that was the target of automated operation was modified due to a system upgrade, there was a problem in that all scenarios and rules related to the system screen in question had to be modified.

[0008] The present invention has been made in consideration of the above, and aims to provide a processing device, processing method, and processing program that can reduce the impact of environmental changes on scenario and rule modifications. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems and achieve the object, the processing device of the present invention is characterized by having: a registration unit that specifies an automatic operation target displayed on a system screen, and registers and manages the location information and alias of the specified automatic operation target as target information; a description unit that uses the target information registered by the registration unit to describe control of the automatic operation using low-code and / or no-code, describes the execution conditions if there are execution conditions, and converts them into a specified script; a monitoring unit that monitors information of an application on the desktop, determines whether the execution conditions set by the description unit are met, and outputs a corresponding script if it is determined that the execution conditions are met; and an execution unit that interprets the script converted by the description unit, replaces the alias with object information of the automatic operation target, processes the script sequentially, and performs automatic operation of the system screen. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the influence of environmental changes on the modification of scenarios and rules. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of a processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a flowchart showing the procedure of the automatic operation process of the system screen according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating the processing procedure of the target registration processing shown in FIG. [Figure 4] FIG. 4 is a flowchart showing the processing procedure of the control description processing shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 6] FIG. 6 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 7]FIG. 7 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 8] FIG. 8 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 9] FIG. 9 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 10] FIG. 10 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 11] FIG. 11 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 12] FIG. 12 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 13] FIG. 13 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 14] FIG. 14 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 15] FIG. 15 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 16] FIG. 16 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 17] FIG. 17 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 18] FIG. 18 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 19] FIG. 19 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 20] FIG. 20 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 21] FIG. 21 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 22] FIG. 22 is a flowchart showing the procedure for updating the target list when the system screen changes. [Figure 23]FIG. 23 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 24] FIG. 24 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 25] FIG. 25 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 26] FIG. 26 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 27] FIG. 27 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 28] FIG. 28 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 29] FIG. 29 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 30] FIG. 30 is a diagram showing an example of a menu screen displayed on the automatic operation target device. [Figure 31] FIG. 31 is a diagram illustrating an example of a computer that implements a processing device by executing a program. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0013] [Embodiment Mode] In this embodiment, a processing device and a processing method for automatically operating a system screen will be described.

[0014] In this embodiment, information related to automatic operations is managed by separating it into control information and target information. Here, target refers to an object that is the target of automatic operation, such as a button or text box on a system screen. Control refers to processing such as setting a value for the target of automatic operation, processing such as obtaining the set value, processing such as repeating these processing, and processing such as branching based on conditions.

[0015] In the embodiment, information that is easily affected by environmental changes such as system upgrades is generated for each system as target information (aliases, monitoring conditions), and various controls for the same system are described using common target information (aliases, monitoring conditions). As a result, in the embodiment, when an environmental change such as a change in the system screen occurs, only the target information needs to be modified, eliminating the need to modify the controls described using the target information.

[0016] [Processing equipment] The processing device according to the embodiment will be described below. As shown in Fig. 1, the processing device 10 includes an input unit 11, an output unit 12, a communication unit 13, a storage unit 14, and a control unit 15.

[0017] The input unit 11 is realized using input devices such as a keyboard, mouse, microphone, etc., and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a speaker, a display device (desktop) such as a liquid crystal display, a printing device such as a printer, etc.

[0018] The communication unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and external devices via telecommunication lines such as a LAN (Local Area Network) or the Internet.

[0019] The storage unit 14 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an optical disk. The storage unit 14 may be a data-rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non-Volatile Static Random Access Memory). The storage unit 14 stores in advance processing programs for operating the processing device 10 and data used during execution of the processing programs, or stores these temporarily each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication unit 13.

[0020] The storage unit 14 has a target list 141 and a condition list 142. The target list 141 is a list of target information in which the location information and aliases of automatic operation targets are registered. The condition list 142 is a list of monitoring conditions set using aliases. The conditions include existence conditions, value conditions, and / or timing conditions.

[0021] The control unit 15 controls the entire processing device 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0022] The control unit 15 also has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. The control unit 15 also functions as various processing units when various programs are run. For example, the control unit 15 has a target registration unit 151 (registration unit), a control description unit 152 (description unit), a target monitoring unit 153 (monitoring unit), and an execution unit 154.

[0023] The target registration unit 151 specifies an automatic operation target displayed on the system screen, and registers and manages the location information and alias of the specified automatic operation target as target information. The target registration unit 151 generates information that is easily affected by environmental changes such as system renewal as target information (aliases, monitoring conditions) for each system, and describes various controls for the same system using common target information (aliases, monitoring conditions).

[0024] The location information registered by the object registration unit 151 is expressed hierarchically by object information or indexes. The object registration unit 151 defines a unique alias for each piece of location information.

[0025] The target registration unit 151 sets monitoring conditions (hereinafter referred to as conditions) using aliases. The conditions include existence conditions, value conditions, and / or timing conditions. The target registration unit 151 defines a unique condition name for each condition.

[0026] The control description unit 152 describes the control of the automatic operation in low code and / or no code using the target information registered by the target registration unit 151. The control description unit 152 describes the execution conditions if any.

[0027] Then, the control description unit 152 converts the script into a predetermined script that can be interpreted by the execution unit 154 (described later). The control description unit 152 describes the control using either an immediate execution method or an execution method that is executed when an execution condition set by combining condition names is satisfied.

[0028] The target monitoring unit 153 monitors information about the application displayed on the desktop and determines whether it satisfies the execution conditions set by the control description unit 152. If the target monitoring unit 153 determines that the execution conditions are satisfied, in other words, if a situation that matches the execution conditions set by the control description unit 152 occurs, the target monitoring unit 153 outputs a script corresponding to the execution conditions and passes it to the execution unit 154.

[0029] The execution unit 154 is implemented by an arbitrary script engine. The execution unit 154 interprets the script converted by the control description unit 152. The execution unit 154 replaces the alias with object information to be automatically operated, and processes the script sequentially to automatically operate the system screen.

[0030] The execution unit 154 acquires the location information from the alias of the target information described in the script, searches for an automatic operation target that matches the location information from among the applications currently displayed on the desktop, and replaces it with object information. The execution unit 154 sequentially processes the replaced scripts and executes the automatic operation.

[0031] [Automatic operation processing of system screen] FIG. 2 is a flowchart showing the procedure of the automatic operation process of the system screen according to the embodiment.

[0032] As shown in FIG. 2, first, the processing device 10 specifies an automatic operation target displayed on the system screen, and performs a target registration process to register the position information and alias of the specified automatic operation target as target information (step S1).

[0033] The processing device 10 performs a control description process (step S2) in which the control of the automatic operation is described using low code and / or no code, and if there are execution conditions, the execution conditions are described and converted into a specified script that can be interpreted by the execution unit 154.

[0034] The target monitoring unit 153 determines whether or not there is an execution condition for the output in the control description processing (step S3). If there is an execution condition (step S3: Yes), the target monitoring unit 153 performs target monitoring to monitor the application displayed on the desktop (step S4), and determines whether or not a situation has occurred that satisfies the execution condition given in the control description processing (step S5).

[0035] If the target monitoring unit 153 determines that a situation satisfying the execution condition has not occurred (step S5: No), the process returns to step S4 and performs target monitoring. If the target monitoring unit 153 determines that a situation satisfying the execution condition has occurred (step S5: Yes), the target monitoring unit 153 passes the corresponding script to the execution unit 154 (step S6).

[0036] If there are no execution conditions (step S3: None), or after processing step S6, the execution unit 154 interprets the corresponding script, replaces the alias with object information to be automatically operated, processes the script sequentially, and executes the automatic operation of the system screen (step S7).

[0037] [Target registration process] Next, the object registration process (step S1) will be described. Fig. 3 is a flowchart showing the processing procedure of the object registration process shown in Fig. 2.

[0038] 3, when an automatic operation target is designated by mouse operation (step S11), the target registration unit 151 generates position information of the automatic operation target (step S12) and sets an alias for the automatic operation target (step S13). Then, the target registration unit 151 adds the position information and alias generated in steps S12 and S13 to a target list as a set of target information (step S14).

[0039] The object registration unit 151 determines whether or not the registration of the necessary automatic operation objects has been completed (step S15). If the registration of the necessary automatic operation objects has not been completed (step S15: No), the object registration unit 151 returns to step S11.

[0040] If the registration of the necessary automatic operation targets has been completed (step S15: Yes), the target registration unit 151 determines whether the registration of the conditions to be monitored has been completed (step S16).

[0041] If the registration of the condition to be monitored has not been completed (step S16: No), the target registration unit 151 specifies an alias and a condition (step S17) and sets a condition name (step S18). The target registration unit 151 adds the condition name and (alias, condition) specified and set in steps S17 and S18 to the condition list (step S19), and returns to step S16.

[0042] When the registration of the conditions to be monitored is completed (step S16: Yes), the target registration unit 151 outputs a target list and a condition list (step S20).

[0043] [Control statement processing] Next, the control description process (step S2) will be described. Fig. 4 is a flowchart showing the processing procedure of the control description process shown in Fig. 2.

[0044] 4, the control description unit 152 refers to the target list 141 and the condition list 142 (step S31). The control description unit 152 acquires an alias from the target list (step S32) and describes control of the automatic operation using any low-code / no-code method and the alias (step S33). The control description unit 152 converts the low-code and / or no-code description into a predetermined script that can be interpreted by the execution unit 154 (step S34).

[0045] The control description unit 152 determines whether or not there is an execution condition corresponding to this control (step S35). If there is no execution condition (step S35: NO), the control description unit 152 outputs the corresponding script to the execution unit 154 (step S36).

[0046] If there is an execution condition (step S35: YES), the control description unit 152 acquires the condition name from the condition list (step S37), and describes the execution condition by combining the acquired condition name (step S38). The control description unit 152 outputs the described execution condition and script to the execution unit 154 (step S39).

[0047] [Implementation example] Next, an example of implementation of the automatic operation control process in this embodiment will be described. Figures 5 to 20 are diagrams showing examples of menu screens displayed on an automatic operation target device. The automatic operation target device may be the processing device 10, or may be a terminal device to which the processing device 10 is connected.

[0048] 5, this embodiment is realized, for example, as software that implements an execution unit 154, a target registration unit 151, and a target monitoring unit 153 as an application that resides in a task tray. As shown in Fig. 5, when the software (inside the frame) that resides in the task tray is selected by the user, a menu M1 is displayed as a developer tool.

[0049] When automatic operation targets are set, the Dictionary screen of menu M1 displays a list of targets in area R1 on the left ((1) in Figure 6). The Dictionary screen displays location information in area R2 on the right ((2) in Figure 6), and aliases in area R3 on the right ((3) in Figure 6). The Dictionary screen also displays a button B1 for starting the registration process and a button B2 for updating the registration.

[0050] [Target registration] First, the target registration process will be described. As shown in Fig. 7, the user first selects the start button B1 on the Dictionary screen of the menu M1 to instruct the start of the registration process ((1) in Fig. 7).

[0051] Accordingly, the dictionary acquisition start menu M2-2 is displayed. When the user performs an operation such as clicking with the mouse on the target C1 of the automatic operation that the user wants to register, the clicked event is observed and the target is highlighted in a red frame W1 (Figure 8 (2)). In this example, the target C1 is a telephone number, so the dictionary acquisition start menu M2-2 lists the observed events (in this example, the event of focusing on the telephone number is listed). If the target C1 added in the red frame W1 is the target that the user wants to be the target of automatic operation, the user selects the add button B3 (Figure 8 (3)).

[0052] The object registration unit 151 designates the selected object C1 and generates location information for the designated object C1. As a result, the location information for the object C1 is automatically generated ((4) in FIG. 9) and displayed in the area R2 on the left side of the menu M1. Then, the object registration unit 151 sets the alias for this object C1 as "telephone number" ((5) in FIG. 9, area R3). The telephone number of the object C1 selected in the application information reception menu M2-1 in FIG. 8 is displayed in the telephone number field C2 of "customer information."

[0053] When the user selects the registration button B4 in area R4 ((6) in Figure 9), the object registration unit 151 adds a set of the telephone number of object C1, the alias "telephone number", and the location information of object C1 to the object list ((7) in Figure 10, area R1-2).

[0054] Here, Fig. 11 shows an example of a menu M2 for setting properties of an automatic operation target. As shown in area R21 of menu M2, the position information of target C1 is specified by object information (properties) ((1) of Fig. 11). Also, as shown in area R22 of menu M2, the position information of target C1 is specified by an index in each layer of the automatic operation target ((2) of Fig. 12).

[0055] As shown in FIG. 13, the user can also specify what objects to pass through at intermediate points in the hierarchy in the location information route setting area R2-2 of the menu M1 ((1) in FIG. 13). Specifying intermediate points can speed up the identification of objects. In this way, by combining the properties and indexes of objects as the target location information and allowing intermediate points to be flexibly set, location information can be flexibly set and identified even when the display position of the target object is not fixed at one position.

[0056] Then, as shown in Fig. 14, conditions can be registered on the Condition screen of menu M1. As shown in Fig. 14, on the Condition screen of menu M1, a list of conditions is displayed in an area R31 on the left ((1) in Fig. 14). On the Condition screen, a condition name field in which a condition name can be defined is displayed in an area R32 on the right ((2) in Fig. 14).

[0057] The Condition screen then displays a setting field in area R33 on the right side where an alias can be set ((3) in FIG. 14). The Condition screen then displays a setting field in area R34 on the right side where a condition can be set ((4) in FIG. 14). In the example of FIG. 14, "when a telephone number including 070 is set" is set as the condition. The Condition screen also displays an update button for registration. When the user sets each field and selects the update button, the object registration unit 151 sets the conditions for the object C1 registered on the Dictionary screen of menu M1 in accordance with the settings on the Condition screen.

[0058] [Monitoring of automated operation targets] Next, a description will be given of the operation of the target monitoring unit 153. For example, as shown in Fig. 15, a case will be described where "090" is entered in the phone number field (area R41) of the application information reception menu M4-1.

[0059] The target monitoring unit 153 constantly monitors the system screen and determines whether the condition is met (FIG. 15(2)). In this case, the condition "when a telephone number including 070 is set" set in FIG. 14 is not met (FIG. 15(1)). If the condition is not met, the target monitoring unit 153 displays the name of the condition in red, for example, on the Condition screen of menu M4-2 (area R42 in FIG. 15).

[0060] In contrast to this, as shown in Fig. 16, a case where "070" is entered in the telephone number field (area R43) of the application information reception menu M4-3 will be described as an example. The target monitoring unit 153 constantly monitors the system screen and determines whether the condition is met (Fig. 16(4)). In this case, the condition "when a telephone number including 070 is entered" set in Fig. 14 is met (Fig. 16(3)). Therefore, when the condition is met, the target monitoring unit 153 displays the condition name in green, for example, on the Condition screen of menu M4-4 (area R44 in Fig. 16).

[0061] [execution] Next, the operation of the execution unit 154 will be described. For example, as shown in console screen M5-2 in FIG. 17, the execution unit 154 interprets and processes the corresponding script ((1) in FIG. 17). FIG. 17 illustrates the Lua script language as an example. The script is written using the alias set in the object registration unit 151 ((2) in FIG. 17, areas R51 and R52).

[0062] The execution unit 154 replaces the alias with object information that is the target of automatic operation. Then, the execution unit 154 sequentially processes the script, and automatic operation of the system screen is executed ((3) in FIG. 17). As a result, "dummy@ntt.com" is automatically entered in the email address field C51 of the application information reception menu M5-1.

[0063] As shown in FIG. 18, the user can output (export) the target list in the area R11 by pulling down File in the menu M1 and selecting the Export column D1.

[0064] [Control Description] Next, a description will be given of an example of software that implements the control description part 152. In the menu M6 in Fig. 19, the target list is imported by selecting the import button B61 ((1) and area R61 in Fig. 19).

[0065] As shown in menu M7, the control description unit 152 describes the control of automatic operations for targets in the target list using low code and / or no code ((1) in FIG. 20). Note that FIG. 20 illustrates an example of block-type visual programming. The control description unit 152 sets the description content using the alias of the imported target list ((2) in FIG. 20 and areas R71 to R73). The control description unit 152 then converts it into a script that can be interpreted by the execution unit 154 ((3) in FIG. 21 and frame M75 in FIG. 21).

[0066] [Update target list when system screen changes] Next, a procedure for updating the target list 141 when the system screen changes will be described. Fig. 22 is a flowchart showing the procedure for updating the target list when the system screen changes.

[0067] As shown in FIG. 22, the object registration unit 151 performs an object check on the system screen to be checked (step S41), and if there is any object in the object list 141 for which location information is not found, the object is highlighted (step S42).

[0068] When a highlighted object is selected (step S43), the object registration unit 151 reacquires the location information of the highlighted object on the system screen (step S44), and updates the location information of this object in the object list 141 to the reacquired location information.

[0069] The object registration unit 151 determines whether or not the necessary corrections have been completed for the system screen to be checked (step S45). If the necessary corrections have not been completed for the system screen to be checked (step S45: No), the object registration unit 151 returns to step S43. If the necessary corrections have been completed for the system screen to be checked (step S45: Yes), the object list update process for this system screen is terminated.

[0070] 23 to 28 are diagrams showing examples of menu screens displayed on an automatic operation target device. An example will be described in which the system screen changes from menu M8 to menu M9 in Fig. 23. In menu M9, the location information of the telephone number has changed compared to menu M8.

[0071] In this case, the user clicks CheckAccess from the Group pull-down menu on the Dictionary screen of menu M10 in FIG. 24 ((1) in FIG. 24).

[0072] As a result, as shown in FIG. 25, in the information reception menu M11, a frame (for example, a red frame) is displayed around an object for which a change in location information has been found ((2) in FIG. 25).

[0073] As shown in menu M12 in Figure 26, if the location information of an object displayed on the system screen cannot be found in the object list, the missing object (in this case, a phone number) will be displayed, for example, in red ((3) in Figure 26).

[0074] The user selects the target in red as shown in Fig. 27 ((1) in Fig. 27). In this case, the target registration unit 151 updates only the location information of the target in red (in this case, the phone number) without changing the alias by reacquiring the location information of the phone number field on the system screen after the change ((2) in Fig. 27).

[0075] As shown in Fig. 28, when the target whose location information has been corrected is clicked with the mouse on the changed system screen ((2) in Fig. 28), processing related to automatic operation is executed. The target registration unit 151 updates only the location information of the target list for the same target (telephone number in this case) without correcting the alias.

[0076] In this way, when the system screen is changed, if there is any system screen after the change for which location information cannot be found in the target list, the target registration unit 151 re-acquires the location information of this target and updates the location information of this target in the target list to the re-acquired location information.

[0077] [Effects of the embodiment] Figure 29 is a diagram explaining conventional RPA. In existing RPA, the information on the target of automated operation and its control are described as a single entity in the form of scenarios and rules. For this reason, in conventional RPA, when a system screen that is the target of automated operation is modified due to an update to System 1 (for example, when a button label is changed from "Register" to "Submit") ((1) in Figure 29), it is necessary to modify all scenarios and rules related to the system screen in question ((2) in Figure 29).

[0078] Fig. 30 is a diagram illustrating a processing device 10 according to an embodiment. As shown in Fig. 29, the processing device 10 manages automatic operations by separating control from automatic operation targets. The processing device 10 aggregates information that is susceptible to environmental influences, such as system screens, into target information (aliases, monitoring conditions) and generates this information for each system ((1) in Fig. 30).

[0079] Therefore, in the processing device 10, even if the system screen that is the target of the dynamic operation is modified due to a system update ((2) in FIG. 30), the scope of modification is limited to the target information ((3) in FIG. 30). Since the processing device 10 does not require modifications related to the control of the system 1, the control description can be written using the modified target information.

[0080] Therefore, even if an environmental change such as a change in the system screen occurs, the processing device 10 only modifies the target information, and no modification is required for the control described using the target information ((4) in FIG. 30). In other words, since the processing device 10 limits the scope of modification to the target information only, it is possible to reduce the impact of environmental changes on the modification of scenarios and rules.

[0081] Furthermore, in the processing device 10, the position information of the automatic operation target registered by the target registration unit 151 is expressed hierarchically using object information or indexes, so the position information of the automatic operation target can be registered appropriately. Furthermore, in the processing device 10, a unique alias is defined for each piece of position information, so the position information of the automatic operation target can be managed appropriately.

[0082] Furthermore, in the processing device 10, the control description unit 152 uses aliases to set conditions including existence conditions, value conditions, and / or timing conditions, and defines a unique condition name for each condition, so that the conditions for automatic operations can be managed appropriately.

[0083] In the processing device 10, the control description unit 152 describes control using either of the following execution methods: immediate execution, or execution when an execution condition set by combining condition names is met. Therefore, compared to the conventional method in which all scenarios and rules are created as a single unit, the processing device 10 does not need to take all scenarios and rules into consideration, and can easily describe procedures such as "waiting until a state in which a scenario or rule can be executed by monitoring the screen" such as waiting until the system screen transitions to the next screen and then performing automatic operation.

[0084] [System configuration of the embodiment] Each component of the processing device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the processing device 10 is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0085] Furthermore, all or any part of the processes performed by the processing device 10 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Furthermore, each process performed by the processing device 10 may be realized as hardware using wired logic.

[0086] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0087] [program] 31 is a diagram showing an example of a computer in which a program is executed to implement the processing device 10. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0088] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0089] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the processing device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the processing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0090] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0091] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0092] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention. [Explanation of symbols]

[0093] 10 Processing equipment 11 Input section 12 Output section 13 Communications Department 14 Storage section 15 Control Unit 141 Target List 142 Conditions List 151 Subject Registration Department 152 Control Description Section 153 Target Monitoring Department 154 Executive Department

Claims

1. a registration unit that designates an automatic operation target displayed on a system screen, and registers and manages location information and an alias of the designated automatic operation target as target information; a description unit that uses the target information registered by the registration unit to describe control of automatic operation using low code and / or no code, describes execution conditions if any, and converts them into a predetermined script; a monitoring unit that monitors information about an application on a desktop, determines whether or not the execution condition set by the description unit is satisfied, and outputs a corresponding script when it is determined that the execution condition is satisfied; an execution unit that interprets the script converted by the description unit, replaces the alias with object information to be automatically operated, sequentially processes the script, and automatically operates the system screen; and The processing device characterized in that the position information of the automatic operation target registered by the registration unit is expressed hierarchically using object information or indexes, and objects to be passed through at relay points in the hierarchy can be set.

2. The processing device described in Claim 1, characterized in that the registration unit defines a unique alias for each location information.

3. A registration unit that specifies an automatic operation target displayed on a system screen and registers and manages the location information and alias of the specified automatic operation target as target information; a description unit that describes control of automatic operation using low-code and / or no-code by using the target information registered by the registration unit, describes execution conditions if any, converts the execution conditions into a predetermined script, and outputs the script; a monitoring unit that determines whether or not an execution condition exists in the output of the description unit, and outputs the script output from the description unit if the execution condition does not exist, and monitors information about the application on the desktop if the execution condition exists, determines whether or not the execution condition set by the description unit is satisfied, and outputs the corresponding script if it is determined that the execution condition is satisfied; an execution unit that interprets the script output from the monitoring unit, replaces the alias with object information that is an object to be automatically operated, sequentially processes the script, and automatically operates the system screen; A processing device comprising:

4. The position information of the automatic operation target registered by the registration unit is expressed hierarchically by object information or indexes, 4. The processing device according to claim 3, wherein the registration unit defines a unique alias for each piece of location information.

5. 4. The processing device according to claim 1, wherein the registration unit uses an alias to set conditions including an existence condition, a value condition, and / or a timing condition, and defines a unique condition name for each condition.

6. 4. The processing device according to claim 1, wherein the description section describes control using one of an immediate execution method and an execution method in which the execution is performed when an execution condition set by combining condition names is satisfied.

7. The processing device described in claim 1 or 3, characterized in that when the system screen is changed, if there is a first object for which location information cannot be found in the object information for the changed system screen, the registration unit re-acquires the location information of the first object and updates the location information of the first object in the object information to the re-acquired location information.

8. A processing method executed by a processing device, a registration step of designating an automatic operation target displayed on a system screen, and registering and managing the location information and alias of the designated automatic operation target as target information; a description process of describing control of automatic operation using low-code and / or no-code by using the target information registered in the registration process, describing the execution conditions if any, and converting them into a predetermined script; a monitoring step of monitoring information about the application on the desktop, determining whether the execution conditions set in the writing step are satisfied, and outputting a corresponding script when it is determined that the execution conditions are satisfied; an execution step of interpreting the script converted in the description step, replacing the alias with object information to be automatically operated, sequentially processing the script, and automatically operating the system screen; Including, A processing method characterized in that the location information of the automatic operation target registered in the registration process is expressed hierarchically using object information or indexes, and objects to be passed through at relay points in the hierarchy can be set.

9. A processing method executed by a processing device, comprising: a registration step of designating an automatic operation target displayed on a system screen, and registering and managing the location information and alias of the designated automatic operation target as target information; a description process of describing control of automatic operation using low-code and / or no-code by using the target information registered in the registration process, describing execution conditions if any, converting the execution conditions into a predetermined script, and outputting the script; a monitoring step of determining whether or not an execution condition exists in the output in the description step, and outputting the script output in the description step if the execution condition does not exist, and monitoring information on the application on the desktop if the execution condition exists, determining whether or not the execution condition set in the description step is satisfied, and outputting the corresponding script if it is determined that the execution condition is satisfied; an execution step of interpreting the script output in the monitoring step, replacing the alias with object information to be automatically operated, sequentially processing the script, and automatically operating the system screen; A processing method comprising:

10. a registration step of specifying an automatic operation target displayed on a system screen, and registering and managing location information and an alias of the specified automatic operation target as target information; a description step of describing control of automatic operation using low-code and / or no-code by using the target information registered in the registration step, describing the execution conditions if any, and converting them into a predetermined script; a monitoring step of monitoring information of the application on the desktop, determining whether or not the execution condition set in the writing step is satisfied, and outputting a corresponding script when it is determined that the execution condition is satisfied; an execution step of interpreting the script converted in the description step, replacing the alias with object information to be automatically operated, sequentially processing the script, and automatically operating the system screen; on the computer, A processing program in which the position information of the automatic operation target registered in the registration step is expressed hierarchically by object information or indexes, and objects to be passed through at relay points in the hierarchy can be set.

11. A registration step of specifying an automatic operation target displayed on a system screen, and registering and managing location information and an alias of the specified automatic operation target as target information; a description step of describing control of automatic operation using low-code and / or no-code by using the target information registered in the registration step, describing the execution conditions if any, converting the execution conditions into a predetermined script, and outputting the script; a monitoring step of determining whether or not an execution condition exists in the output in the description step, and outputting the script output in the description step if the execution condition does not exist, and monitoring information on the application on the desktop if the execution condition exists, determining whether or not the execution condition set in the description step is satisfied, and outputting the corresponding script if it is determined that the execution condition is satisfied; an execution step of interpreting the script output in the monitoring step, replacing the alias with object information to be automatically operated, sequentially processing the script, and automatically operating the system screen; A processing program that causes a computer to execute the above.

Images