How to create RPA script data, how to execute RPA script data, terminal device, image processing device, RPA script data, program

Abstract

To provide a creation method of RPA script data for properly providing instructions on a place to be operated to an RPA without scanning a window when executing the RPA script data, a program and an image processing apparatus.SOLUTION: A remote panel application of a terminal device connected to an image processing apparatus through a TCP connection generates, in accordance with an operation by an operator in a state of window-display, RPA script data 2600 including scripts 3310, 3320 for use in substitution of coordinates in a window of a place of an object of an operation to be performed by an RPA to variables as immediate values, scripts 3410, 3420 for use in substitution of horizontal pixels and vertical pixels of the window to variables as immediate values, coordinates in the window set as these variables, a horizontal pixel E_WINwidth of the window, and a vertical pixel E_WINheight. With using these, the remote panel application generates a script 3510 for re-calculating coordinates of the place of the object of the operation.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Relates to a creation method and an execution method for creating RPA script data to be executed by a terminal device when the terminal device performs an operation by RPA.

Background Art

[0002] A conventional method for creating RPA script data is so-called RPA programming, in which, for a desktop screen, the content of an operation by RPA is described according to the grammar of a programming language such as Python (registered trademark), VBScript (registered trademark), VB.NET (registered trademark), GAS (registered trademark), C# (registered trademark). RPA (Robotic Process Automation) is a technology that substitutes a program for an operation on an operation screen that has conventionally been performed manually by a human and automates it. For the above-mentioned automation, various functions such as an automatic mouse control function, an automatic keyboard input function, an automatic screen shot creation function, an image recognition function, a message box function, etc. can be used. By combining these, in RPA, various desk jobs at a level comparable to those performed by office workers can be realized.

[0003] The position of an operation target to be operated by RPA is expressed in the coordinate system of a so-called desktop screen, that is, the screen displayed at the topmost layer in an operating system. On the other hand, in recent years, there has been an increasing need to create RPA script data for a remote panel system. Patent Document 1 discloses a remote panel system that operates an image processing apparatus using a terminal device. The terminal device displays a bitmap corresponding to the operation screen of the image processing apparatus, accepts from an operator the designation of the position of an operation target to be operated, and causes the image processing apparatus to execute processing corresponding to the designation. If RPA script data is created so as to automatically designate the position of such an operation target to be operated, the creator of the RPA script data can cause RPA to execute the operation of the image processing apparatus. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2016-144097 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Incidentally, in conventional remote panel systems, the operation screen displayed on the terminal device can be arbitrarily resized by the user. In contrast, conventional RPA script data creation methods specify the position of the target to be operated on using the coordinate system of the desktop screen.

[0006] Thus, if the position of the object to be manipulated is specified using the coordinate system of the desktop screen, there is a problem in that if the size of the operation screen is changed, it becomes impossible to properly specify the position of the object to be manipulated after the size has been changed. The first objective of this disclosure is to provide a method for creating RPA script data that can appropriately indicate the operation position after the size of the operation screen has been changed.

[0007] The second objective is to provide a method for executing RPA script data that can appropriately calculate the coordinates of the location to be operated by RPA using the RPA script data created by the above method.

[0008] The third objective is to provide a terminal device capable of performing the above-described creation method.

[0009] The fourth objective is to provide an image processing device that can calculate the coordinates of the location to be operated by RPA using the RPA script data created by the above method.

[0010] The fifth objective is to provide a data structure for RPA script data that can be used in the creation method described above.

[0011] The sixth objective is to provide a program that allows a computer to calculate the coordinates of the location to be operated by RPA using the RPA script data created by the method described above. [Means for solving the problem]

[0012] The first objective described above is achieved by a method for creating RPA script data for an RPA in a terminal device that displays an operation screen for an image processing device on its own terminal and accepts operations by RPA, the method comprising: a first step of detecting the position of an operation target displayed on the operation screen and selected by the operator; and a second step of normalizing the detected position of the operation target based on the size of the operation screen to generate position information indicating the normalized position, and describing the position information in the RPA script data.

[0013] The terminal device may display the operation screen in a window, and the normalization of the position of the target to be operated in the second step may be performed using the display size when the operation screen is displayed in a window and the in-window coordinates of the position of the target to be operated.

[0014] The second step may involve generating a script to obtain the size of the window display, and a calculation script to calculate the in-window coordinates indicating the position of the target using the obtained window display size and the normalized position of the target indicated in the position information, and including these in the RPA script data.

[0015] The RPA script data includes resolution information indicating the screen resolution of the terminal device when the position of the object to be operated on is detected, and the calculation script generated in the second step may use the ratio between the screen resolution indicated in the resolution information of the RPA script data and the screen resolution of the terminal device when the RPA script data is executed in order to calculate the in-window coordinates.

[0016] The second step may involve generating a calculation script that calculates the coordinates within the operation screen of the image processing device using the screen resolution of the image processing device and the normalized position of the object being operated on, as indicated in the position information, and including this script in the RPA script data.

[0017] The second objective described above is achieved by an RPA script execution method for a terminal device that displays an operation screen for an image processing device on its own terminal and accepts operations by RPA, the method comprising: a first step of calculating the coordinates of the location to be operated by RPA based on normalized location information using the size of the operation screen; and a second step of instructing the image processing device to perform the RPA operation at the calculated coordinates and perform the corresponding processing.

[0018] The terminal device may display the operation screen of the image processing device in a window, and the calculation of coordinates in the first step may be performed by obtaining the size of the window display and calculating the coordinates within the window using the obtained size and the normalized position of the operation target indicated in the position information.

[0019] The RPA script data includes resolution information indicating the screen resolution of the terminal device when the location of the object to be operated on is detected, 1 The step may use the ratio between the screen resolution indicated in the resolution information of the RPA script data and the screen resolution of the terminal device when executing the RPA script data to calculate the in-window coordinates of the position of the object to be operated on.

[0020] The aforementioned 1 The calculation of coordinates by step may be performed by calculating the in-screen coordinates of the image processing device using the screen resolution of the image processing device and the normalized position of the object being operated on, as indicated in the position information.

[0021] The calculation of the coordinates in the first step may be performed by either the terminal device or the image processing device.

[0022] Furthermore, the process includes a first substep in which connectable image processing devices are presented to the user and the user selects which image processing device to use for processing, and a second substep in which, if an image processing device is selected, the screen resolution of the image processing device is obtained. The calculation of the coordinates in the first step may be performed by calculating the in-screen coordinates of the image processing device using the ratio of the position information of the RPA script data to the obtained screen resolution of the image processing device.

[0023] The RPA script data consists of multiple operation scripts, and when instructing the image processing device to perform an operation based on the RPA script data, it has a third step of determining whether the processing of the image processing device has stopped, and a fourth step of notifying the administrator of the error location in any of the operation scripts if the processing of the image processing device has stopped, and the location information may be included in any of the operation scripts.

[0024] The fourth step includes a fifth step in which the RPA script data is recreated after notification, and the fourth step may include a substep in which the administrator specifies the timing of the operation, and a script may be inserted before and after the operation script notified as an error location to adjust the timing of issuing the operation script according to the specified timing.

[0025] The third objective described above is achieved by a terminal device comprising: a receiving means for receiving a bitmap representing the operation screen of an image forming apparatus from an image processing apparatus; a display means for writing the received bitmap to VRAM and displaying it; a receiving means for receiving a specification from the operator of the position to be operated on from the bitmap written to VRAM when registering an operation to be performed by RPA; and a generating means for obtaining position information that normalizes the specified position according to the size of the displayed bitmap and generating RPA script data that contains the position information.

[0026] The system may also include a calculation means for calculating the coordinates of the target location for the RPA based on the size of the currently displayed bitmap and normalized position information when executing a registered RPA, and an execution means for performing an operation by the RPA at the calculated coordinates and issuing an instruction to the image processing device to operate the target at the corresponding location on the image processing device's operation screen.

[0027] The fourth objective described above is achieved by an image processing apparatus comprising: a transmission means for transmitting a bitmap representing the operation screen of an image forming apparatus to a terminal device; a receiving means for receiving normalized position information from the terminal device when performing an operation with RPA; and a calculation means for calculating the coordinates of the position to be operated on by RPA based on the size of the bitmap displayed on the terminal device and the normalized position information, wherein the terminal device performs an operation by RPA using the coordinates calculated by the calculation means.

[0029] The above 5 The objective is achieved by a program that causes a computer displaying an operation screen for an image processing device to perform an operation using RPA, the program comprising: a first code that causes the computer to calculate the coordinates of the target of the operation on the displayed operation screen based on position information normalized using the size of the operation screen; and a second code that causes the computer to perform an operation using RPA at the calculated coordinates.

[0030] Here, normalization refers to a transformation method that converts the coordinates of a specified position within the operation screen of an image processing device so that they fall within a certain numerical range, regardless of how they are displayed on the terminal device, based on the size of the operation screen.

[0031] There are various methods for such conversions. For example, 1) A conversion method that involves calculating the X and Y coordinates of a specified position within the remote panel displayed in the window, and then dividing them by the horizontal and vertical pixel counts of the window. 2) A conversion method in which the X and Y coordinates of a specified position within the windowed remote panel are expressed as an internal division ratio with respect to the long and short sides of the window. 3) A conversion method that treats lines passing through specified positions within the window (display frame) and remote panel as a single vector graphic and represents them as vector data. 4) A transformation method that converts the X and Y coordinates of a specified position within the operation screen of an image processing device to another coordinate system, such as a polar coordinate system. 5) A transformation method using a mapping method can be used to map the X and Y coordinates indicating the touch position within the window to a predetermined numerical range. Explaining all forms of normalization would be complicated, so in the following embodiments, the transformation method described in 1) above is used to represent the position specified by the operator within the operation screen.

[0032] The positional information indicating the normalized position is the calculation formula representing the above conversion method, or the numerical value obtained from the calculation of the above conversion method, and can be created, for example, using various notations of RPA programming languages.

[0033] For example, this can be represented by a script that performs the calculation corresponding to the above conversion method, an immediate value set in a variable of the RPA script data, a constant area that stores constants that should be set in the variable of the RPA script data, an array variable, etc. Since explaining all forms of this location information would be complicated, the following embodiments will describe an example of representing the conversion method in 1) using a script and an immediate value. [Effects of the Invention]

[0034] The location information normalizes the detected location of the target to be operated on based on the size of the operation screen at the time the location was detected. Therefore, if the size of the operation screen is changed, the RPA can appropriately indicate the location where it should operate after the size change. [Brief explanation of the drawing]

[0035] [Figure 1] This shows the configuration of the RPA system. [Figure 2] The hardware configuration of terminal device 2000 is shown. [Figure 3] Figure 3(a) shows the software stacks of the programs installed on HDD2005 of terminal device 2000 and HDD1005 of image processing device 1000. Figure 3(b) shows the protocol stack for communication between terminal device 2000 and image processing device 1000. [Figure 4] This shows an example of the windows assigned to each application by a multi-window system. [Figure 5] (a) Shows the structure of RPA script data 2600. (b) Shows an example of a script that normalizes the in-window coordinates of the touched position and recalculates those in-window coordinates. [Figure 6] Desktop screen 4001 shows the desktop screen used when creating RPA script data, and desktop screen 4002 shows the desktop screen used when executing RPA script data. [Figure 7] This is a flowchart showing the procedure for creating RPA script data using RPA Tool 2500. [Figure 8] This is a flowchart showing the processing procedure for the Remote Panel Application 2400. [Figure 9] (a) shows an example of the remote panel displayed on LCD2001 in step S203. (b) is the screen displayed on the RPA terminal when button 4160, which calls up the settings menu, is touched. [Figure 10] This is a flowchart showing the procedure for executing RPA script data. [Figure 11] This is a flowchart showing the processing procedure of the RPA tool 2500 according to the second embodiment. [Figure 12] This figure shows an example of a screen for specifying the wait time. [Figure 13] By identifying the wait time, we can determine the action script that is inserted immediately before the point where the error occurs. [Figure 14] This flowchart shows the processing procedure of the RPA tool 2500 according to the second embodiment. [Figure 15] This flowchart shows the processing steps of the RPA tool 2500 for the modified example. [Modes for carrying out the invention]

[0036] [1] System configuration Hereinafter, embodiments of the image processing apparatus and terminal device according to this disclosure will be described with reference to the drawings. Figure 1 shows the configuration of the RPA system according to this embodiment. As shown in this figure, the RPA system consists of image processing apparatuses 1000, 1010, and 1020, and a terminal device 2000. RPA (Robotic Process Automation) is a program that automates operations targeting the desktop screen. In the RPA system, data indicating a series of procedures to be executed by RPA (RPA script data) is created by the terminal device 2000, stored in the storage of the terminal device 2000, and made available for use.

[0037] Image processing unit 1000 is an MFP (Multi-function Peripheral) capable of forming images on A3 paper, image processing unit 1010 is an SFP (Single-function Peripheral) capable of forming images on A4 paper, and image processing unit 1020 is an AIO (All In One) device.

[0038] The touch panel display 1001 of the image processing device 1000 is, for example, 10 inches, while the touch panel displays 1011 and 1021 of the image processing devices 1010 and 1020 are, for example, 8 inches, and the touch panel sizes differ from one another. Furthermore, the screen resolutions are different.

[0039] In this embodiment, the screen resolution of the touch panels 1010, 1020, and 1030 of each image processing device is expressed as a pair of horizontal and vertical pixels for the entire screen. This is to simplify the calculation of window coordinates, which will be described later. The following describes a case where the image processing device 1000 is operated using the terminal device 2000.

[0040] The terminal device 2000 is a device used by a user operating the image processing device 1000 together with the image processing device 1000. The terminal device 2000 displays the same screen (remote panel) 2001D as the screen 1001D displayed on the touch panel display 1001 of the image processing device 1000 on the LCD 2001, receives operations from the user on the remote panel, and commands the image processing device 1000 to perform processing according to those operations.

[0041] [2] Hardware configuration of terminal device 2000 and image processing device 1000 Figure 2 shows the hardware configuration of terminal device 2000.

[0042] As shown in this figure, the terminal device 2000 includes an LCD 2001 that provides an interactive operating environment to the user, a mouse 2002 which is an example of a pointing device, a keyboard 2003, a network interface 2004, an HDD 2005 on which various programs are installed, a RAM 2006 on which necessary programs from those installed on the HDD 2005 are loaded, a CPU 2007 that executes the programs loaded into RAM, a flash ROM 2008 for saving necessary settings for program execution, and peripheral I / O 2009 for receiving input operations to the mouse 2002 and keyboard 2003.

[0043] Network I / F2004 is a communication module that performs LAN and WAN connections. The communication module includes short-range communication modules such as Bluetooth®, Wi-Fi, EnOcean, and low-power radio (Wi-SUN), and performs the transmission and reception of operation information and image data wirelessly. RAM2006 has VRAM, and pixel data representing the pixels for one screen displayed by LCD2001 is stored here.

[0044] The image processing device 1000 has the same hardware configuration as the terminal device 2000 (in addition to the touch panel display 1001, it includes the network I / F 1004, HDD 1005, RAM 1006, CPU 1007, and ROM 1008 shown in Figure 3(a)), and the necessary programs are installed on the HDD. Operations on the touch panel display 1001 are performed using multiple page data stored in the HDD 1005. Each of the multiple page data has a hierarchical menu structure and includes a tag that defines the target of operation. Each tag indicates the file path (consisting of directory name and file name) of the file storing the graphics data representing the target of operation, coordinate information indicating the location coordinates of the target of operation, and a script to be executed when the operation on the target of operation is confirmed. Some scripts specify links to other page data, and others specify setting processes related to image formation.

[0045] In contrast, the operation screen displayed on the LCD2001 is a bitmap representing the same operation screen as the touch panel display 1001, and the content of the operation screen is represented by an additive mixing of red pixels (R), green pixels (G), and blue pixels (B). Because it is a bitmap, it is not possible to change the state of GUI components or switch pages. Therefore, in order to instruct the image processing device 1000 to control the system, a click operation must be accepted on the remote panel (operation screen for the operation screen) bitmap displayed on the LCD2001, operation information indicating the coordinates of the clicked position must be sent to the image processing device 1000, and the image processing device 1000 must perform the same operation as if the touch panel display 1001 had been touched at those coordinates.

[0046] The programs stored in HDD2005 include a program that demonstrates a series of operations performed by RPA. This program consists of code embedded in the page data (in this embodiment, the RPA script). The page data in which the RPA script is embedded is called the RPA script data.

[0047] [3] Software stack Figure 3(a) shows the software stacks of the programs installed on HDD2005 of terminal device 2000 and HDD1005 of image processing device 1000. The software stack of terminal device 2000 has a control hierarchy in which the operating system (OS) 2200 is installed on the hardware 2100 of terminal device 2000, and the applications 2300 are installed on top of that.

[0048] The software stack of the image processing device 1000, like that of the terminal device 2000, has a control hierarchy in which the OS 1200 is installed on the hardware 1100 of the image processing device 1000, and the application 1300 is installed on top of that.

[0049] The OS 2200 of terminal device 2000 and the OS 1200 of image processing device 1000 perform data communication via a protocol stack through network I / F 2004 and 1004. Applications 2300 and 1300 establish a TCP connection 2104 using TCP protocols 1134 and 2034, IP protocols 1124 and 2024, and short-range communication protocols 1114 and 2014 as shown in Figure 3(b), and perform sending and receiving HTTP messages through the TCP connection 2104.

[0050] TCP connection 2104 is established when terminal device 2000 acts as the host, application 2300 sends a connection request specifying the TCP port number of the other application 1300 and its own port number, and application 1300 confirms that the port requested by the connecting side is listening for a connection and sends a response. The application 2300 running on the terminal device 2000 and the application 1300 running on the image processing device 1000 use the TCP connection 2104, using the port number specified by the host at the time of connection, to send and receive image data and operation information.

[0051] (3-2) Software stack of the image processing device 1000 Application 1300 includes an MFP panel application 1400 and a processing execution application 1500. Additionally, OS 1200 includes an operation system control unit 1600 as a component for remote operation mode.

[0052] The panel application 1400 reads multiple page data stored in the HDD 1005 and displays the operation screen. It also transitions the operation screen in response to operation input from the touch panel display 1001. Operation input from the touch panel display 1001 is operation information indicating which location on the touch panel display 1001 was operated on, and it shows the coordinates in the panel coordinate system of the touch panel display 1001 and the type of operation (touch, drag) performed on those coordinates. By confirming the operation target at the coordinates in the page data corresponding to the current operation screen, the script corresponding to that operation target is executed. If the script corresponding to the confirmed operation target is a link script that specifies a link to other page data, the operation screen switches to the operation screen corresponding to the other page data. If the script corresponding to the confirmed operation target is a panel setting or job setting related to image formation, those panel settings and job settings are executed.

[0053] The processing execution application 1500 executes panel settings according to the operation input made to the MFP panel application 1400, and sets up and executes jobs and image processing devices 1000 when they are installed at the business site.

[0054] The operation system control unit 1600 switches the operation system in remote operation mode. Specifically, in remote operation mode, it enables operation input via the TCP connection 2104 with the terminal device 2000 instead of operation input from the panel display 1001, and outputs the bitmap of the operation screen drawn using page data to the terminal device 2000 via the TCP connection. By enabling operation input from the TCP connection 2104 instead of operation input from the touch panel display 1001, the operation screen on the image processing device 1000 will be switched by operation input from the terminal device 2000.

[0055] (3-3) Software stack of terminal device 2000 OS2200 includes kernel 2210 and multi-window system 2220. Kernel 2210 allocates multiple sub-regions of RAM2006 to each application, launches multiple applications, and monitors the operation of each application. The multi-window system 2220 provides users with an interactive operating environment by assigning a window to each of multiple applications for display. A window is a unit of display area assigned to individual applications by the multi-window system 2220. As shown in Figure 4, the application screen 3002 (a screen showing the content drawn by the application) can be displayed within the display frame 3001. Since the entire application screen is displayed to fill the display frame, the display frame 3001 becomes the entire operation screen (remote panel) displayed on the terminal device 2000.

[0056] Various GUI components are arranged around the display frame 3001. These GUI components include a horizontal scroll bar 3003 for horizontal scrolling, a vertical scroll bar 3004 for vertical scrolling, a menu bar 3005 for various menu pull-downs, a title bar 3006 showing the application name, a status bar 3007 showing the application status, a minimize button 3008, a full-screen button 3009, and a close button 3010. The user can resize the application screen by dragging the window border 3011. They can also move it to a desired position. When a window becomes active and accepts operations on these GUI components and the application screen, that window becomes active. The multi-window system 2220 displays the operation screen window corresponding to the image processing device 1000 by writing the bitmap received via the TCP connection 2104 to the VRAM in RAM 2006.

[0057] Typical examples of applications 2300 managed by kernel 2210 include the remote panel application 2400 and the RPA tool 2500 shown in Figure 3(a). The remote panel application 2400 is one of the applications 2300 and displays a bitmap output from the image processing device 1000 in a window. Because it is a bitmap, clicking on the window border will resize it, and the size of the bitmap will change accordingly, altering the horizontal and vertical pixel counts.

[0058] The remote panel application 2400 outputs operation information to the panel application 1400 when operation input from the terminal device 2000 is enabled in the image processing device 1000, causing the panel application 1400 to perform screen transitions in accordance with the operation input from the terminal device 2000.

[0059] The RPA tool 2500 can register operations to be performed by RPA and execute registered RPA operations. This registration is done by generating RPA script data 2600, which shows a series of operations to be performed by RPA, and writing it to HDD2005. The execution of registered RPA operations is done by reading the RPA script data 2600 showing the registered operations from HDD2005 and having the RPA tool 2500 execute it. In creating the RPA script data 2600, the remote panel, which is a bitmap of the operation screen of the image processing device 1000, is displayed on the terminal device, user operations on the remote panel are accepted, the position where input such as touch is made by the user operation is detected pixel by pixel to obtain coordinate information, and a script is created from the detected coordinate information. The RPA script data 2600 is obtained by repeating this script creation from coordinate information. On the other hand, when executing RPA script data, the coordinate information of each script that constitutes each step of the RPA script data 2600 is sent to the remote panel application 2400, and the remote panel application is instructed to perform remote operations based on that coordinate information.

[0060] [4] Structure of RPA script data Through this calculation, the position to be manipulated by RPA is represented within the range of 0 to 1, with the top-left corner of the window being (0,0) and the bottom-right corner being (1,1). As mentioned above, since bitmaps do not contain information elements such as tags for the object to be manipulated, the coordinates in the RPA script data simply specify one of the pixels in the bitmap.

[0061] Figure 5(a) shows the data structure of the RPA script data 2600. As shown in this figure, the RPA script data 2600 consists of a field 2601 in which the script representing the first step is written, a field 2602 in which the script representing the second step is written, a script 2603 in which the script representing the third step is written, and so on, up to a field 2604 in which the script representing the mth step is written.

[0062] (4-1) RPA functions that can be used to describe RPA script data When executing the above RPA script data using the RPA tool 2500, you can use PYAutoGUI, for example, to describe these steps. PYAutoGUI is a program library that provides functions for RPA. Among the RPA functions provided by PYAutoGUI, the following define operations that target the coordinate system of the desktop screen. ·click(x,y,button,clicks=right / left,interval) At the coordinates (x,y) specified by the argument, the operation event specified by the argument clicks is generated for the time interval specified by the argument interval.

[0063] The coordinates handled by the above function are expressed in the desktop screen coordinate system, with the top left of the desktop screens 4001 and 4002 in the terminal device 2000 shown in Figure 6 as the origin, the right direction of the screen as the X-axis, and the bottom direction of the screen as the Y-axis. On the other hand, the remote panel is displayed inside the window display frame 3001 by the multi-window system 2220. The location within the window is expressed in the in-window coordinate system, with the top left of the application area (display frame 3001) shown in Figure 4 as the origin, the right direction of the screen as the X-axis, and the bottom direction of the screen as the Y-axis.

[0064] (4-2) Differences in window display size The window display size may differ between the time the RPA script data is created and the time the RPA script data is executed. In this embodiment, the window display size is represented by a pair of horizontal and vertical pixels of the application screen 3002 within the window's display frame.

[0065] The desktop screen 4001 is the screen for creating RPA script data, and the desktop screen 4002 is the screen for executing RPA script data. The desktop screen 4001 at the time of creation has a window 4100 with a remote panel arranged. The window 4100 is set to be rather large, and its horizontal pixel count and vertical pixel count are the variables M_WIN width , M_WIN height which are set to.

[0066] Also, buttons 4110, 4120, 4130, 4140, 4150, 4160 for operating the image processing apparatus 1000 are arranged. Among the remote panel, the location operated by the user (in FIG. 6, the central position of the button 4160) is the variables M_BTN x , M_BTN y which are specified by.

[0067] On the other hand, in the desktop screen 4002 at the time of execution, since a plurality of windows are displayed simultaneously, the size of the window 4200 with the remote panel arranged and made active is smaller. The horizontal pixel count and vertical pixel count of the window 4200 that becomes active when executing such RPA script data are the variables E_WIN width , E_WIN height which are set to.

[0068] When comparing with the time of creation, when the window size at the time of execution is small, the positional relationship of the buttons 4110, 4120, 4130, 4140, 4150, 4160 is the same or almost the same regardless of the size, but since the distance from the upper left point of the window to each button changes, the area they occupy also changes. In this remote panel at the time of execution, the location to be operated is the variables E_BTN x , E_BTN y which are specified by. A major feature of the RPA script data 2600 is that when registering RPA, the variables E_BTN x , E_BTN yThe key feature is that the location to be operated on, as explicitly indicated, is normalized according to the size of the operation screen displayed in the window. An example of a script that normalizes and displays the coordinates of the touched position is shown in Figure 5(b).

[0069] In Figure 5(b), the script 3110 listed at the very bottom (corresponding to the first code of the program invention) is the variable E_BTN shown in Figure 6. x , E_BTN y The argument is used to instruct the RPA tool 2500 to generate a click event at the location specified by these arguments.

[0070] Scripts 3210 and 3220 use the variable E_BTN, which is an argument to script 3110. x , E_BTN y This defines the formula for recalculating and setting the value. Script 3210 (corresponding to the second code of the program invention) calculates the difference between the variables M_BTNx and M_WINx, and then calculates M_WIN width And, E_WIN width The RPA tool 2500 is instructed to perform a recalculation, multiplying the result by the ratio and setting the result to the variable E_BTNx.

[0071] Script 3220 (like Script 3210, corresponds to the second code of the program invention) sets the difference between the variables M_BTNy and M_WINy to M_WIN height And, E_WIN height The RPA tool 2500 is instructed to perform a calculation that multiplies the ratio of and sets the result to the variable E_BTNy. Among the operands of scripts 3210 and 3220, E_WIN width E_WIN height This value is determined when the RPA script data is executed.

[0072] The remaining variables have their values ​​determined when the RPA script data is created. Scripts 3310 and 3320 use the variables M_BTNx and M_BTN, which are used in scripts 3210 and 3220. ySet immediate values. Here, immediate values ​​refer to the X and Y coordinates within the window of the location where the operation was performed when the RPA script data was created.

[0073] Scripts 3410 and 3420 use the variable M_WIN, which is used in the calculation formulas of scripts 3210 and 3220. width M_WIN height Set immediate values ​​here. These immediate values ​​are the horizontal and vertical pixel counts of the window containing the remote panel that was manipulated when the RPA script data was created.

[0074] Scripts 3430 and 3440 use the variable M_WIN, which is used in the calculation formulas of scripts 3210 and 3220. x M_WIN y Set immediate values ​​here. These immediate values ​​are the top-left X and top-left Y coordinates of the window containing the remote panel that was manipulated when the RPA script data was created.

[0075] Scripts 3511-3513 use a function to control the multi-window system 2220, and the variable E_WIN used in scripts 3210 and 3220 is used in this way. width E_WIN height Set it.

[0076] Script 3511 is a function call script that takes the variable hWnd and the structure variable rw as arguments and calls the GetWindowRect function. The GetWindowRect function is a function in Microsoft's Windows (registered trademark) that provides information about a window, and when this function is called, the return value is set to the member variables rw.right, rw.left, rw.bottom, and rw.top of the structure variable rw.

[0077] rw.right is set to the X coordinate of the bottom right corner of the window, rw.left to the X coordinate of the top left corner of the window, rw.bottom to the Y coordinate of the bottom right corner of the window, and rw.top to the Y coordinate of the top left corner of the window.

[0078] Scripts 3512 and 3513 calculate the horizontal and vertical pixel counts of the window using these member variables. Specifically, script 3512 calculates the difference between the member variables rw.right and rw.left using E_WIN. width The operation script 3513 then calculates the difference between member variables rw.top and rw.bottom, E_WIN height In this embodiment, the horizontal and vertical pixel counts of the window were calculated using functions provided by Microsoft's Windows®, but this is not limited to that. The horizontal and vertical pixel counts may also be calculated using functions provided by the multi-window systems of other operating systems.

[0079] In the RPA script data, before script 3110, which causes the RPA tool 2500 to move the cursor, there are scripts 3210 and 3220 that calculate the argument variables, and scripts 3310, 3320, 3410, 3420, 3430, and 3440 that set immediate values. Scripts 3511, 3512, and 3513 that calculate the horizontal and vertical pixels of the active window are also included. When the RPA script data is created, the coordinates of the touched position within the window are normalized using the window's horizontal and vertical pixels. Therefore, when the RPA script data is executed, E_WIN height E_WIN width By determining the value of [this parameter], it is possible to calculate the coordinates that should be instructed to the RPA tool 2500.

[0080] [5] Processing procedures for RPA tool 2500 and remote panel application 2400 (5-1) Opening a window using RPA tool 2500 The RPA tool 2500 consists of a sequence of codes that instructs CPU2007 to execute the processing steps shown in the flowcharts in Figures 7 and 10. The remote panel application 2400 consists of a sequence of codes that instructs CPU2007 to execute the processing steps shown in the flowchart in Figure 8. Figure 7 is a flowchart showing the processing steps of the RPA tool 2500, and Figure 8 is a flowchart showing the processing steps of the remote panel application 2400. In Figure 7, i is a variable that instructs each of the multiple steps to be instructed in the RPA script data, and takes values ​​from 1 to m.

[0081] In step S101 of Figure 7, a window including the RPA menu bar and title bar is opened, and the remote panel application 2400 is launched.

[0082] (5-2) Display and update of the remote panel using the Remote Panel Application 2400 Step S201 in Figure 8 determines whether the system was started by the RPA tool 2500. If it was started by the RPA tool 2500 (Yes in step S201), it determines whether image data showing the screen of the image processing device 1000 has been received (step S202). If it has been received, the remote panel is placed in the window opened by the RPA tool 2500 (step S203).

[0083] Figure 9(a) shows an example of a remote panel displayed on the LCD2001 in step S203. As shown in this figure, the remote panel of the image processing device is placed on the window 5000, with a tab 5100 located within the display area of ​​this window. The tab 5100 includes a model name display 5110, a device name display 5120, an RPA create / execute button 5130 for accepting the creation and execution of RPA script data, an update button 5140 for accepting updates to RPA script data, and an exit button 5150.

[0084] At the bottom of window 5000 are buttons 4410, 4420, 4430, and 4440 corresponding to the registration keys of the image processing device 1000, as well as a start button 4210 to instruct the start of a job, a stop button 4220 to stop a job, and a reset button 4230 to reset the panel settings.

[0085] The remote panel in Figure 9(a) is the top screen, and it is equipped with buttons 4110 for setting copy settings, 4120 for setting fax / scan settings, 4130 for box operation, 4140 for setting address book settings, 4150 for setting simple security settings, and 4160 for calling up the settings menu.

[0086] The horizontal pixel count of the MFP panel is determined from the model of the connected image processing device and the resolution of the touch panel display 1001 of the selected image processing device. width Vertical pixel count MFP height The data is obtained (step S204). The model of the image processing device is then displayed (step S205), and the process proceeds to the loop of steps S207-S208.

[0087] Step S208 determines whether operation information including E_BTNx and E_BTNy has been output by the RPA tool. If it has been output (Yes in step S208), in step S211, the coordinates E_BTNx and E_BTNy of the location to be operated on in the active window, and the horizontal pixel count E_WIN which constitutes the resolution of the touch panel display 1001 are determined. width Vertical pixel count E_WIN height The horizontal pixel count that constitutes the resolution of the touch panel display 1001 MFP width Vertical pixel count MFP height Based on the following equation 1, the coordinates MFP X MFP Y Calculate (step S211).

[0088]

number

[0089] MFP coordinates X MFP Y The operation information, including the operation information, is output to the image processing device 1000 (step S212).

[0090] Step S207 determines whether image data showing the updated operation screen has been received. If it has been received (Yes in step S207), the remote panel is updated using the image data showing the updated operation screen (step S213), and the process returns to the loop of steps S206 to S208.

[0091] (5-3) Generating operation scripts using RPA tool 2500 In RPA tool 2500, the variable i is set to 1 (step S102 in Figure 7), and the program proceeds to a loop consisting of steps S103 to S112.

[0092] Step S103 is to determine whether an operation was performed on the remote panel. If an operation was performed (Yes in step S103), it is determined whether the operation performed on the remote panel involved specifying a location (step S104).

[0093] If no location is specified (No in step S104), the operation script that performs the i-th procedure is the operation script that performs the same operation as the operation performed by the operator (step S105).

[0094] If a location is specified (Yes in step S104), the action script that performs the same operation as the operation performed by the registrant at the location indicated by variables E_BTNx and E_BTNy is set as the action script for the i-th procedure (step S106).

[0095] A script is generated that normalizes and shows the position of the object to be operated on. The scripts that normalize and show the position of the object to be operated on are the following scripts 1) to 5), and these are used as the script for the i-th procedure (step S107).

[0096] 1) Variable E_WIN width E_WIN height A script to set the horizontal and vertical pixel counts of the active window. 2) Variable M_WIN width M_WIN height A script to set the horizontal and vertical pixel dimensions of the window when creating a script. 3) Variable M_WIN x M_WIN y A script to set the top-left X-coordinate and top-left Y-coordinate when creating a script. 4) Variable M_BTN x , M_BTN y A script to set the X and Y coordinates within the window that is pointed to in the remote panel. 5) Variable M_BTN x , M_BTN y From the variable E_BTN x , E_BTN y Script for the calculation formula that leads to this After generating the above script, the remote panel application 2400 is instructed to update the screen and output operation information to the image processing device 1000. In step S108, E_WIN, which indicates the horizontal and vertical pixel counts of the window, is generated. width ni M_WIN width The value of E_WIN height ni M_WIN height The value of the variable E_BTN x ni M_BTN x The value of the variable E_BTN y ni M_BTN y Set the respective values. Subsequently, output operation information to the remote panel application 2400 (step S109). The operation information includes E_BTNx and E_BTNy calculated by the script described above. Subsequently, wait for the remote panel to be updated by the i-th operation (No in step S110).

[0097] Figure 9(b) shows the screen displayed on the terminal device 2000 when the button 4160 for calling up the settings menu is touched. It includes buttons 4310 for accepting environment settings, 4320 for accepting administrator / device registration, 4330 for accepting destination / box registration, 4340 for accepting user authentication / department settings, 4350 for accepting network settings, 4360 for accepting copy settings, 4370 for accepting printer settings, and 4380 for accepting system connection.

[0098] If the remote panel is updated (Yes in step S110), it is determined whether the stop key has been pressed to indicate that the creation process has ended (step S111). If the creation process has not ended (No in step S111), the variable i is incremented (step S112), and the process returns to step S103. The process from steps S103 to S112 is repeated until the creation process is terminated.

[0099] Once the creation process is complete (Yes in step S111), the action script representing the procedures created so far is saved as RPA script data (step S113).

[0100] (5-4) RPA script data execution procedure Figure 10 is a flowchart showing the RPA script data execution procedure. First, in step 301, a list of connectable image processing devices (image processing devices 1000, 1010, and 1020 in Figure 1) is displayed (step S301), and the system waits for one of the image processing devices to be selected (No in step S302). If selected (Yes in step S302), a window including the RPA menu bar and title bar is opened, similar to step S101 in Figure 7, the selected image processing device is specified, and the remote panel application is launched (step S303). With this specification, the remote panel application 2400 receives image data showing the screen of the selected image processing device (Yes in step S202 in Figure 8), and displays the window using the window opened by the RPA tool 2500 and the selected remote panel (step S203 in Figure 8).

[0101] In step S304 of Figure 10, the RPA script data stored in HDD2005 is displayed in a list and presented to the operator. In step S305, it is determined whether any of the RPA script data has been selected by the operator.

[0102] If any RPA script data is selected (Yes in step S305), the RPA execution button is enabled (step S306), the system waits for the RPA execution button to be pressed (No in step S307), and if it is pressed (Yes in step S307), the variable i is set to 1 (step S308), and the loop consisting of steps S309 to S311 is executed.

[0103] In this loop, the script that constitutes the i-th step of the RPA script data is executed (step S309), it is determined whether the variable i has fallen below the total number m (step S310), and if it has (Yes in step S310), the variable i is incremented (step S311), and the process returns to step S309. The loop continuation requirement is that i falls below m, so the process returns to step S309 until i is between 1 and m-1, and when i becomes m, step S310 becomes No, and the process ends.

[0104] (5-5) Cases where the created RPA script data is used on other devices This section describes a scenario where the same remote panel as the image processing device 1000's operation screen is displayed on the terminal device 2000, RPA script data is created targeting this panel, and then the terminal device 2000 connects to the image processing device 1010 and executes the RPA script data.

[0105] In this case, when the image processing devices are displayed in a list in step S301 of Figure 10, the image processing device 1010 is selected. In this case, in step S203 of Figure 8, the remote panel application 2400 places a remote panel corresponding to the operation screen of the image processing device 1010 in the window opened by the RPA tool 2500, and in step S204, it obtains the model of the connected image processing device, the horizontal pixel count and vertical pixel count of the touch panel display, and stores the obtained horizontal pixel count and vertical pixel count in the variables MFPwidth and MFPheight as described above.

[0106] The remote panel is displayed in the remote panel application 2400. As described above, the screen resolution of the touch panel display 1011 of the image forming apparatus 1010 is different from the screen resolution of the touch panel display 1001 of the image processing apparatus 1000, so the remote panel of the image processing apparatus 1010 is placed in a window of a different size than the remote panel of the image processing apparatus 1000.

[0107] After the remote panel is displayed during the above process, and once the RPA script data to be executed is selected, step S309 in Figure 10 executes the scripts that make up the individual steps of the RPA script data.

[0108] Among these scripts, scripts 3511, 3512, 3513, 3210, and 3220 shown in Figure 5(b) are scripts that normalize and show the position of the object to be manipulated, and by executing these scripts, the coordinates within the window are calculated.

[0109] Specifically, by executing scripts 3511-3513, the horizontal and vertical pixel counts of the current remote panel placed in the active window are calculated, and E_WIN width E_WIN height Set to E_WIN. Script 3210 is E_WIN width And, M_WIN width Based on the ratio, the coordinates within the window being manipulated are calculated and stored in E_BTNx, and script 3220 then executes E_WIN heigh And, M_WIN heigh Based on the ratio, the coordinates within the window being manipulated are calculated and stored in E_BTNy.

[0110] After setting the coordinates for E_BTNx and E_BTNy, a click event is generated by script 3110.

[0111] On the other hand, in the remote panel application 2400, in step S211 of Figure 8, the variable MFP, which indicates the number of horizontal pixels acquired from the image processing device 1010, is used. width The variable MFP indicates the number of vertical pixels obtained from the image processing device 1010. height E_WIN indicates the current horizontal pixel count of the remote panel. width E_WIN indicates the current number of vertical pixels on the remote panel. height Using the above calculation in equation 1, the coordinate MFP X MFP Y The system calculates the coordinates and instructs the image processing device 1010 that these coordinates have been touched.

[0112] Through the above process, the image processing device 1010 can be remotely operated from the terminal device 2000 using RPA script data created for the image processing device 1000. The same applies when operating the remote panel of the image processing device 1020.

[0113] [6] Summary As described above, according to this embodiment, the RPA script data includes a script that sets the horizontal and vertical pixel counts of the window at the time the RPA script data was created as variables, a script that normalizes and displays the coordinates within the window where the user touched the location at the time the RPA script data was created, and a script that recalculates those coordinates. Therefore, when executing the RPA script data, the remote panel can be displayed using the window without having to be aware of the size of the window used to display the remote panel, and the operation procedures defined by the RPA script data can be instructed to be executed.

[0114] [3] Second embodiment In the first embodiment, a script was generated that assigns coordinates manipulated within a window to the variables M_BTNx and M_BTNy. However, due to some accident, the execution of RPA script data containing such a script may terminate with an error. This embodiment relates to error handling when the execution of RPA script data terminates with an error. Specifically, in this embodiment, the error is resolved by increasing the wait time before instructing one step of the RPA script data.

[0115] Figure 11 is a flowchart showing the processing procedure of the RPA tool 2500 according to the second embodiment. This flowchart is based on the flowchart in Figure 10, but differs from Figure 10 in that a decision step S401 is inserted between step S309 and step 310, and steps S402 to S407 are added as processing to be executed when step S401 is Yes.

[0116] Step S401 determines whether, after executing the script that constitutes the i-th step of the RPA script data in step S309 to instruct the image processing device 1000 to perform an operation from the terminal device 2000, the image processing device has not performed any processing and remains stopped. If the image processing device is not stopped, the loop of steps S309 to S311 and step S401 is repeated.

[0117] If the image processing device 1000 stops (Yes in step S401), the error location is detected from the screen transition state of the image processing device at the time of the stoppage and the RPA script data (step S402), and the error location is notified to the administrator (step S403). Subsequently, the terminal device is put into recreation mode (step S404), a screen for adding operation timings (operation timing specification screen) is displayed (step S405), and input of operation timings is accepted (step S406).

[0118] Figure 12 shows an example of the operation timing specification screen. The operation timing specification screen in this figure includes a display of the RPA script data to be executed 6100, a display of the error location in the RPA script data 6200, and a wait time display field 6310. The wait time display field clearly shows the wait time set at the error location. The + key 6320 and - key 6330 accept operations to increase or decrease the wait time. In other words, pressing the + key 6320 increases the wait time, and pressing the - key 6330 decreases the wait time. By repeatedly pressing these keys, the appropriate wait time can be adjusted to an appropriate value.

[0119] The OK button 6410 accepts the operation to confirm the wait time displayed in the wait time display field. The Cancel button 6420 accepts the operation to reset the wait time displayed in the wait time display field to its original state.

[0120] If the wait time is determined using the operation timing specification screen in Figure 12, step S406 becomes Yes, and in step S407, an operation script specifying the operation timing is inserted immediately before the error stop location. In the error location column of the operation timing specification screen in Figure 12, the operation script Click(E_BTN x , E_BTN yIf the image processing device stops working during the execution of ), an operation script Sleep(yyy) is inserted to delay the execution of Click(E_BTNx, E_BTNy) by the wait time specified in the operation timing specification screen in Figure 12, as shown in Figure 13.

[0121] Inserting these action scripts completes the recreation and storage of the RPA script data, thus ending the process in this flowchart.

[0122] As described above, according to this embodiment, after executing the RPA script data, it is determined whether the image processing device 1000 has stopped processing. If it has stopped, the operation script that caused the error is identified, and the RPA script data is recreated so that the execution of the operation script that caused the image processing device to stop is delayed by the corresponding wait time. Therefore, the quality of the RPA script data can be improved with each repeated execution.

[0123] [7] Variant Although the present invention has been described above based on embodiments, it goes without saying that the present invention is not limited to the embodiments described above, and the following modifications are possible.

[0124] (1) Terminal devices 2000 include types with replaceable displays and types with changeable display resolutions. In these types of terminal devices, the display resolution may differ between the time the RPA script data is created and the time the RPA script data is executed. In this case, the horizontal pixel count M_SCN of the LCD2001 screen resolution at the time the RPA script data is created. width Vertical pixel count M_SCN height The horizontal pixel count of the LCD2001 screen resolution when the RPA script data is executed E_SCN width Vertical pixel count E_SCN height Using the following equation (Equation 2), the window coordinates E_BTN x , E_BTN y You may calculate this.

[0125]

number

[0126] (2) The RPA tool 2500 may calculate the coordinates on the touch panel display 1001 and instruct the MFP 1000 on the operations to be performed, rather than via the remote panel application 2400. If the resolution of the LCD 2001 does not change between the creation and execution of the RPA script data, the horizontal pixel count that constitutes the resolution of the touch panel display 1001 is calculated by the MFP width Vertical pixel count MFP height By performing calculations using Mathematics III, the RPA tool 2500 determines the X coordinate of the operation position on the touch panel display 1001. x Y coordinate MFP y You may also request this.

[0127]

number

[0128] (3) If the resolution of LCD2001 changes between the creation and execution of the RPA script data, the X coordinate MFP of the operation position on the touch panel display 1001 can be calculated by performing the calculation in equation 4 below. x Y coordinate MFP y We seek.

[0129]

number

[0130] (4) In RPA creation mode, the script file containing the RPA script sequence created by the RPA tool 2500 is written to and stored on the HDD2005 of the terminal device 2000, but this is not limited to that. In RPA creation mode, the script file containing the RPA script sequence created by the RPA tool 2500 is sent to the image processing device 1000 and stored in the image processing device 1000. In RPA execution mode, files with extensions specific to script files are read from the internal storage of the image processing device 1000 and displayed as a list on the LCD2001 of the terminal device 2000. Then, the RPA script sequence selected by the user from the displayed list is executed. Alternatively, in RPA creation mode, the script file containing the RPA script sequence created by the RPA tool 2500 may be written to and stored on the storage of a server device on an external network.

[0131] (5) In the above embodiment, the RPA tool 2500 is started on the terminal device 2000 and the RPA script data is executed by the RPA tool 2500, but the embodiment is not limited to this. The program file of the RPA tool 2500 that creates and executes the RPA script data may be installed on the image processing device 1000, and the creation and execution of the RPA script data may be performed on the image processing device 1000 in response to instructions from the terminal device 2000.

[0132] In this modified example, the image processing device 1000 is equipped with server and browser functions and accepts user operations via the browser screen. In this modified example, the browser of the terminal device 2000 sends an RPA script data creation request to the RPA tool 2500 of the image processing device 1000, which starts the RPA tool 2500 and generates the scripts shown in the above embodiment, which save the X coordinate of the top left of the window to a variable, the Y coordinate of the top left to a variable, the horizontal number of pixels of the window to a variable, the vertical number of pixels to a variable, and the touch coordinates within the window to a variable, and saves the RPA script data including these scripts to the HDD 1005 of the image processing device 1000.

[0133] When a request to execute RPA script data is received from the browser of terminal device 2000, a list of RPA script data stored on HDD1005 is displayed, and the user is accepted to select the RPA script data to be executed. Once the RPA script data to be executed is selected, the selected RPA script data is executed.

[0134] (6) The RPA script data includes an action script to calculate the horizontal and vertical pixels of the window containing the operation screen, but this is not limited to that. The RPA tool may be configured to calculate the horizontal and vertical pixels of the window containing the operation screen when executing the RPA script data.

[0135] (7) The display size is expressed as the horizontal and vertical pixels of the window, but is not limited to this. When creating the RPA script data, it may also be expressed as the scaling ratio of the window placed on the remote panel. If the aspect ratio is the same as the desktop screen (4002), the scaling ratio is expressed as 100%. If it is 1 / n of the desktop screen (where n is an integer greater than or equal to 2), it is expressed as 1 / n. The resolution of the image processing device is expressed as a pair of horizontal and vertical pixels, but is not limited to this. It may also be expressed as dots per inch (dpi).

[0136] (8) In the second embodiment, if the image processing device stops due to the execution of RPA script data, the error location is notified to the administrator and the RPA script data is recreated, but this is not limited to the second embodiment. The recreation of the RPA script data may also be performed by automatically inserting a wait time.

[0137] Figure 14 is a flowchart showing the processing procedure of the RPA tool 2500 according to a modified example of the second embodiment. This figure is based on Figure 10, but differs from the second embodiment in that steps S401, S412-S415, and S421-S423 are inserted between step S309 and step S310, and steps S424 and S425 are added after step S310.

[0138] The variable `h` is a counter variable that counts the number of errors that occur, and is initially set to 0. The variable `e` is a variable that indicates the step in which the error occurred, and the variable `i` at the time the error occurred is copied to it.

[0139] Step S412 is a determination step that is executed when the image processing device stops and step S401 is determined to be Yes, and it determines whether the variable h is 0. If the error occurred for the first time in the i-th procedure and h=0 (Yes in step S412), the variable i, which indicates the current procedure, is set to the variable e, which indicates the location of the error (step S413). From there, the variable h is incremented (step S414).

[0140] Next, in step S415, the system waits for a predetermined unit of time to elapse (No in step S415). Once the time has elapsed (Yes in step S415), the system proceeds back to step S309 and issues a script for the i-th step. The waiting period for the unit of time and the instruction for the i-th step are repeated as long as the error persists. Each time an instruction is issued, the variable h is incremented. Step S401 becomes No. Once the error is resolved by repeating the waiting period for the unit of time and the instruction for the i-th step, in step S421, it is determined whether the variables e and h are 0.

[0141] If variables h and e are 0, steps S422 and S423 are skipped and the process proceeds to step S310. If they are 0, variables e and h are recorded as error history (step S422), then variables h and e are set to 0 (step S423), and the process proceeds to step S310. The process continues as in the first embodiment, repeating steps S309-S311 and S421-S423 until variable i becomes a prime number m. Subsequently, if step S310 is No and all scripts specified by the RPA script data have been executed, step S424 determines whether an error occurred during the execution process of the RPA script data up to that point. If an error occurs (Yes in step S424), a script is added before the i-th step script to indicate that a wait of unit time × h will be performed (step S425).

[0142] (9) In the above modified example, when an error occurred, the error was resolved by increasing the weight according to the number of times the error occurred. In contrast, in this modified example, after giving coordinate instructions to the image processing device 1000, if an error occurs in the procedure instructions to the image processing device 1000, the error is resolved by incrementing the window's X coordinate and window's Y coordinate.

[0143] Figure 15 is a flowchart showing the processing procedure of the RPA tool 2500 in a modified example. Compared to Figure 14, Figure 15 differs in that step S415 is replaced by step S435 and step S425 is replaced by step S445. Specifically, if it is determined that the image processing device has stopped in the i-th step of the RPA script data (Yes in step S401), the variable h is incremented (step S414), and then the window-internal X and Y coordinates that should be set in the variables M_BTNx and M_BTNy in the script that makes up the i-th step are corrected according to the rule (h) corresponding to the variable h (step S435), and the process returns to step S309. The rule (h) is a rule that increases the window-internal X and Y coordinates according to the variable h. Specifically, if the variable h is an odd number, the window-internal X coordinate is increased by that odd value. If the variable h is an even number, the window-internal Y coordinate is increased by that even value. Therefore, as the variable h increases, the window-internal X coordinate increases to the right, and the window-internal Y coordinate increases downwards.

[0144] Each time an error occurs, the X and Y coordinates within the window that should be set to variables M_BTNx and M_BTNy change according to variable h, and the process returns to step S309. As this process is repeated, the number of X and Y coordinates within the window that should be set to M_BTNx and M_BTNy increases. In this way, as the number of X and Y coordinates within the window increases, the probability of correctly pressing the position of a GUI component on the remote panel increases. If, after repeating steps S309, S401, steps S412-S414, and step S435, the script of procedure i is executed and no errors occur (No in step S401), then the process proceeds to step S421. Step S421 determines whether variables h and e are 0. If these variables are not 0, variable e is shown in the error history (step S422), and then variables h and e are set to 0 (step S423).

[0145] Furthermore, after step S401 becomes No and the loop of steps SS309~S311, S401, and S421~S423 is exited, it is determined whether an error has occurred so far (step S424). If an error has occurred so far (Yes in step S424), the script for the procedure shown in the error history is rewritten according to the corrected window coordinate system X coordinate and window Y coordinate (step S445), and the main flow is returned.

[0146] (10) The scripts that can be used are Python®, VBScript®, VB.NET®, GAS®, and C#®.

[0147] VBScript (registered trademark) is a language used in WinActor. VB.NET (registered product) is a visual basic language used in UiPath and is suitable for operating Microsoft Office products (registered trademark).

[0148] GAS (Google App Script: registered trademark) is a programming language developed based on JavaScript (registered trademark), and it can be used to connect with and manipulate services such as Google Maps, email, and spreadsheets.

[0149] C# (registered trademark) is a programming language developed by Microsoft Corporation that allows users to utilize APIs for operating Windows (registered trademark).

[0150] These are not the only options; any programming language that can be executed without compilation or linking may be used.

[0151] (11) The paper type loaded into the paper cassette may be automatically set by describing the paper settings according to the detection results of the media sensor in RPA script data and having the RPA tool 2500 execute it. The media sensor measures the reflectance of green (G) light, the transmittance of near-infrared light (Ir), the transmittance of blue light (B), and the transmittance of ultrasound using an optical sensor, an infrared sensor, and an ultrasonic sensor, and determines from these reflectances and transmittances whether the type of sheet used for printing is recycled paper, coated paper, or an envelope. When the media sensor detects a paper type, a string indicating the detected paper type is displayed on the screen, and this detected string is set as the paper type for the corresponding paper cassette. In this way, the detection result from the media sensor can be reflected in the paper type of the paper cassette without human intervention. Alternatively, a script that sets the in-window coordinates of the location where the media sensor detection result is displayed, and the top-left coordinates of the window, as variables, and a script that sets the in-window coordinates of the location where the paper type for the paper cassette is set, and the top-left coordinates of the window, as variables can be generated as RPA script data, and the device settings can be performed using such RPA script data.

[0152] Alternatively, such detection results may be registered as paper information for the image processing device on a server that manages the image processing device.

[0153] (12) In terminal device 2000, the pointing device that accepts user input is a mouse or keyboard, but is not limited to these. It may also be a keypad or stylus. Terminal device 2000 is described as a laptop computer, but is not limited to this. The terminal device may be a desktop computer operated with a mouse, keypad, etc., a smartphone, a tablet device, or a cash register terminal. (13) Although examples of image processing devices such as MFPs, SFPs, and AIOs have been described, the image processing device is not limited to these. It may also be a single-function scanner or a facsimile machine. It may be installed in a production printing machine, or in a label printer, postcard printer, or ticket machine. It may also be installed in a single-function copier or a single-function peripheral device (printer) for a personal computer. [Industrial applicability]

[0154] This invention allows for the automation of device settings when introducing an image processing device to a business premises and panel settings when executing jobs on the image processing device using RPA on a terminal device. Therefore, it has the potential to be used in various industrial fields, including the office automation equipment and information equipment industries, as well as retail, rental, real estate, advertising, transportation, publishing, and more. [Explanation of Symbols]

[0155] 1000, 1010, 1020 Image Processing Device 1001, 1011, 1021 Touch Panel Display 1300 applications 1400 MFP Panel App 1500 Processing Execution Apps 2000 Terminal device 2200 OS 2210 kernel 2220 Multi-Window System 2300, 1300 applications 2400 Remote Panel Apps 2500 RPA tools 2600 RPA script data

Claims

1. A method for creating RPA script data for an image processing device, in a terminal device that displays an operation screen for an image processing device on its own terminal and accepts operations via RPA, The first step is to detect the position of the target to be operated, which is displayed on the operation screen and selected by the operator, The process includes a second step of normalizing the detected position of the target to be operated on based on the size of the operation screen, generating position information indicating the normalized position, and recording this position information in the RPA script data. A method for creating RPA script data characterized by the following features.

2. The terminal device displays the operation screen in a window, The normalization of the position of the object to be operated on in the second step is performed using the display size when the operation screen is displayed in a window and the in-window coordinates of the position of the object to be operated on. The method for creating RPA script data as described in feature 1.

3. The second step described above is: A script to obtain the size of the aforementioned window display, A calculation script is generated to calculate the in-window coordinates indicating the position of the target, using the acquired window display size and the normalized position of the target indicated in the position information, and this script is included in the RPA script data. The method for creating RPA script data according to feature 2.

4. The RPA script data includes resolution information indicating the screen resolution of the terminal device when the position of the target to be operated on is detected. The calculation script generated in the second step uses the ratio between the screen resolution indicated in the resolution information of the RPA script data and the screen resolution of the terminal device when the RPA script data is executed to calculate the coordinates within the window. The method for creating RPA script data according to feature 3.

5. The second step described above is: A calculation script is generated to calculate the coordinates within the operation screen of the image processing device, using the screen resolution of the image processing device and the normalized position of the object being operated on, as indicated in the position information, and this script is included in the RPA script data. The method for creating RPA script data as described in feature 1.

6. A terminal device that displays an operation screen for an image processing device on its own terminal and accepts operations via RPA, a method for executing RPA script data to perform RPA operations by executing RPA script data, The first step is to calculate the coordinates of the location to be operated by RPA based on normalized location information using the size of the operation screen, The process includes a second step of instructing an image processing device to perform operations using RPA at the calculated coordinates and to carry out the corresponding processing. A method for executing RPA script data, characterized by the following features.

7. The terminal device displays the operation screen of the image processing device in a window. The calculation of coordinates by the first step described above is This is done by obtaining the size of the window display and using the obtained size and the normalized position of the target being manipulated, as indicated in the position information, to calculate the coordinates within the window. The method for executing RPA script data according to feature 6.

8. The RPA script data includes resolution information indicating the screen resolution of the terminal device when the position of the target to be operated on is detected. The first step uses the ratio between the screen resolution indicated in the resolution information of the RPA script data and the screen resolution of the terminal device when executing the RPA script data to calculate the in-window coordinates of the position of the object to be operated on. The method for executing RPA script data according to feature 7.

9. The calculation of coordinates by the first step described above is This is done by calculating the in-screen coordinates of the image processing device using the screen resolution of the image processing device and the normalized position of the object being operated on, as indicated in the position information. A method for executing RPA script data according to claim 6.

10. The calculation of the coordinates in the first step is performed by either the terminal device or the image processing device. A method for executing RPA script data as described in any one of 6 to 9.

11. Furthermore, a first substep presents the user with connectable image processing devices and accepts their selection of which image processing device to use for processing. If an image processing device is selected, the second substep includes obtaining the screen resolution of the image processing device. The calculation of the coordinates in the first step is performed by calculating the in-screen coordinates of the image processing device using the ratio of the position information of the RPA script data to the screen resolution of the acquired image processing device. The method for executing RPA script data according to feature 6.

12. The aforementioned RPA script data consists of multiple operation scripts, In the third step of instructing the image processing device to perform an operation based on the RPA script data, it is necessary to determine whether the image processing device has stopped. If processing by the image processing device stops, the system has a fourth step of notifying the administrator of the location of the error in one of the aforementioned operation scripts. The aforementioned position information is included in any of the operation scripts. The method for executing RPA script data according to feature 6.

13. The fourth step involves a fifth step in which the RPA script data is recreated after the notification in the fourth step. The fourth step includes a substep for receiving the operation timing specification from the administrator. Before and after the action script that was reported as an error location, a script is inserted to adjust the timing of the action script's execution according to the specified operation timing. The method for executing RPA script data according to claim 12.

14. A receiving means that receives a bitmap representing the operation screen of an image forming apparatus from an image processing apparatus, A display means that writes the received bitmap to VRAM and displays it, When registering operations to be performed by RPA, a receiving means is provided to receive the operator's specification of the location to be operated on within the bitmap written to VRAM, A generation means that obtains location information that normalizes the specified position according to the size of the displayed bitmap, and generates RPA script data that contains the location information. A terminal device characterized by being equipped with the following features.

15. A transmission means for transmitting a bitmap representing the operation screen of an image forming apparatus to a terminal device, When performing an operation using RPA, a receiving means is used to receive normalized location information from a terminal device, The system includes a calculation means for calculating the coordinates of the location to be targeted by RPA based on the size of the bitmap displayed on the terminal device and normalized location information. The terminal device is an image processing device characterized by performing RPA operations using coordinates calculated by the calculation means.

16. A program that causes a computer displaying an operation screen for an image processing device to perform operations using RPA, A first code that calculates the coordinates of the target of the operation on the displayed operation screen based on position information normalized using the size of the operation screen, The calculated coordinates include a second code that causes the computer to perform operations using RPA. A program characterized by the following features.

Images