Control method, image processing device, terminal device, program

Abstract

To provide a control method capable of executing processing intended to be automatic in a RPA without terminating the processing in the middle.SOLUTION: In a RPA system, a MFP 1000 includes a remote operation mode that receives a control instruction from a terminal device 2000. In the terminal device 2000, the terminal device that transits the MFP 1000 to the remote operation mode receives an operation by the RPA, and performs a control instruction to the MFP 1000. The MFP 1000 timeouts a remote operation mode when a period without a communication with the terminal device 2000 becomes a constant time. On the other hand, the terminal device 2000 performs a maintain control for maintaining the remote operation mode of the MFP 1000 in a period shorter than the period until the remote operation timeouts after performing the control instruction to the MFP 1000.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an improvement in realizing control in a remote operation mode in a system composed of an image processing apparatus and a terminal device.

Background Art

[0002] Control in the remote operation mode is a technique of displaying a screen (remote panel) with the same content as the operation screen of the image processing apparatus on an external device connected to the image processing apparatus and accepting operations on the remote panel, thereby instructing the processes to be executed on the image processing apparatus. The remote panel may be used by a general user for operations, or may be used by a service technician of the image processing apparatus for device settings and maintenance management at the time of introducing the image processing apparatus.

[0003] Even if the installation location of the image processing apparatus is different, service technicians often repeat the same setting work and maintenance management work. Therefore, in recent years, there has been an active study on reducing the burden on service technicians by operating the remote panel by RPA.

[0004] Regarding the prior art of the remote panel, an image processing apparatus and a terminal device described in Patent Document 1 below are known.

Prior Art Document

Patent Document

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, in conventional remote operation mode, the image processing device disables all operations on the local panel (the device's control panel) other than remote operations by the user. However, it is extremely inconvenient to have operations on the local panel unnecessarily prohibited for extended periods. Therefore, if there are no operations on the terminal device's remote panel for an extended period and no communication occurs with the terminal device, the image processing device will time out the remote operation mode and enable user operations on the local panel.

[0007] On the other hand, if the image processing device takes a long time to process, causing a delay in the remote panel switching, the program that instructs the RPA on the operation procedure (called RPA script data) can be written to wait for processing to complete between the script for one operation procedure and the script for the next operation procedure, so as to wait for the remote panel screen content to switch reliably.

[0008] However, if a script that waits for processing is inserted in this way, the terminal device running the RPA will not issue processing instructions to the image processing device while waiting for the remote panel to switch. This can lead to cases where the image processing device times out and exits remote operation mode due to a lack of communication from the terminal device. When remote operation mode is exited due to such a timeout, it becomes impossible to maintain the operation performed by the RPA.

[0009] The problem of deactivating remote control mode is not limited to cases where the image processing device takes a long time to process. It can also occur when the operator has difficulty configuring the image processing device and does not operate the device for an extended period.

[0010] The first objective of this disclosure is to provide a control method that enables the execution of processes intended for automation using RPA without premature termination.

[0011] The second objective is to provide a terminal device capable of performing the control method described above.

[0013] The 3 The objective is to provide a program that allows a computer to execute the control methods described above. [Means for solving the problem]

[0014] The first objective described above is achieved by a control method for an image processing device having a remote operation mode for receiving control instructions from a terminal device, comprising: a first step of transitioning the image processing device to remote operation mode; and a second step of the terminal device receiving an operation by RPA and issuing a control instruction to the image processing device, wherein the image processing device cancels the remote operation mode when there is no communication with the terminal device for a certain period of time, and the second step is achieved by the terminal device repeatedly communicating with the image processing device at a cycle shorter than the certain period of time after issuing a control instruction to the image processing device.

[0015] The repetition of the communication may be performed by the terminal device repeatedly requesting the image processing device to report the progress of the processing corresponding to the control instruction at a interval shorter than the fixed time, and the image processing device returning a report in response to the request to the terminal device.

[0016] The progress report indicates whether the process is incomplete or complete. The second step may choose not to perform any subsequent operations if the report indicates the process is incomplete, or to perform any subsequent operations if the report indicates the process is complete.

[0017] The repeated communication involves repeatedly transmitting image data showing the contents of the operation screen of the image processing device. The terminal device may determine whether the process for which a control instruction has been given has been completed based on whether the contents of the operation screen shown in the image data transmitted from the image processing device are the same as the contents of the operation screen shown in the image data transmitted immediately before.

[0018] The timeout for the remote operation mode is the disconnection of the connection established between the terminal device and the image processing device. Communication with the image forming apparatus This may involve transmitting information for maintaining the connection from one terminal device and the other image processing device within the specified time period. The progress of the process indicated in the report may be whether the panel settings of the image processing device are complete or incomplete.

[0019] If the progress of the process indicated by the progress report indicates the completion of the panel settings of the image processing device, the report may also indicate whether a restart is in progress to reflect the panel settings in the image processing device, or whether the restart has been completed.

[0020] If the progress report indicates that a restart is in progress, the report may also indicate the remaining time until the restart is complete.

[0021] The progress status of the process indicated in the aforementioned progress report may be whether the job commanded to the image processing device has been completed or not.

[0022] The progress status of the process, as indicated by the progress report, may indicate whether the job for which a control instruction was given is completed or incomplete, as well as whether or not the type of job involves print output. If the job for which a control instruction was given is incomplete, but the type of job currently running involves print output, the second step may be to have the RPA perform the subsequent operations.

[0023] In the first step, a third step may be performed in which the remote operation mode is notified that the remote operation mode is performed by the RPA, and when the image processing device receives the notification, it displays on the operation panel of the image processing device that the remote operation mode by the RPA is being performed.

[0024] In the third step, further, when the process corresponding to the first control instruction by the RPA is completed, it may be displayed on the operation panel to that effect.

[0026] The second object is achieved by a terminal device that issues a control instruction to an image processing apparatus having a remote operation mode, the terminal device including a transition means for transitioning the image processing apparatus to the remote operation mode and an instruction means for receiving an operation by RPA and issuing a control instruction to the image processing apparatus, wherein when the communication with the image processing apparatus has been interrupted for a certain period of time and the remote operation mode is to be canceled, after issuing a control instruction to the image processing apparatus, the instruction means repeats communication with the image processing apparatus at a cycle shorter than the certain period of time.

[0028] The above-mentioned second 3 object is achieved by a program that causes a computer to execute a process of issuing a control instruction to an image processing apparatus having a remote operation mode, the program causing the computer to execute a first step of transitioning the image processing apparatus to the remote operation mode and a second step of receiving an operation by RPA and issuing a control instruction to the image processing apparatus, and when the communication with the computer has been interrupted for a certain period of time and the image processing apparatus cancels the remote operation mode, the second step causes the computer to repeat communication with the image processing apparatus at a cycle shorter than the certain period of time after issuing a control instruction to the image processing apparatus.

Advantages of the Invention

[0030] In the second step, after issuing a control instruction to the image processing apparatus, communication with the image processing apparatus is repeated at a cycle shorter than a certain period of time until the remote operation mode is canceled. Therefore, even if the execution of the process for which the control instruction was issued takes time, the remote operation mode will not be canceled. The process intended by the RPA can be executed without interruption.

Brief Description of the Drawings

[0031] [Figure 1] This shows the configuration of the RPA system according to this embodiment. [Figure 2] The hardware configuration of terminal device 2000 is shown. [Figure 3] Figure 3(a) shows the software stacks of the programs installed on HDD2005 of terminal device 2000 and HDD1005 of MFP1000. Figure 3(b) shows the connection used for communication between terminal device 2000 and MFP1000. [Figure 4] Figure 4(a) shows the sequence when the MFP1000 processes without delay and there is almost no period of no input. Figure 4(b) shows the case where the MFP processing time Et is longer than the timeout value Ct for the period of no input. [Figure 5] This example script allows you to send a fax transmission report screen. [Figure 6] (a) Shows the FAX top screen 3000 that is displayed when FAX is selected as the job type. (b) Shows an example of the menu displayed by the RPA tool 2500. [Figure 7] This is a flowchart showing the steps for executing RPA script data. [Figure 8] This is a flowchart showing the procedure for checking the progress of the MFP (Multi-Function Program). [Figure 9] (a) Shows the remote panel displayed on terminal device 2000 while the RPA script data is being executed. (b) Shows the remote panel after the RPA script data has finished executing. [Figure 10] (a) The communication sequence between the RPA tool 2500, the remote panel application 2400, and the progress confirmation application 1700 in the second embodiment. (b) The multiple image data transmitted by the progress confirmation application 1700 in the second embodiment. [Figure 11] This is a flowchart showing the processing procedure of the progress confirmation application 1700 according to the second embodiment. [Figure 12] Here is an example of RPA script data for initial setup. [Figure 13] Here is an example of RPA script data for initial setup. [Figure 14] Here is an example of RPA script data for initial setup. [Figure 15] This shows the screen used for initial setup when installing the MFP1000. [Figure 16] A sequence diagram according to the fourth embodiment is shown. [Figure 17] This is a flowchart showing the processing procedure of the RPA tool 2500 according to the fourth embodiment. [Figure 18] A flowchart showing the details of progress status assessment. [Figure 19] This is a flowchart showing the procedure for sending a progress report request message according to the fourth embodiment. [Figure 20] This is a flowchart showing the procedure for sending a progress report request message according to the fourth embodiment. [Figure 21] This is a flowchart of the main routine of RPA Tool 2500 when performing integration settings. [Figure 22] An example of the integration settings screen is shown below. [Modes for carrying out the invention]

[0032] [1] System configuration Hereinafter, embodiments of the image processing apparatus and terminal device according to this disclosure will be described with reference to the drawings. Figure 1 shows the configuration of the RPA system according to this embodiment. As shown in this figure, the RPA system consists of an MFP 1000 and a terminal device 2000, and realizes control by remote operation mode.

[0033] The MFP (Multifunction Peripheral) 1000 is an image forming apparatus that is one form of an image processing apparatus. In addition to operations on the touch panel display 1001, it performs device settings and job settings in response to instructions from a remote panel displayed on the terminal device 2000, and executes jobs according to these settings. Jobs include scanning, copying, faxing, and memory card printing. Operations from the touch panel display 1001 and operations from the remote panel 2001D on the terminal device 2000 are mutually exclusive. While operation input from the remote panel is enabled, operation input from the touch panel display 1001 is disabled.

[0034] The terminal device 2000 is a device used by the user operating the MFP 1000 together with the MFP 1000. It has a remote panel 2001D on the LCD 2001 that is the same as the screen 1001D displayed on the MFP 1000's touch panel display 1001. The terminal device 2000 receives operations from the user on the remote panel 2001D and commands the MFP 1000 to perform processing according to those operations.

[0035] [2] Hardware configuration of terminal device 2000 and MFP1000 Figure 2 shows the hardware configuration of terminal device 2000. As shown in this figure, terminal device 2000 includes an LCD 2001 that provides an interactive operating environment to the user, a mouse 2002 which is an example of a pointing device, a keyboard 2003, a network I / F 2004, an HDD 2005 on which various programs are installed, a RAM 2006 on which necessary programs from those installed on the HDD 2005 are loaded, a CPU 2007 that executes the programs loaded into RAM, a flash ROM 2008 for saving necessary settings for program execution, and peripheral I / O 2009 for receiving input operations to the mouse 2002 and keyboard 2003.

[0036] The Network I / F2004 is a communication module that provides LAN and WAN connectivity. This communication module includes short-range communication modules such as Bluetooth®, Wi-Fi, EnOcean, and low-power wireless (Wi-SUN), and performs wireless transmission and reception of operation information and image data.

[0037] The MFP1000 has the same hardware configuration as the terminal device 2000 (in addition to the touch panel display 1001, it has the network interface 1004, HDD 1005, RAM 1006, CPU 1007, and ROM 1008 shown in Figure 3(a)), and the necessary programs are installed on the HDD 1005.

[0038] [3] Software stack Figure 3(a) shows the software stacks of the programs installed on HDD2005 of terminal device 2000 and HDD1005 of MFP1000.

[0039] The software stack of terminal device 2000 has a control hierarchy in which the operating system (OS) 2200 is installed on the hardware 2100 of terminal device 2000, and the applications 2300 are installed on top of that.

[0040] The software stack of the MFP1000, like that of the terminal device 2000, has a control hierarchy in which the OS 1200 is placed on the MFP1000 hardware 1100, and the application 1300 is placed on top of that.

[0041] The OS 2200 of terminal device 2000 and the OS 1200 of MFP 1000 perform data communication via a protocol stack through network I / F 2004 and 1004. Applications 2300 and 1300 establish a TCP connection 2104 using TCP protocols 1134 and 2034, IP protocols 1124 and 2024, and short-range communication protocols 1114 and 2014, and perform sending and receiving HTTP messages through the TCP connection 2104.

[0042] TCP connection 2104 is established when terminal device 2000 acts as the host, application 2300 sends a connection request specifying the TCP port number of the other application 1300 and its own port number, and application 1300 confirms that the port requested by the connecting side is waiting for a connection and returns a progress report message.

[0043] The application 2300 running on terminal device 2000 and the application 1300 running on MFP 1000 use TCP connection 2104, using the port number specified by the host at the time of connection, to send and receive image data and operation information.

[0044] On the other hand, TCP connection 2104 has a time limit, so if no data is sent or received within the specified time, TCP connection 2104 will be disconnected.

[0045] (3-1) Software stack of terminal device 2000 OS2200 includes kernel 2210 and multi-window system 2220.

[0046] Kernel 2210 allocates multiple sub-regions of RAM2006 to each application, launches multiple applications, and monitors the operation of each application.

[0047] The multi-window system 2220, when commanded to open a file or launch an application, opens a new window and allocates the area within that window to the opened file or application.

[0048] Typical examples of applications 2300 managed by kernel 2210 include the remote panel application 2400 and the RPA tool 2500.

[0049] The remote panel application 2400 places a remote panel, which is the same screen displayed on the MFP1000, in one of the windows provided by the multi-window system 2220, allowing the user to perform remote operations.

[0050] Remote operation is achieved through data transmission and reception with the MFP1000 as the other party. This data transmission and reception consists of sending operation data indicating the operation performed by the operator on the terminal device 2000, and receiving screen data after the screen has been switched (transitioned) when the MFP1000 receives the operation data and switches screens. By repeating this data transmission and reception, screen transitions similar to those on the MFP1000 are realized on the terminal device 2000.

[0051] The RPA tool 2500 creates RPA script data 2600 targeting the screen data of the MFP1000 received by the remote panel application 2400, and performs control in remote operation mode based on the RPA script data 2600. In creating the RPA script data 2600, the screen data of the MFP1000 is displayed on the remote terminal screen, user operations on the screen data are accepted, the coordinates of the positions where input such as touches are made by the user operations are detected, and a script is created from the detected coordinate information. The RPA script data 2600 is obtained by repeating this script creation from coordinate information.

[0052] On the other hand, when executing RPA script data, the coordinate information of each script that constitutes each step of the RPA script data is sent to the remote panel application, and the remote panel application is instructed to perform remote operations based on that coordinate information.

[0053] (3-2) Software stack of the MFP1000 The programs installed on the HDD1005 of the MFP1000 also have the same software stack as the terminal device 2000. Specifically, the MFP1000's hardware 1100 is configured with the OS 1200 and applications 1300. Applications 1300 include the MFP panel application 1400, the processing execution application 1500, and the progress confirmation application 1700, while the OS 1200 includes the operation system control unit 1600.

[0054] The MFP panel application 1400 is compatible with the remote panel application 2400, and in the case of the remote panel system, it causes the processing execution application 1500 to execute the processing necessary for device settings and jobs based on either an operation performed by the operator on the touch panel display 1001 or an operation performed on the remote panel.

[0055] The processing execution application 1500 executes panel settings and performs device settings and jobs when installing the MFP 1000 at a business site, in accordance with the operation input made to the MFP panel application 1400.

[0056] The operation system control unit 1600 is one of the basic input / output programs in OS1200 and switches between operating the MFP panel application 1400 via local operation from the touch panel display 1001 or via remote operation from the remote panel application 2400. By default, the MFP panel application 1400 is operated via local operation from the touch panel display 1001. On the other hand, if the RPA tool 2500 requests to start RPA operation mode, local operation from the touch panel display 1001 is prohibited, and the MFP panel application 1400 is made to perform jobs and device settings via remote operation from the remote panel application 2400. The operation system control unit 1600 includes an input-free period counter 1610 and monitors whether operations from the remote panel application 2400 are being performed continuously. The no-input period counter 1610 starts timing how long the period of no input without remote operation continues if there is no remote operation to the MFP panel application 1400. When this timing reaches a predetermined upper limit (timeout period), the remote operation mode times out. Specifically, it disables operation input from the remote panel application 2400 and enables operation via the touch panel display 1001.

[0057] The progress confirmation application 1700, in RPA operation mode, prevents the operation system control unit 1600 from switching to the touch panel display 1001 even if there is a long period of no input, and performs control to maintain operation input by the remote panel application 2400. This control is performed in cooperation with the RPA tool 2500. The control to maintain operation input in remote operation mode will be explained below with reference to the sequence diagrams in Figures 4(a) and (b).

[0058] [4] Device integration using Remote Panel App 2400 and RPA Tool 2500 Figure 4(a) shows the sequence when the MFP1000 side processes without delay and there is virtually no period of no input.

[0059] In this case, the sequence consists of a phase in which a connection is established between the RPA tool 2500 and the MFP panel application 1400, and a phase in which the MFP panel application 1400 is instructed to perform processing via the remote panel application 2400.

[0060] The former phase is accomplished when the RPA tool 2500 sends an MFP connection request message 2401 to the MFP panel application 1400, and in response, the MFP panel application 1400 sends a connection permission message 1010 to the RPA tool 2500.

[0061] The latter phase is performed when the remote panel application 2400 issues the first step instruction message 2411, and the remote panel application 2400 sends the MFP coordinate instruction 2311 to the MFP 1000, and in response, the MFP panel application 1400 sends a panel screen response message 1011 to the remote panel application 2400. The time from when the MFP panel application 1400 receives the MFP coordinate instruction 2311 until when it sends the panel screen response message 1011 is the processing time in the MFP 1000 (let's call this processing time Et). On the other hand, when the MFP coordinate instruction 2311 is received, the no-input period counter 1610 starts counting the no-input period. Ct is the threshold for the no-input period. In the case of Figure 4(a), the time from the receipt of the MFP coordinate instruction 2311 to the transmission of the panel screen response message 1011 is short, so the no-input period count does not time out.

[0062] Figure 4(b) shows a case where the processing time Et from when the MFP panel application 1400 receives the MFP coordinate instruction 2311 until it sends the panel screen response message 1031 is longer than the timeout value Ct for the period of no input. This case occurs because the processing execution application 1500 takes too long to perform the processing instructed by the RPA tool 2500, causing the TCP connection to be disconnected and resulting in a prolonged period of no input.

[0063] As the processing time Et increases, the count value of the no-input period from the MFP coordinate instruction 2311 becomes the threshold Ct, and if left as is, the operation input to the MFP panel application 1400 will switch from remote operation to local operation. To avoid this switch, the progress confirmation application 1700 starts timing the transmission period P, and when the control period P has elapsed, it sends a progress report request message 2432 to the MFP 1000. This transmission period P is shorter than the no-input period threshold Ct mentioned above (for example, about 10 seconds). When the progress report request message 2432 is sent, the progress confirmation application 1700 returns a progress report message 1032 indicating that the processing is incomplete.

[0064] Upon receiving a progress report message 1032 indicating incompleteness, the RPA tool 2500 sends another progress report request message 2433 to the progress confirmation application 1700 after the transmission cycle P has elapsed. Upon receiving a progress report message 1033 from the progress confirmation application 1700 in response, the RPA tool 2500 sends a progress report request message 2434 to the MFP 1000 after the above transmission cycle P has elapsed. This prevents the MFP 1000 from timing out even if its processing takes a long time.

[0065] When the processing execution application 1500 finishes executing the process instructed by the remote panel application 2400, it switches the panel screen according to the MFP coordinate instruction 2331 and returns a panel screen response message 1011 to the remote panel application 2400. After the instructed process is completed, when the next progress report request message 2435 is sent from the RPA tool 2500 to the MFP panel application 1400 of the MFP 1000, it returns a progress report message 1035 indicating that the process is complete.

[0066] When MFP1000 notifies a progress report message 1035 indicating completion, the RPA tool 2500 executes the next script in the RPA script data and issues the instruction message 2441 for the k+1th step.

[0067] [5] Processing procedure of RPA tool 2500 (5-1) Example of RPA script data RPA script data 2600 consists of a script that shows a series of tasks performed by the user. Figure 5 shows an example of RPA script data for making the MFP1000 send a fax. The following explanation of RPA script data will refer to Figure 5.

[0068] Step 1 of the RPA script data in Figure 5 is written by a script that instructs the system to generate a click event at coordinates (x1, y1) of the UI image "Direct Input" on the FAX top screen 3000 shown in Figure 6(a). For example, it is written by a script that calls the click function in PYAutoGUI. The format of the click function is as follows: it generates the operation event specified by the clicks argument at the coordinates (x, y) specified by the argument, for the time interval specified by the interval argument. click(x,y,button,clicks=right / left,interval) Step 2 is written by a script that instructs the system to generate an event that inputs the string "03-XXX-XXXX," for example, a script that calls the write and press functions in PYAutoGUI.

[0069] The format of the write function is as follows, and it outputs the string specified by the argument.

[0070] write("string"): The format of the press function is as follows: Enter the key code of the key specified by the argument KEYBOARD_KEY.

[0071] press("KEYBOARD_KEY") Step 3 consists of a script that instructs the system to generate a click event at the coordinates (x2, y2) of the location where the image "Original Quality Text" is located on the FAX top screen 3000. Step 3, like Step 1, consists of a script that calls the click function.

[0072] Step 4 consists of a script that instructs the system to generate a click event at the coordinates (x3, y3) of the location of the image "text / photo" on the desktop screen. For example, a script that calls the click function, similar to Step 1. The RPA tool 2500 interprets and executes this Step 4 script, issuing an MFP coordinate instruction to the MFP panel application 1400 indicating that the coordinates (x3, y3) have been touched.

[0073] Step 5 consists of a script that displays the menu (RPA menu) shown in Figure 6(b). The script in Step 5 calls the following functions.

[0074] Prompt(text="string", title="string", default="string") This function pops up a menu with the title specified by the argument `title`, displays the string specified by the argument `text`, accepts text input, and accepts input via the OK and Cancel buttons. An example of the menu displayed in step 5 is shown in Figure 6(b). This menu includes the text string "Have you finished sending the fax?" 3210 and the text string "Send a screenshot of the transmission report screen to the email address" 3220. The OK button 3230 accepts the instruction that the fax transmission is complete, and the Cancel button 3240 accepts the instruction to cancel the sending of the transmission report.

[0075] Step 6 involves taking a screenshot of the submitted report screen. The screenshot is taken using the following function.

[0076] Screenshot("filename", region=(top-left coordinates, width, height)) This script takes a screenshot of the specified width x height area starting from the top-left coordinates specified by the argument `region`, and saves the resulting pixels with the filename specified by the argument. Saving screen data sent from the MFP1000 is performed using this screenshot. In faxing, hundreds of pages of documents may be sent, and the sending of the transmission report step in the RPA script data may take time. Therefore, a script (wait setting) is inserted before step 6 to wait for processing on the MFP side. The wait setting specifies a waiting time in seconds, during which the RPA tool waits. This delays the issuance of scripts for subsequent steps.

[0077] By inserting a script that waits for processing on the MFP side, the script for step 6 is issued to the RPA tool 2500 only after this waiting period has elapsed.

[0078] Step 7 is the procedure for sending the screenshot image file to the recipient aaa@bbb.co.jp.

[0079] [6] Processing procedure using RPA tool 2500 Figure 7 shows the procedure for executing RPA script data using RPA Tool 2500. Figure 7 is a flowchart of the processing procedure of RPA Tool 2500.

[0080] The variable i is a variable that indicates the individual steps shown in a single RPA script column, and it varies in the range from 1 to m. In this flow, this variable i is used as a control variable.

[0081] The processing procedure by RPA tool 2500 has the following structure: it determines whether or not it is in RPA operation mode (step S200), if it is in RPA operation mode (Yes in step S200), it connects to MFP1000 (step S201), initializes the variable i to 1 (step S202), and executes the loop from steps S203 to S209; if it is not in RPA operation mode (No in step S200), it executes general RPA processing (step S213).

[0082] The process repeated in the loop from steps S203 to S209 is as follows: The script that constitutes the i-th step in the RPA script data is executed to cause the remote panel application 2400 to perform operations on the remote panel (step S203).

[0083] When i=1 and the script corresponding to step 1 in Figure 5 is executed, the remote panel application 2400 sends coordinate information indicating the coordinates (x1, y1) of the UI image to the MFP panel application 1400 of the MFP 1000. The MFP panel application 1400 instructs the remote panel application 2400 of the terminal device 2000 to perform the process that should be executed when the UI image is input, in this case to switch the screen to the FAX number input screen 3100. The remote panel application 2400 instructs the MFP panel application 1400 to transition from the FAX top screen 3000 in Figure 6(a) to the FAX number input screen 3100. Image data showing the screen after the transition is sent to the remote panel application 2400, and the FAX top screen 3000 in Figure 6 switches to the FAX number input screen 3100.

[0084] The loop from steps S204 to S205 is repeated. The loop from steps S204 to S205 repeatedly checks whether an interrupt has been notified from the MFP1000 (step S204) and whether the transmission cycle for the progress report request message has elapsed (step S205). The interrupt in step S204 refers to another user pressing the stop key on the MFP1000, stopping the execution of the currently running job, and then commanding a job by that other user. In step S214, the system waits for the interrupt processing to finish. On the other hand, when the cycle for sending the progress status report request message arrives (Yes in step S205), the progress status report request message is sent (step S206), and the system waits for the receipt of the progress status report message (step S207). Once the progress status report is received (Yes in step S207), the system determines whether the received progress status report indicates completion of processing (step S208).

[0085] If completion of processing is not reported (No in step S208), return to step S204 and repeat steps S204-S205. Steps S204-S208 are repeated as long as the incompleteness of the i-th script is reported.

[0086] When the completion of the i-th script is notified (Yes in step S208), it is determined whether there was a wait instruction (step S209). If there is no wait instruction (No in step S209), the process proceeds to step S211. Step S211 is to determine whether variable i is less than or equal to variable m. If variable i is less than m (Yes in step S211), variable i is incremented (step S212), and the next script is executed.

[0087] If the i-th script has a wait specified (Yes in step S209), step S210 determines whether the wait time has elapsed. If the wait time has not elapsed (No in step S210), the process returns to step S204 and continues the loop from steps S204 to S205.

[0088] If the wait time elapses (Yes in step S210), the process proceeds to step S211. Step S211 determines whether the variable i falls below m. By repeating steps S203 to S212, even if the wait time elapses in the script, the process can wait for completion in the MFP1000.

[0089] (6-1) Processing procedure for the progress status confirmation app 1700 Figure 8 is a flowchart showing the procedure for sending a progress report request message to the MFP. The system waits for a connection from the RPA tool (No in step S400), and once a connection is established (Yes in step S400), it displays "RPA operation mode" on the screen (step S401). After initializing variable i to 1 (step S402), it determines whether an instruction based on MFP coordinates has been issued (step S403). If it has been issued (Yes in step S403), the system proceeds to the loop of steps S404-S405. In this loop, the system repeatedly determines whether an interrupt has occurred (step S404) and whether the i-th progress report request message has been received (step S405). If an interrupt has occurred (Yes in step S404), the system notifies the RPA tool 2500 of the interrupt (step S411) and executes the interrupt (step S412).

[0090] When a progress report request message is received (Yes in step S405), it is determined whether a panel screen response message has been sent to the remote panel application 2400 for the i-th step (step S406). If a panel screen response message has been sent (Yes in step S406), a message indicating that the execution of script i has been completed is displayed on the MFP panel (step S407), and a message indicating that the i-th task has been completed is sent to the RPA tool 2500 (step S408).

[0091] On the other hand, if the process is incomplete (No in step S406), return "incomplete" to the RPA tool (step S409).

[0092] Figure 9(a) shows the remote panel displayed on terminal device 2000 when RPA script data is being executed. As shown in this figure, an indicator 3600 indicating that RPA is running is displayed at the top of the screen, and a stop button 3700 that accepts stopping the execution of the RPA script data and a button 3800 that accepts restarting after stopping are located at the bottom. Even when the RPA tool 2500 is executing the RPA script data, the user can monitor whether the RPA script data is being executed correctly by looking at this remote panel, and if there is a problem, the execution of the RPA script data can be stopped by pressing the stop button 3700.

[0093] Figure 9(b) shows the remote panel after the execution of the RPA script data has been completed. As shown in this figure, indicator 3600 is switched to RPA processing complete, indicating that the execution of the RPA script data has been completed.

[0094] (6-2) Summary Even when the RPA is in a long wait state, the communication of progress report request messages 2432, 2433, 2434, 2435 and incomplete messages 1032, 1033, 1034, 1035 is performed at intervals shorter than the timeout value for periods of no input, as shown in Figure 4(b). As a result, the MFP panel application 1400 remains in a state where it accepts instructions from the remote panel without switching from the remote panel to the local panel. This ensures that even if the processing on the MFP 1000 side takes a long time and the RPA menu is waiting for the MFP 1000 side to complete, the input to the MFP panel application 1400 does not switch from the remote panel to the local panel. Therefore, even after the MFP 1000 has finished processing after a long time, the series of steps described in the RPA script data are executed without interruption.

[0095] [7] Second embodiment In the first embodiment, the MFP 1000 receives a progress report indicating whether the process commanded by the RPA tool 2500 has been completed, and a decision is made on whether to instruct the next step. In contrast, in the second embodiment, the contents of the MFP 1000's touch panel display 1001 are transmitted to the remote panel application 2400.

[0096] (7-1) Communication sequence in the second embodiment Figure 10(a) shows the communication sequence between the RPA tool 2500, the remote panel application 2400, and the progress confirmation application 1700 in the second embodiment. This figure is based on the sequence diagram in Figure 4(b), and depicts only the portion of the sequence diagram in Figure 4(b) from when the instruction message 2431 for the kth step is issued and the MFP coordinate instruction 2331 is issued, until when the instruction message 2432 for the (k+1)th step is issued and the MFP coordinate instruction 2433 is issued. During this time, in the second embodiment, the progress confirmation application 1700 sends screen data 1061, 1062, 1063, and 1065 to the RPA tool 2500 indicating that processing is in progress. It also sends screen data 1065 indicating that processing is complete. While this screen data transmission is being repeated, the operation input to the MFP panel application 1400 is switched to the remote panel side. Therefore, when the image data 1064 is transmitted, the RPA tool 2500, upon completion of processing on the MFP side, instructs the remote panel to perform the operation of the next step in the RPA script data, which is the (k+1)th step, and to perform the corresponding processing.

[0097] The content of the image data 1061-1064 transmitted with each transmission cycle is as shown in Figure 10(b), consisting of the text "Now Loading" surrounded by nine circles, the color of which changes for each image data. The remote panel application 2400 displays this image data on the remote panel, making it clear that processing is currently being performed on the image processing device.

[0098] Subsequently, in Figure 10(a), all the steps described in the RPA script data are executed, and when the RPA tool 2500 notifies the RPA operation mode to be canceled 2433, the progress confirmation application 1700 terminates the transmission of image data showing the operation screen of the touch panel display 1001. By terminating the transmission of image data, the period of no input continues, and when the period of no input reaches the timeout period Ct, the operation system control unit 1600 switches the operation input to the MFP panel application 1400 from the remote panel to the touch panel display 1001.

[0099] (7-2) Processing procedure for the progress status confirmation app 1700 Figure 11 is a flowchart showing the processing procedure of the progress confirmation application 1700 according to the second embodiment. This flowchart is based on the flowchart in Figure 8, with the changes being that in the loop consisting of steps S404 to S405 in Figure 8, step S405 is replaced with step S421, and step S422 is added as the processing when step S421 becomes Yes.

[0100] Step S421 is a determination of whether the transmission cycle Q shown in Figure 10 has elapsed. If it has elapsed (Yes in step S421), image data showing the screen content of the touch panel display 1001 is sent to the remote panel application 2400 (step S422). The progress confirmation application 1700 repeats these transmissions, so that image data is sent to the remote panel application 2400 at a cycle shorter than the period of no input.

[0101] The remote panel application 2400 of the terminal device 2000 receives image data transmitted from the progress confirmation application 1700 and displays it on the LCD 2001.

[0102] The following describes the RPA script data 2600 and RPA tool 2500 used in terminal device 2000.

[0103] (7-3) RPA script data in the second embodiment In the first embodiment, the RPA script data 2600 explicitly specified a waiting time in seconds and delayed the issuance of operation instructions by specifying a wait time for the image processing device 1000 to process for that amount of time. In contrast, in the second embodiment, the loop continues indefinitely unless certain conditions are met.

[0104] Each time image data showing the operation screen of the MFP1000 is transmitted, the RPA tool 2500 determines whether a certain condition for exiting the loop of the RPA script data 2600 has been met. This condition for exiting the loop is that the image data transmitted in each transmission cycle has changed significantly from the previous image data, and it is determined that the operation screen of the image processing device 1000 has changed.

[0105] Specifically, the RPA tool 2500, each time it receives new image data, compares the received image data with the previously received image data and counts how many pixels are different. It then calculates the ratio of these different pixels to the total area of ​​the screen. Finally, it determines whether the calculated ratio exceeds a threshold.

[0106] When an animation indicating a processing delay is displayed, as shown in image data 1061-1064 in Figure 10(b), each transmitted image data contains pixels with different content. However, in the animation in Figure 10(b), the differences between individual screens are only differences in the color of some circles, and the ratio of pixels with changed content to the entire screen is below a threshold.

[0107] On the other hand, when the content of the image data changes from image data 1961-1064 showing an animation as shown in Figure 10(b) to image data 1065 showing the operation screen, the ratio of the pixels whose content has changed to the entire screen will be above a threshold.

[0108] By calculating these ratios and comparing them with thresholds, the switching of the operation screen in the MFP1000 is determined.

[0109] As described above, according to this embodiment, the next operation is executed without waiting for the specified waiting time to elapse, depending on whether the content of the image data has changed significantly from the previous image data. Therefore, each step defined in the RPA script data can be executed by the MFP1000 at the appropriate timing.

[0110] [8] Third Implementation In this embodiment, the initial setup of the MFP1000 when it is installed at a business site is automated using RPA. Figures 12 to 14 show examples of RPA script data for performing the initial setup. These scripts are intended to operate the setting screen shown in Figure 15. Figure 15 shows the screen used for setting up the MFP1000 when it is installed. This screen includes a service screen 5100 and an administrator screen 5200. These include various setting items, and service technicians need to set them sequentially when installing the MFP1000.

[0111] (8-1) Screens to be configured The service screen 5100 in Figure 15 includes serial number 5110, password 5120, authentication settings 5130, first system settings 5140, and second system settings 5150. Clicking serial number 5110 brings up the display keypad 5111, and clicking authentication settings 5130 brings up the CE authentication function 5131 and administrator password 5132.

[0112] The first system setting 5140 can access a menu that includes destination setting 5141 and serial number input 5142. The second system setting 5150 can access a menu that includes software switch setting 5151, ADF installation setting 5152, and storage type setting 5153.

[0113] The administrator screen 5200 includes a settings menu 5210 and security settings 5220. The settings menu 5210 can access administrator settings 5211, and security settings 5220 can access simplified security settings 5221. (8-2) RPA script data for device configuration Figure 12 shows a series of scripts that implement the initial setup of the device, from ROM version verification to CE certification function 5131, destination setting, and software SW setting. Step 1 consists of a script that displays the firmware version installed in ROM1008 of the MFP1000. Step 2 includes a script that downloads the appropriate firmware version for the MFP1000 from a URL that accesses the MFP1000 manufacturer's official website and writes it to the memory card connected to terminal device 2000. This script writes the firmware to the memory card, and the operator connects the memory card with the new firmware to the MFP1000 to perform the firmware upgrade from the memory card. Once the firmware upgrade from the memory card is complete, the operator completes the process by operating the screen presented in this procedure. Since downloading and copying the firmware takes time, a wait script is inserted before issuing Step 3 to allow the MFP1000 to complete its processing.

[0114] Step 3 consists of a procedure to generate a click event at coordinates (x1, y1) of serial number 5110 from the service screen, and Step 4 consists of a script that generates a text input event.

[0115] Step 5 involves triggering a click event from the service screen 5100 at the coordinates (x2, y2) of the authentication settings 5130.

[0116] Step 6 generates a click event from the service screen 5100 at coordinates (x3, y3) of the location of the CE authentication function 5131, and Step 7 generates a click event from the service screen 5100 at coordinates (x4, y4) of the location of the first system setting 5140.

[0117] Step 8 consists of a script that generates a click event at the coordinates (x5, y5) of the location from the first system setting 5140 to the destination setting 5141, and Step 9 consists of a script that generates an event indicating that the destination should be entered as text.

[0118] Step 10 consists of a script that generates a click event at the coordinates (x6, y6) of the location of the second system setting 5150 from the service screen 5100, and Step 11 consists of a script that generates a click event at the coordinates (x7, y7) of the software switch setting 5151 and inputs the setting value of the software SW as text.

[0119] In the process described above, backing up the firmware to be stored in ROM 1008 requires the user's expert judgment, and since it requires user judgment and manual operation, it is time-consuming. Because the RPA tool 2500 and the MFP panel application 1400 repeatedly send the progress report request message and the progress report shown in Figure 4(b), even if the user has trouble entering the serial number, the operation input to the MFP panel application 1400 does not switch to the touch panel display 1001, and the system maintains a state where it accepts operation input from the remote panel application 2400.

[0120] Figure 13 shows a series of steps that accept the serial number input, administrator password 5132, ADF installation setting 5152, and storage type setting 5153. Step 12 consists of a script that presents and prompts the user to input the serial number 5142. The script in step 12 calls the following function of PYAutoGUI.

[0121] Prompt(text="string", title="string", default=") This function pops up a menu with the argument `title` as the title, displays the string specified by the argument `text`, accepts text input, and accepts input from OK and Cancel buttons. Serial number input is handled using this prompt.

[0122] Since entering the serial number on the input screen takes time, a wait instruction is inserted before issuing the serial number in step 13 to indicate that the MFP1000 will wait for processing to complete.

[0123] Step 13 consists of a script that triggers a click event at the coordinates (x2, y2) of the service screen 5100.

[0124] Step 14 involves a script that generates a click event at coordinates (x12, y12) with administrator password 5132, and an event that prompts the user to enter a password that is common to all devices. Step 15 consists of a script that triggers a click event at the coordinates (x6, y6) of the location of the second system setting 5150.

[0125] Step 16 consists of a script that generates a click event at the coordinates (x14, y14) of the location of ADF mounting setting 5152.

[0126] Step 17 consists of a script that generates a click event at the coordinates (x15, y15) of the location where the ADF is installed.

[0127] Step 18 consists of a script that triggers a click event at the coordinates (x16, y16) of storage type setting 5153. Step 19 consists of a script that triggers a click event at the coordinates (x17, y17) of the location of "Installed Storage".

[0128] Of the information entered through the above procedure, the serial number is unique to the device and must be prepared separately from the manual, packaging, etc., and entered by the user using keyboard 2003. Therefore, in the script column of Figure 13, the serial number input screen is presented in step 12 to prompt the user to enter the serial number, and then the wait specification is executed. In this way, the sending of the progress report request message and the sending of the progress report shown in Figure 4(b) are repeatedly performed between the RPA tool 2500 and the MFP panel application 1400. As a result, even if the user has trouble entering the serial number, the operation input to the MFP panel application 1400 does not switch to the touch panel display 1001, and the state of accepting operation input from the remote panel application 2400 is maintained.

[0129] Figure 14 shows a series of steps from restarting the system to configuring administrator settings and security settings. Step 20 presents the restart instruction screen and prompts the user for input.

[0130] The script in step 20 consists of calling the Alert function.

[0131] Step 21 generates a click event at the coordinates (x21, y21) of the call button on the administrator screen 5200. Step 22 generates a click event at the coordinates (x22, y22) of the settings menu 5210. Step 23 generates a click event at the coordinates (x23, y23) of the administrator settings 5211. Step 24 generates a click event at the coordinates (x24, y24) of the security settings 5220. Step 25 generates a click event at the coordinates (x25, y25) of the easy security settings 5221. Step 26 generates a click event at the coordinates (x26, y26) of the security warning display. Step 27 generates a click event at the coordinates (x27, y27) of "OFF".

[0132] In the above procedure, restarting the MFP1000 requires manual action from the user. Therefore, in the script column of Figure 14, a restart instruction 5230 is presented in step 20 to the user to perform the restart operation, and then the wait instruction is executed. In this way, the RPA tool 2500 and the MFP panel application 1400 repeatedly send the progress report request message and the progress report shown in Figure 4(b), so even if the user has trouble restarting, the operation input to the MFP panel application 1400 does not switch to the touch panel display 1001, and the state of accepting operation input from the remote panel application 2400 is maintained.

[0133] (8-3) Summary As described above, according to this embodiment, various settings for introducing an image processing device can be automatically input, set, and changed using RPA script data.

[0134] [9] Fourth Embodiment In the first embodiment, the RPA tool 2500 receives a progress report from the MFP 1000 indicating whether the process it commanded has been completed, and then decides whether to instruct the next step. In contrast, in the fourth embodiment, the RPA tool 2500 transmits information to maintain the connection status between the terminal device 2000 and the MFP 1000.

[0135] (9-1) Connection status between terminal device 2000 and MFP1000 The connection status between terminal device 2000 and MFP1000 is that a TCP connection has been established in the protocol stack shown in Figure 3(b).

[0136] At the physical layer, because the connection is via short-range communication, the TCP connection timeout period is set to be relatively short. Since such a TCP connection timeout results in a timeout in remote operation mode, this embodiment performs control to maintain the connection state.

[0137] (9-2) Sequence according to the fourth embodiment Figure 16 shows a sequence diagram according to the fourth embodiment. This figure is based on Figure 4(b). The difference from Figure 4(b) is that although the RPA tool 2500 sends connection status maintenance data 2451, 2452, 2453, and 2454 each time a period R has elapsed, the progress status confirmation application 1700 does not send a progress status report message, resulting in one-way transmission. In this way, ICMP commands and Ping commands can be used as information for maintaining the TCP connection.

[0138] Figure 17 is a flowchart showing the processing procedure of the RPA tool 2500 according to the fourth embodiment. This figure is based on Figure 7, and only the differences are shown. The differences from Figure 7 are that step S205 is replaced with step S221, and steps S222 and S223 are added as steps that are executed when step S221 becomes Yes, instead of steps S206 and S207.

[0139] Step S221 determines whether the transmission cycle for maintaining the TCP connection has arrived. If the transmission cycle has not yet arrived (No in step S221), the loop of steps S204 and S221 is repeated. If the transmission cycle has arrived (Yes in step S221), the connection status maintenance data is sent (step S222), and then in step S223, it is determined whether the remote panel application 2400 has received a panel screen response message. If it has not been received (No in step S223), the process proceeds to step S204. If the panel screen response message has already been received (Yes in step S223), the process proceeds to step S209.

[0140] (9-3) Summary As described above, according to this embodiment, a TCP connection is maintained between the remote panel application 2400 and the RPA tool 2500, so that the operation input to the MFP panel application 1400 does not switch to the touch panel display 1001, and the state of accepting operation input from the remote panel application 2400 is maintained.

[0141]

[10] Fifth embodiment In the first embodiment, status information indicating whether the process commanded by the RPA tool 2500 was completed was received from the MFP 1000, and a decision was made on whether to instruct the next step. In contrast, in this embodiment, the progress of the MFP 1000's processing is indicated by a progress report message from the MFP 1000.

[0142] (10-1) Flowchart for the progress tracking app In this embodiment, the progress confirmation application 1300 checks the progress status according to the flowcharts in Figures 18 and 19.

[0143] Figure 18 is a flowchart detailing the progress status determination. Step S501 determines whether the panel is being configured, and step S502 determines whether the instructed task is a job.

[0144] If the panel is being configured (Yes in step S501), the progress report in step S505 will be set to "Panel configuration in progress". If the panel is not being configured (No in step S501) and the panel configuration is complete, in step S502 it will be determined whether the instructed task is a job or initial setup. If it is a job (Yes in step S502), steps S503 and S504 will be executed.

[0145] Step S503 determines whether a job is currently running, and step S504 determines whether the job type is a print job.

[0146] There are two types of jobs: those that primarily involve printing image data (print-based jobs) and those that do not involve printing image data but primarily involve generating and transmitting image data (non-print-based jobs).

[0147] If the progress report message indicates that the job is running and it is not a print system, step S503 is Yes and step S504 is No, and the progress report message is set to Panel settings complete, Job running, Non-print system (step S508). If the progress report indicates that the job is running and it is a print system, step S503 is Yes and step S504 is Yes, and the progress report message is set to Panel settings complete, Job running, Print system (step S507). If the job is not running (No in step S503), the progress report message is set to Panel settings complete, Job execution complete (step S506).

[0148] Figure 19 shows the processing procedure for the progress report when No is selected in step S502 and the initial settings are completed.

[0149] Step S510 determines whether the instructed task is an initial setup, step S511 determines whether an initial setup is in progress, step S512 determines whether a restart is required, and step S513 determines whether a restart is in progress.

[0150] If initial setup is in progress, step S511 will be Yes, and in step S521, the progress report message will be Panel setup complete - Initial setup in progress. If initial setup is complete but a restart is not required, step S511 will be No, step S512 will be No, and in step S522, the progress report message will be Panel setup complete - Initial setup complete - No restart required. If initial setup is complete and a restart is required but the system is restarting (Yes in step S513), in step S523, the progress report message will be Panel setup complete - Initial setup complete - Restarting in progress. The time required for the restart will also be included in the progress report message. If initial setup is complete, a restart is required, and the restart is complete (No in step S513), in step 524, the progress report message will be Initial setup complete - Restart required - Restart complete.

[0151] (10-2) Flowchart of RPA tool 2500 according to the fifth embodiment Figure 20 shows the processing procedure of the RPA tool 2500 according to the fifth embodiment. The flowchart in Figure 20 differs from that in Figure 7 in that steps S240 to S243 are provided instead of steps S205 to S201.

[0152] In this embodiment, if a wait time is specified, the system proceeds to a loop of steps S204, 240, and S242. Step S240 determines whether the cycle for sending the progress status report request message has arrived. Step S241 sends the progress status report request message, and step S242 determines whether the progress status report message has been received. If the cycle for sending the progress status request has not arrived, the loop of steps S204 to S240 is repeated. If the transmission cycle has arrived (Yes in step S240), the progress status report request message is sent (step S241), and it is determined whether the progress status report message has been received (step S242). If the progress status report message has not been received (No in step S242), the system returns to step S204. If it has been received (Yes in step S242), the system proceeds to steps S243 to S245.

[0153] Step S243 determines whether the progress status shown in the progress report indicates panel setup in progress, step S246 determines whether the panel setup is complete, and step S247 determines whether it is a print system or not. If the progress report message indicates panel setup, panel setup complete, or job complete, then step S243 becomes No, step S245 becomes Job, step S246 becomes Yes, and the process proceeds to step S211.

[0154] If the progress report message indicates panel setup, panel setup complete, job running, or print system, then step S243 becomes No, step S245 becomes Job, step S246 becomes No, step S247 becomes Yes, and the process proceeds to step S211. In this case, it is assumed that the next step can be instructed.

[0155] If the progress report message indicates panel setup complete, job running, and non-print system, then step S243 becomes No, step S245 becomes Job, step S246 becomes No, step S247 becomes No, and the process proceeds to step S204.

[0156] Step S248 determines whether initial setup is in progress. Step S249 determines whether a restart is required, and step S250 determines whether the restart is complete. If panel settings are complete but initial setup is still in progress, step S245 will be No (initial setup), step S248 will be Yes, and the process will proceed to step S204.

[0157] If the panel settings are complete and the initial setup is finished and a restart is not required, step S245 will be No (initial setup), step S248 will be No, step S249 will be No, and the system will proceed to step S211.

[0158] If panel settings are complete, initial setup is complete, and a restart is required but has not yet been completed, then steps S245 will be No (initial setup), S248 will be No, S249 will be Yes, and S250 will be No. In this case, wait for the restart time indicated in the progress report to elapse (No in step S251), and once it has elapsed (Yes in step S251), proceed to step S211. If panel settings are complete, initial setup is complete, and a restart is required but has been completed, then steps S245 will be No (initial setup), S248 will be No, S249 will be Yes, and S250 will be Yes. In this case, proceed to step S211.

[0159] (10-3) Summary As described above, according to this embodiment, the progress of the work on the MFP1000, such as whether the panel settings have been completed, whether a print job is being executed, whether the initial setup is complete and a restart has been performed, is shown in the progress report, so the interval between issuing individual work instructions in the RPA script data can be shortened.

[0160]

[11] Variant Although the present invention has been described above based on embodiments, it goes without saying that the present invention is not limited to the embodiments described above, and the following modifications are possible.

[0161] (1) It is desirable to perform the integration settings before starting the RPA tool 2500. Figure 21 shows the processing procedure for the RPA tool 2500 when performing the integration settings. First, the device linkage settings screen is displayed (step S101), and then the system waits for the settings to be entered (step S102). Figure 22 shows an example of the linkage settings screen. The linkage settings screen includes a checkbox 2101 indicating whether or not to link with the MFP1000, an input field 2102 where the IP address of the MFP1000 can be entered, a username input field 2103, and a password input field 2104, and is configured to accept alphanumeric input from the keyboard. If the save button 2015 is pressed on this screen, the entered user ID and password will be saved, and password entry can be omitted. If the settings are entered (Yes in step S102), the integration settings are executed (step S103), and buttons for creating and executing RPA script data are displayed (step S104). Next, the user is prompted to choose whether to create or execute the touch panel display (step S105). If RPA script data is created, the operations performed on the remote panel 2001D are acquired (step S106), each of the acquired operations is converted into a script (step S107), and the RPA script data consisting of the converted scripts is saved (step S108).

[0162] Subsequently, a screen prompting the user to specify a Wait (step S109) is displayed, and the system waits for a script to be specified that should be set to Wait (step S110). If a script is specified, a Wait specification is inserted before the specified script in the RPA script data (step S111). After the above processing, the system returns to the loop in step S102. If the RPA script data is to be executed, a list of saved RPA script data is displayed (step S121), and it is determined whether the RPA script data has been selected (step S122). If the RPA script data is selected (Yes in step S122), the selected RPA script data is executed (step S123).

[0163] (2) In each embodiment, the remote operation mode is maintained by the RPA tool 2500 and the progress confirmation application 1700 communicating at a transmission cycle shorter than a certain period of time until the remote operation mode times out, but is not limited to this. The remote operation mode may also be maintained by having the input system control unit 1600 perform control specific to the remote operation mode. Specifically, even if there is no communication with the terminal device 2000 while the job execution application 1500 is executing a job in accordance with control instructions made from the terminal device in remote operation mode, the input system will not be switched from the remote panel to the local panel, and the remote operation mode will be maintained. This reduces the amount of data communication between the MFP 1000 and the terminal device 2000.

[0164] (3) In RPA creation mode, the script file containing the RPA script sequence created by the RPA tool 2500 is written to and stored on the HDD2005 of the terminal device 2000, but this is not limited to that. In RPA creation mode, the script file containing the RPA script sequence created by the RPA tool 2500 may be written to the internal storage of the MFP1000. In RPA execution mode, files with extensions specific to script files are read from the HDD1005, which is the internal storage of the MFP1000, and displayed as a list on the LCD2001 of the terminal device 2000. Then, the RPA script sequence selected by the user from the displayed list may be executed.

[0165] In RPA creation mode, the script file containing the RPA script sequence created by RPA Tool 2500 may be written to and stored on the storage of a server device on an external network.

[0166] (4) The scripts can be written using Python®, VBScript®, VB.NET®, GAS®, or C#®.

[0167] VBScript (registered trademark) is a language used in WinActor. VB.NET (registered trademark) is a Visual Basic language used by UiPath and is suitable for manipulating Office products.

[0168] GAS (Google Apps Script) is a programming language developed based on JavaScript, and it can be used to connect with and manipulate services such as Google Maps, email, and spreadsheets.

[0169] C# (registered trademark) is a programming language developed by Microsoft Corporation that allows users to utilize APIs for operating Windows. These are not the only options; any programming language that can be executed without compilation or linking may be used.

[0170] (5) The paper type loaded in the paper cassette may be automatically set by describing the paper settings in RPA script data according to the detection results of the media sensor and having the RPA tool 2500 execute it. The media sensor measures the reflectance of green (G) light, the transmittance of near-infrared light (Ir), the transmittance of blue light (B), and the transmittance of ultrasound using an optical sensor, an infrared sensor, and an ultrasonic sensor, and determines from these reflectances and transmittances whether the sheet used for printing is recycled paper, coated paper, or an envelope. When the media sensor detects a paper type, a string indicating the detected paper type is displayed on the screen, and this string is set to the corresponding paper type in the paper cassette. In this way, the detection result from the media sensor can be reflected in the paper type of the paper cassette without human intervention. Since detection by the media sensor can take time, a wait statement is added before the script that writes the detection result. In this way, even if detection by the media sensor takes time, the operation input to the MFP panel application 1400 will not switch to the touch panel display 1001, and the system will continue to accept operation input from the remote panel application 2400.

[0171] (6) Screen numbers may be used to specify the progress status in the MFP1000. A screen number is a number that indicates each of the hierarchical screens displayed on the touch panel display 1001, and a script that shows the individual steps of the RPA script data indicates the screen number of its target. When the progress status confirmation application 1700 sends a progress status report, it obtains the screen number of the screen currently displayed on the touch panel display 1001 and returns a progress status report including that screen number to the RPA tool 2500. By returning the current screen number in this way, it becomes possible to confirm whether the scripts that make up the RPA script data are ready to be executed, and the RPA tool 2500 can wait for the screen on the MFP1000 to change for sure and then instruct the MFP1000 to execute the next step.

[0172] (7) It is desirable to perform a check to see if the wait time in the RPA script data is shorter than the timeout value for the period of no input. If the specified wait time is longer than the timeout value for the period of no input, as shown in the first embodiment, after instructing the MFP1000 to process, a progress report request message is sent at the above-mentioned intervals. If the specified wait time is shorter than the timeout value for the period of no input, after instructing the MFP1000 to process, the system waits for the wait period to elapse and then sends the above-mentioned progress report request message at the above-mentioned intervals.

[0173] (8) In the terminal device 2000, the pointing device that accepts user input is a mouse or keyboard, but is not limited to these. It may also be a keypad or a stylus.

[0174] (9) Although an MFP (Multifunction Printer) has been described as an example of an image processing device, it is not limited to this. It may also be a single-function scanner or a facsimile machine. It may be installed in a production printing machine, or in a label printer, postcard printer, or ticket machine. It may also be installed in a single-function copier or a single-function peripheral device (printer) for a personal computer. [Industrial applicability]

[0175] This invention allows for the automation of device settings when introducing an image processing device to a business premises and panel settings when executing jobs on the image processing device using RPA. Therefore, it has the potential to be used in various industrial fields, including the office automation equipment and information equipment industries, as well as retail, rental, real estate, advertising, transportation, publishing, and more. [Explanation of symbols]

[0176] 1000 MFP 1000 Image Processing Devices 1000 terminal devices 1001 Touch Panel Display 1005 HDD 1006 RAM 1007 CPU 1008 ROM 1100 Hardware 1200 OS 1300 applications 1300 Progress tracking app 1400 MFP Panel App 1500 Job Execution Apps 1500 Processing Execution Apps 1600 Control System Control Unit 1600 Input System Control Unit 1610 No Input Period Counter 1700 Progress tracking app 2000 Terminal device 2200 OS 2210 kernel 2220 Multi-Window System 2300 applications 2300,1300 applications

Claims

1. A control method for an image processing apparatus having a remote operation mode that receives control instructions from a terminal device, The first step is to switch the image processing device to remote operation mode, The terminal device receives an operation via RPA and issues a control instruction to the image processing device, and the second step includes the following: The image processing device, when it has not communicated with the terminal device for a certain period of time, will deactivate the remote operation mode. The second step involves the terminal device issuing a control instruction to the image processing device, and then repeating communication with the image processing device at intervals shorter than the specified time. A control method characterized by the following:

2. The control method according to claim 1, wherein the repetition of the communication is performed by the terminal device repeatedly requesting the image processing device to report the progress of the processing corresponding to the control instruction at a period shorter than the fixed time, and the image processing device returning a report in response to the request to the terminal device.

3. The aforementioned progress report indicates whether the process is incomplete or completed. The second step, if the report indicates that processing is incomplete, does not perform the subsequent operation; if the report indicates that processing is complete, performs the subsequent operation. The control method according to feature 2.

4. The aforementioned repetition of communication involves repeatedly transmitting image data that shows the contents of the operation screen of the image processing device. The terminal device determines whether the process for which a control instruction was given has been completed by checking whether the content of the operation screen shown in the image data transmitted from the image processing device is the same as the content of the operation screen shown in the image data transmitted immediately before. The control method according to feature 1.

5. The timeout for the remote operation mode is the disconnection of the connection established between the terminal device and the image processing device. Communication with the image processing device involves transmitting information for maintaining the connection from one terminal device to the other within a certain period of time. The control method according to claim 1, which is performed by...

6. The progress status of the process as indicated in the aforementioned report refers to whether the panel settings of the image processing device have been completed or not. The control method according to feature 2.

7. If the progress status of the process indicated by the progress report indicates the completion of the panel settings of the image processing device, the report indicates whether a restart is in progress to reflect the panel settings in the image processing device, or whether the restart has been completed. The control method according to feature 6.

8. If the progress report indicates that a restart is in progress, the report will show the remaining time until the restart is complete. The control method according to feature 7.

9. The progress of the processing indicated in the aforementioned progress report refers to the processing of the image processing device instructed to process Whether the job is completed or incomplete. The control method according to claim 2.

10. The progress status of the process, as indicated in the aforementioned progress report, indicates whether the job for which a control instruction was issued is completed or incomplete, as well as the type of job, whether or not it involves print output. If the job for which the aforementioned control instructions were given is incomplete, but the type of job currently running involves print output, the second step is to perform subsequent operations using RPA. The control method according to feature 9.

11. In the first step, it is further notified that the remote operation mode is a remote operation mode performed by the RPA, When the image processing device receives the notification, it performs a third step of displaying on the control panel of the image processing device that the remote operation mode by RPA will be performed. The control method according to feature 1.

12. In the third step, once the processing corresponding to the control instruction by the RPA is completed, a message to that effect is displayed on the operation panel. The control method according to feature 11.

13. A terminal device that issues control instructions to an image processing device having a remote operation mode, The image processing device is provided with a transition means for switching to remote operation mode, The system includes an instruction means that receives operations via RPA and issues control instructions to the image processing device, When the instruction means has not communicated with the image processing device for a certain period of time, and it is to cancel the remote operation mode, it issues a control instruction to the image processing device and then repeatedly communicates with the image processing device at intervals shorter than the specified period of time. A terminal device characterized by the following features.

14. A program that causes a computer to execute a process of issuing control instructions to an image processing device having a remote operation mode, The first step is to switch the image processing device to remote operation mode, The computer is instructed to perform a second step of receiving an operation via RPA and issuing a control instruction to the image processing device. If a certain period of time has passed without communication with the computer and the image processing device deactivates the remote operation mode, the second step involves issuing a control instruction to the image processing device and then causing the computer to repeatedly communicate with the image processing device at intervals shorter than the aforementioned certain period of time. A program characterized by the following features.

Images