System and Image Forming Apparatus

Abstract

To provide a new technology for remotely controlling an image forming apparatus from an information processing device. [Solution] When the MFP1 of the remote control system 100 receives a command to start cloud remote operation from PC5 via the cloud storage service 200, it uploads an executable file 26 that can be run on PC5's browser 62 to the cloud storage service 200 and starts providing cloud remote operation. When the MFP1 receives a command to start local remote operation from PC7 via local communication, if it is not in a state to provide cloud remote operation, it sends a browser program 84 that can be run on PC7's browser 92 to PC7 and starts providing local remote operation. Furthermore, if the MFP1 receives a command to start local remote operation while it is in a state to provide cloud remote operation, it does not start providing local remote operation.

Description

Technical Field

[0001] The technical field disclosed in this specification relates to a system for controlling an image forming apparatus from an information processing apparatus and an image forming apparatus.

Background Art

[0002] There is known a technology of an image forming apparatus capable of performing image formation such as printing and scanning, and receiving various input operations via an operation panel. For example, Patent Document 1 discloses a function execution apparatus including a printing unit, an image reading unit, and an operation panel, where the operation panel includes a hardware key and a touch panel, and is configured to receive an input operation on the hardware key or an input operation on an icon or button displayed on the touch panel.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, there has been an increasing demand for a so-called remote operation that enables an information processing apparatus to perform an input operation that can be received by the operation panel of an image forming apparatus without touching the operation panel of the image forming apparatus. Patent Document 1 does not disclose a technology for remotely controlling an image forming apparatus, and there is room for improvement.

Means for Solving the Problems

[0005] The system developed to solve this problem includes an image forming apparatus, a first information processing apparatus, and a second information processing apparatus, wherein the first information processing apparatus, the second information processing apparatus, and the image forming apparatus are connected to the Internet, the first information processing apparatus and the second information processing apparatus are capable of installing a browser, and when the image forming apparatus receives a command to start a first remote operation from the first information processing apparatus via cloud storage connected to the Internet, it executes a program upload process to upload a first remote operation program, which can be run in the browser, to the cloud storage, and once the first remote operation program is uploaded to the cloud storage, the browser installed on the first information processing apparatus can cause the first information processing apparatus to download the first remote operation program from the cloud storage, and the image forming apparatus further receives the command to start the first remote operation and then uploads first image data showing the first remote operation screen to the cloud storage The first remote operation program is capable of performing an image upload process to upload images to the cloud storage, and the first remote operation program is capable of causing the first information processing device to download the first image data from the cloud storage and to perform a screen display process to display the first remote operation screen shown in the first image data on the user interface of the first information processing device using the browser, and the first remote operation program is capable of, when it receives an operation on the first remote operation screen via the user interface of the first information processing device, causing the first information processing device to perform an operation upload process to upload first operation data indicating the operation content to the cloud storage, and when the first operation data is uploaded to the cloud storage, the image forming device downloads the first operation data from the cloud storage and performs a process corresponding to the operation shown in the first operation data, and if the operation shown in the downloaded first operation data is an operation related to image formation, it performs an image formation process as the process corresponding to that operation, and the image forming device,When the second information processing device receives a command to start the second remote operation via local communication without using the cloud storage, it executes a program transmission process to send a second remote operation program, which is operable in the browser, to the second information processing device without using the cloud storage. The browser installed in the second information processing device acquires the second remote operation program transmitted from the image forming apparatus. The image forming apparatus can further execute an image transmission process to send a second image data showing the second remote operation screen to the second information processing device without using the cloud storage after receiving the command to start the second remote operation. The second remote operation program, upon receiving the second image data, causes the second information processing device to execute a local screen display process to display the second remote operation screen shown in the received second image data on the user interface of the second information processing device using the browser. The second remote operation program further causes the user interface of the second information processing device to... When the image forming apparatus receives an operation to the second remote operation screen via the cloud storage, it executes an operation transmission process to send second operation data indicating the operation content to the image forming apparatus without going through the cloud storage. When the image forming apparatus receives the second operation data from the second information processing apparatus, it executes a process corresponding to the operation indicated in the received second operation data. If the operation indicated in the received second operation data is an operation related to image forming, it executes an image forming process as the process corresponding to that operation. After the image forming apparatus receives the start instruction for the first remote operation, it starts providing the first remote operation by executing the program upload process, and remains in the first remote operation provision state until the termination conditions for the first remote operation are met. After the image forming apparatus receives the start instruction for the second remote operation, it starts providing the second remote operation by executing the program transmission process, and remains in the second remote operation provision state until the termination conditions for the second remote operation are met. When the image forming apparatus is not in the first remote operation provision state,When the second information processing device receives the start instruction for the second remote operation, the image forming apparatus is configured to enable the provision of the second remote operation. However, when the image forming apparatus is in the first remote operation provision state and receives the start instruction for the second remote operation from the second information processing device, it is configured not to initiate the provision of the second remote operation.

[0006] The first remote operation via cloud storage and the second remote operation via local communication in the system disclosed herein are both functions for remotely operating an image forming apparatus. When the image forming apparatus starts providing the first remote operation, it enters the first remote operation provision state and does not start providing the second remote operation until the termination conditions for the first remote operation are met. Therefore, the image forming apparatus does not start the second remote operation in parallel after starting the first remote operation by the first information processing device. This avoids confusion in operations while in the first remote operation provision state.

[0007] The control methods for realizing the functions of the above system, the image forming apparatus included in the system, the computer programs executed by each apparatus, and the computer-readable storage media that store the computer programs are all novel and useful. [Effects of the Invention]

[0008] The technology disclosed herein enables a new technology for remotely controlling an image forming apparatus from an information processing device. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram illustrating the outline of the remote control system in this configuration. [Figure 2] This is an explanatory diagram illustrating the outline of the remote control system in this configuration. [Figure 3] This is an explanatory diagram illustrating the outline of the remote control system in this configuration. [Figure 4] This is a sequence diagram showing an example of a start instruction procedure. [Figure 5] It is an explanatory diagram showing an example of a data structure. [Figure 6] It is a flowchart showing an example of the procedure of the start determination process. [Figure 7] It is an explanatory diagram showing an example of a license confirmation screen. [Figure 8] It is a sequence diagram showing an example of the screen preparation procedure. [Figure 9] It is an explanatory diagram showing an example of an execution file. [Figure 10] It is a flowchart showing an example of the procedure of the image storage process. [Figure 11] It is an explanatory diagram showing an example of information stored in a structured storage. [Figure 12] It is an explanatory diagram showing an example of a state where screens are synchronized. [Figure 13] It is a sequence diagram showing an example of the remote operation procedure. [Figure 14] It is a flowchart showing an example of the procedure of the browser image change process. [Figure 15] It is a flowchart showing an example of the procedure of the device image change process. [Figure 16] It is a flowchart showing an example of the procedure of the operation information process. [Figure 17] It is a sequence diagram showing an example of the operation end procedure. [Figure 18] It is a sequence diagram showing an example of the local operation procedure. [Figure 19] It is an explanatory diagram showing an example of an HTML file.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments in which the system is embodied will be described in detail with reference to the accompanying drawings. In this specification, a remote operation system that enables an all-in-one machine (hereinafter referred to as "MFP") having an image forming function and a communication function to be operated using a personal computer (hereinafter referred to as "PC") is disclosed.

[0011] The remote operation system 100 of this embodiment includes, for example, as shown in FIGS. 1 to 3, an MFP 1 to be remotely operated, and PCs 5 and 7 for use by a user who remotely operates the MFP 1. The PC 5 is, for example, arranged at the head office, a system management company, or a management organization that manages an operation organization, and the MFP 1 and the PC 7 are, for example, arranged at a branch office, a client company, or an operation organization that operates the business using the MFP 1. The remote operation system 100 is an example of a system.

[0012] Both the MFP 1 and the PC 5 are connected to the Internet and can use the cloud storage service 200 via the Internet. The cloud storage service 200 is a service having a communication function via an Internet line and capable of storing various types of information. Also, the MFP 1 and the PC 7 can perform local communication by local connection. The MFP 1 is an example of an image forming apparatus, the PC 5 is an example of a first information processing apparatus, and the PC 7 is an example of a second information processing apparatus. The remote operation system 100 may further include other MFPs 2, 3 and other PCs 6, 8.

[0013] The MFP 1 of the remote operation system 100 of this embodiment can receive a remote operation via the cloud storage service 200 from a PC 5 or the like that can be connected to the Internet. The MFP 1 can further receive a remote operation without going through the cloud storage service 200 from a locally connected PC 7 or the like. Hereinafter, the remote operation via the cloud storage service 200 will be referred to as "cloud remote operation". Also, a remote operation by local communication without going through the cloud storage service 200 will be referred to as "local remote operation". The cloud remote operation is an example of a first remote operation, and the local remote operation is an example of a second remote operation.

[0014] The MFP1, which is operated by the remote control system 100, includes a controller 10 that includes a CPU 11 and memory 12, as shown in Figure 1, for example. The MFP1 also includes a user interface (hereinafter referred to as "user IF") 13, a communication interface (hereinafter referred to as "communication IF") 14, a print engine 15, and a scanner 16, all of which are electrically connected to the controller 10.

[0015] The CPU 11 of the MFP1 executes various processes according to the program read from memory 12 and based on user operations. The memory 12 of the MFP1 can store various programs and data, including an image processing program 21, display image data 22, a remote control program 23, access information 24, identification information 25, and an executable file 26. Memory 12 is also used as a workspace when various processes are executed. The buffer provided by the CPU 11 is also an example of memory 12.

[0016] The image processing program 21 is a program that causes the MFP1 to perform various image processing operations. The display image data 22 is a data set that shows images to be used as components of various screens displayed on the user interface 13 of the MFP1. The identification information 25 is unique information for identifying the MFP1. The identification information 25 may include, for example, a device ID 251 and a password 252, as shown in Figure 2.

[0017] The remote control program 23, access information 24, and executable file 26 are the information necessary for MFP1 to receive cloud remote control. The remote control program 23 is a program for accepting cloud remote control requests. Access information 24 is information for accessing the cloud storage service 200. The executable file 26 is a data file containing a program for causing PC5, etc., to perform cloud remote control. Details of the programs and data will be described later.

[0018] Note that the access information 24 is not stored in memory 12 when the MFP1 is shipped from the factory. Also, the remote control program 23 and the executable file 26 may or may not be stored in memory 12 when the MFP1 is shipped from the factory.

[0019] Note that the memory 12 is not limited to ROM, RAM, HDD, etc., built into the MFP1, but may also be a storage medium that the CPU 11 can read and write to. A storage medium that a computer can read is a non-transitory medium. In addition to the above examples, non-transitory media also include recording media such as CD-ROMs and DVD-ROMs. Furthermore, non-transitory media are also tangible media. On the other hand, electrical signals that carry programs downloaded from servers on the internet are a type of computer-readable medium, which is a computer-readable signal medium, but they are not included in non-transitory computer-readable storage media.

[0020] The user interface 13 includes hardware that displays a screen to inform the user of information and hardware that accepts user operations. The user interface 13 of the MFP1 includes a touch panel 131 having a screen display function and an operation acceptance function, and a plurality of hardware keys (hereinafter referred to as "hard keys") 132. The user interface 13 is an example of an operation panel, and the touch panel 131 is also an example of a display.

[0021] The communication interface 14 includes hardware for communicating with external devices. The communication interface 14 includes functions that support communication standards such as Wi-Fi®, Ethernet®, and USB. The communication interface 14 can communicate with servers on the internet using the HTTPS protocol.

[0022] The printing engine 15 includes a configuration for printing an image based on image data onto a printing medium such as a sheet. The printing method of the printing engine 15 is, for example, an electrophotographic method or an inkjet method. The scanner 16 includes a configuration for reading an image of a document and acquiring image data.

[0023] The PC 5 for cloud remote control includes, for example, a controller 50 which includes a CPU 51 and memory 52, as shown in Figure 1. The PC 5 also includes a user interface 53 and a communication interface 54, which are electrically connected to the controller 50. The user interface 53 includes hardware that displays a screen for informing the user of information and hardware that accepts user operations. The user interface 53 may or may not have a touch panel. The communication interface 54 has the capability to connect to the internet using the HTTPS protocol.

[0024] The PC5's memory 52 can store various programs and data, including the operating system (hereinafter referred to as "OS") 61, a browser 62, and an application program (hereinafter referred to as "startup application") 63 for assisting in initiating remote operation. Details of the programs and data will be described later.

[0025] Cloud storage service 200 is a service operated by a third party other than the company using MFP1 or the MFP1 vendor, and is equipped with a group of storage accessible via the internet. Cloud storage service 200 includes a first service which has an unstructured storage area where various types of data, such as files containing binary data, can be placed, and a second service which has an area where structured text data can be stored.

[0026] Cloud storage service 200 is, for example, the Azure® storage service provided by Microsoft®. The first service may be a BLOB storage service and the second service a Table storage service. Cloud storage service 200 may also be, for example, Google® Cloud or Amazon® AWS®. Cloud storage service 200 may also be, for example, a service operated by a vendor of image forming equipment.

[0027] In addition to its storage functionality as a data repository, Cloud Storage Service 200 also includes web server functionality. For example, the first and second services of Cloud Storage Service 200 can behave as web servers when they receive HTTPS access, returning responses that can be displayed by a browser.

[0028] For example, a system administrator at a company using the remote control system 100 in this form prepares to use remote operation via the cloud storage service 200. Specifically, the system administrator contracts with the company that operates the cloud storage service 200 and obtains an account name to use the cloud storage service 200. The account name may be the same for the entire company, or it may be different for each country, region, business unit, etc.

[0029] Furthermore, the system administrator sets up storage areas corresponding to the accounts indicated by the acquired account names within the cloud storage service 200, for both the first service and the second service, and obtains tokens for each service. In the following, the storage areas within the cloud storage service 200 that correspond to the accounts used by the remote control system 100, with the storage area set up in the first service being referred to as unstructured storage 201 and the storage area set up in the second service being referred to as structured storage 202. Unstructured storage 201 and structured storage 202 are examples of cloud storage.

[0030] Furthermore, the system administrator must configure at least unstructured storage 201 to prohibit anonymous access. Areas configured to prohibit anonymous access will only accept access requests via URLs with a valid token attached as a query; access will be denied if no query is attached, or if the attached query is incorrect. In addition, even if a file written to an area configured to prohibit anonymous access contains link information to other files, link access using that link information will be prohibited. Each token is, for example, a SAS (Shared Access Signature) token.

[0031] Furthermore, as shown in Figure 2, the system administrator causes each MFP targeted by the remote control system 100 to store access information 24, which includes access information for unstructured storage 241, which is access information for accessing unstructured storage 201, and access information for structured storage 242, which is access information for accessing structured storage 202. The system administrator stores the access information 24 in a non-volatile area of ​​the memory of each MFP.

[0032] The access information 241 for unstructured storage includes, for example, an account name, which is text indicating an account obtained by the system administrator, an unstructured storage name, which is text indicating the unstructured storage 201, and an unstructured storage token, which is a SAS token for accessing the unstructured storage 201, as shown in Figure 2. In other words, the URL for accessing the unstructured storage 201 includes text that combines each of the pieces of information shown in Figure 2 according to a predetermined rule. The access information 241 for unstructured storage may be information written in the form of a URL, or it may be information that includes each piece of information and the rule for combining them.

[0033] The structured storage access information 242 includes, for example, an account name, which is text indicating an account obtained by the system administrator; a structured storage name, which is text indicating structured storage 202; and a structured storage token, which is a SAS token for accessing structured storage 202. The URL for accessing structured storage 202 includes text that combines each of the pieces of information shown in Figure 2 according to a predetermined rule. The structured storage access information 242 may be information written in the form of a URL, or it may be information that includes each piece of information and the rule for combining them.

[0034] Alternatively, each MFP may obtain its own token. For example, the system administrator may have each MFP store an account name, and the MFP may use that account name to access the cloud storage service 200 and obtain its own token. In that case, each MFP includes the obtained token in the access information 24 and stores it in a non-volatile area of ​​memory.

[0035] MFP1 can create access information 241 for unstructured storage and access information 242 for structured storage based on the information it has stored. For example, a system administrator may have MFP1 store account names, and MFP1 may combine those account names and tokens to create access information 241 for unstructured storage and access information 242 for structured storage. In this case, the tokens may be those acquired by each MFP, or those stored by the system administrator in each MFP.

[0036] As will be explained in more detail later, each MFP, when using the remote control system 100, creates a folder in the unstructured storage 201 that includes its own device ID in the folder name. For example, MFP1 creates folder 2011 in A32 of Figure 4, which will be explained later, using the device ID 251 shown in Figure 2. Furthermore, MFP1 combines the unstructured storage access information 241 stored in memory with its own device ID 251 to create access information for accessing this folder 2011 created in the unstructured storage 201 in an unanonymous manner (for example, A33 in Figure 4). For convenience, this access information is referred to as the self-folder access information 243, as shown in Figure 2. The system administrator may also create folders for each MFP in the unstructured storage 201 in advance and store the access information for accessing those folders in each MFP.

[0037] For each MFP, a data storage area is created in the form of a folder whose name includes the device ID. This allows the unstructured storage 201 to return an appropriate response based on the data stored in the folder corresponding to each MFP, even if HTTPS requests specifying different MFPs are made simultaneously from different browsers. In other words, as will be explained in more detail later, cloud remote control of different MFPs can be performed from different PCs using the cloud storage service 200.

[0038] Furthermore, as shown in Figure 1, the system administrator instructs the startup application 63 installed on PC5 to store access information 631 for accessing the structured storage 202. The access information 631 is information used by the startup application 63 to access the area of ​​the structured storage 202 that MFP1 can access using the structured storage access information 242. The access information 631 may be the same as the structured storage access information 242 used by MFP1, or it may be partially different. The startup application 63 of the PC performing cloud remote operation can perform cloud remote operation only for each MFP that uses the structured storage 202 that it can access.

[0039] Regarding access information 631, instead of having the startup application 63 store the URL, it is also possible to have it store the account name and token, and have the startup application 63 create the URL. Alternatively, the startup application 63 may obtain a token using the account name stored on the PC and use the obtained token to create the access information 631. Specifically, the access information 631 is in the format of a URL.

[0040] On the other hand, MFP1 and PC7 are connected to the same branch office LAN, for example, and can communicate via local communication. Local communication includes, for example, local wireless communication, local LAN communication, and wired communication such as USB communication. MFP1 is equipped with programs and data (hereinafter referred to as "EWS") 80 for realizing embedded web server functionality, as shown in Figure 3, for example. MFP1 can accept access to EWS 80 via local communication. Users of PC7 can remotely operate MFP1 via local communication without going through the cloud storage service 200.

[0041] Local wireless communication may refer to a form of communication where the MFP1 and the PC, etc., communicate wirelessly, for example, directly, or a form of communication where they connect wirelessly without using the internet. A form of wireless connection without using the internet may refer to a form where the MFP1 and the PC, etc., connect wirelessly to a single access point and communicate through that access point, for example, by using a wireless LAN protocol. Local LAN communication may refer to a form of communication where the MFP1 connects to the PC, etc., via a LAN cable, or via a LAN cable and a hub, etc., without using the internet, for example, wired LAN communication. USB communication may refer to a form of communication where the MFP1 connects to the PC, etc., via a USB cable, or via a USB cable and a hub, etc.

[0042] As shown in Figure 3, for example, MFP1 further stores cloud state information 81, local state information 82, selected state information 83, and a browser program 84 in memory 12 (see Figure 1). Cloud state information 81, local state information 82, and selected state information 83 are information that indicates the state of MFP1, and each indicates which of several states it is in. The browser program 84 is a program for accepting local remote operations. The browser program 84 is an example of a second remote operation program. Details of each piece of information will be described later.

[0043] The PC7 for local remote control includes, for example, a user interface 93 and a communication interface 94, as shown in Figure 3. Furthermore, a browser 92 is installed on the PC7. The PC7 may or may not be connected to the internet. The PC7 can access the EWS80 of the MFP1 via local communication.

[0044] Next, the procedure for remotely controlling MFP1 will be described. Note that the following processes basically represent CPU processing according to the instructions written in the program. That is, processes such as "judgment," "extraction," "selection," "calculation," "decision," "identification," "acquisition," "reception," and "control" in the following description represent CPU processing. CPU processing also includes hardware control using the OS API. In this specification, the description of the OS is omitted when describing the operation of each program. That is, in the following description, a statement to the effect that "Program B controls hardware C" may also mean "Program B controls hardware C using the OS API." In addition, CPU processing according to the instructions written in the program may be described using abbreviated language. For example, it may be written as "performed by the CPU." Also, CPU processing according to the instructions written in the program may be described using abbreviated language such as "performed by Program A."

[0045] Furthermore, in this specification, terms such as "notification," "information," "notice," "reply," "response," and "answer" are used not only to mean the transmission of information to a person, but also to mean communication and exchange of information between devices or between components within a device. Note that the components within a device include software.

[0046] Furthermore, "acquisition" is used in a sense that does not necessarily require a request. That is, the process of the CPU receiving data without a request is also included in the concept of "the CPU acquiring data." Also, "data" in this specification is represented by a bit sequence that can be read by a computer. Data with the same substantial meaning but different formats will be treated as the same data. The same applies to "information" in this specification. Also, "request" and "instruct" are concepts that indicate that information indicating a request or instruction is being output to the other party. Information indicating a request or instruction may also be simply referred to as "request" or "instruction."

[0047] Furthermore, the process by which the CPU determines whether information A indicates event B is sometimes conceptually described as "determining from information A whether it is event B or not." Similarly, the process by which the CPU determines whether information A indicates event B or event C is sometimes conceptually described as "determining from information A whether it is event B or event C."

[0048] Furthermore, in this specification, setting items may be simply referred to as "settings." Similarly, setting values ​​may also be simply referred to as "settings." Setting values ​​may also be referred to as "parameters." Additionally, storing setting values ​​in memory or elsewhere may be simply referred to as "settings." Furthermore, the operation for setting or the input for setting may also be simply referred to as "settings."

[0049] The procedure for initiating cloud remote control of MFP1 using the remote control system 100 via PC5 will be explained with reference to the sequence diagram in Figure 4. A user who wants to perform cloud remote control, such as a system administrator, launches the start application 63 on PC5, specifies MFP1 as the device to be controlled, and performs an operation to instruct the start of cloud remote control (A01). The start application 63 has the device IDs of each MFP that can be targeted for cloud remote control pre-registered. In A01, the start application 63 can identify the device ID of the device to be controlled based on the user's operation.

[0050] To initiate cloud remote control, a password included in the identification information of the device to be controlled is required. The user will also need to enter the MFP1 password on A01. Password-based authentication enhances the security of cloud remote control.

[0051] The startup application 63 uploads instruction data, which is information indicating the input startup instruction, to the structured storage 202 (A11). As mentioned above, the startup application 63 is equipped with access information 631 for accessing the structured storage 202, and can perform the upload by accessing the structured storage 202 using this access information 631. Thereafter, unless otherwise specified, the startup application 63 will access the structured storage 202 using the access information 631.

[0052] The structured storage 202 stores a structured database with a structure comprising multiple records. Each record includes, for example, a first key 2021, a second key 2022, a timestamp 2023, a device ID 2024, and a parameter 2025, as shown in Figure 5. The structured storage 202 can store the structured database by registering information indicating the structure of the structured database, including the structure of each record, with the system administrator. The startup application 63 may also register information indicating the structure of the structured database with the structured storage 202.

[0053] The first key 2021 and the second key 2022 are information that indicates the type of record content. For example, in A11, the start application 63 uploads each piece of information to the structured storage 202 so that each piece of information is stored in record R1, as shown in Figure 5(A). Note that uploading information can also be rephrased as uploading data. Furthermore, uploading information so that each record stores its information may also be conveniently described as uploading records. Similarly, downloading information can also be rephrased as downloading data.

[0054] When each device, such as PC5 or MFP1, uploads information by specifying the first key 2021 and the second key 2022, the structured storage 202 stores the uploaded information in the record that contains the information specified in both the first key 2021 and the second key 2022. Furthermore, when each device requests to download information stored in the corresponding record by specifying the first key 2021 and the second key 2022, the structured storage 202 sends the information stored in the record that contains both the first key 2021 and the second key 2022 specified in the request to the requester. In other words, the first key 2021 and the second key 2022 also serve as information that identifies the record.

[0055] In other words, the startup application 63 uploads information to the structured storage 202 at A11 so that it becomes record R1 as shown in Figure 5(A). As a result, the first key 2021 of record R1 stores information indicating that it is a record that stores information categorized as an instruction, and the second key 2022 stores information indicating that it is an instruction for the startup process among the information categorized as an instruction. In addition, the device ID 2024 of record R1 stores the device ID that identifies MFP1. Thus, record R1 becomes a record that shows instruction data addressed to MFP1.

[0056] The timestamp 2023 indicates the date and time when data was stored in that record. The device ID 2024 identifies the device to which the data stored in that record pertains; in record R1, the device ID of the device targeted for cloud remote control is stored. Parameter 2025 contains various information necessary for cloud remote control. Each record in this data structure may contain additional information as needed. For example, in the instruction data uploaded in A11, as shown in Figure 5(A), parameter 2025 contains information indicating that the process to be started is cloud remote control, and the password entered in A01.

[0057] Meanwhile, each MFP that may be subject to cloud remote control periodically accesses structured storage 202 using structured storage access information stored in its own memory and checks if there are any instructions that the MFP itself should process by monitoring the record in which its own device ID 251 (see Figure 2) is stored in device ID 2024 (A21). For example, MFP1 periodically accesses structured storage 202 using structured storage access information 242 (see Figure 2). From now on, unless otherwise specified, MFP1 will access structured storage 202 using structured storage access information 242.

[0058] When record R1, shown in Figure 5(A), is uploaded in A11, MFP1 can download the data stored in record R1 from structured storage 202 (A22). Having obtained instruction data indicating instructions for itself from record R1, MFP1 begins processing based on the received instructions. If the instruction data downloaded in A22 is instruction data indicating the start of cloud remote operation, A22 is an example of a start instruction for the first remote operation.

[0059] As shown in Figure 5(B), MFP1 uploads information indicating the progress as "running" to the structured storage 202 so that it stores the information in record R1. Furthermore, since the instruction in record R1 is an instruction to start cloud remote operation, MFP1 performs a start determination process to determine whether or not to allow the start of cloud remote operation (A23).

[0060] Meanwhile, after A11 uploads instruction data indicating the start of cloud remote operation, the start application 63 on PC5 periodically accesses record R1 in structured storage 202 and monitors whether information indicating a response has been stored in record R1 (A12). For example, if information indicating "running" or "completed" is not stored after a predetermined time has elapsed since the upload of record R1, the start application 63 may terminate the cloud remote operation. Note that "completed" is uploaded by MFP1 so that it is stored in record R1 when the processing based on the instruction received by MFP1 is completed.

[0061] MFP1 can accept requests to initiate cloud remote operations by periodically monitoring the structured storage 202, while also being able to accept requests to initiate local remote operations from locally communicating PCs such as PC7 via the communication IF14. For example, MFP1 determines that it has received a request to initiate local remote operations when it accepts access to EWS80 by specifying access information indicating a request to initiate local remote operations. Access to EWS80 is an example of a second remote operation initiation instruction. MFP1 can accept, for example, requests to initiate cloud remote operations from PC5 and requests to initiate local remote operations from PC7.

[0062] For example, the user of PC7 shown in Figure 3 can launch browser 92 on PC7 and specify access information indicating local remote control of MFP1, thereby causing PC7 to access MFP1's EWS80 and send a request to start local remote control. The user of PC7 may also send a request to start local remote control to PC7 via a web page based on web page data sent from EWS80.

[0063] MFP1 can accept both requests to start cloud remote operations from PC5, etc., and requests to start local remote operations from PC7, etc. When MFP1 receives a request to start cloud remote operations, it executes a start decision process at A23. Similarly, as will be described later, MFP1 also executes a start decision process when it receives a request to start local remote operations. The start decision process determines whether or not to start the remote operation in response to the received start request. The procedure for the start decision process will be explained with reference to the flowchart in Figure 6.

[0064] In the start determination process, the CPU 11 checks the cloud status information 81 and local status information 82 stored in memory 12, as shown in Figure 3, and determines whether both indicate "unused" (S101). The cloud status information 81 and local status information 82 are information that indicates either "unused" or "not unused," respectively. "Unused" indicates a state in which cloud remote operation or local remote operation is not being provided and a start request has not been accepted. "Not unused" indicates a state other than "unused," and indicates a state in which each remote operation is being provided, or a state in which a start request has been accepted and a decision is being made on whether or not to provide the remote operation.

[0065] If the CPU determines that at least one of the cloud status information 81 and the local status information 82 is "not unused" (S101: NO), the CPU 11 decides not to start providing the requested remote operation (S102) and terminates the start determination process. In this configuration, the MFP1 will not respond to any other remote operation start requests if it is currently providing or has received a start request for cloud remote operation or local remote operation.

[0066] If both the cloud status information 81 and the local status information 82 are determined to be "unused" (S101: YES), the CPU 11 determines whether the selected status information 83 stored in memory 12 indicates "rejected" (S111), as shown in Figure 3. The selected status information 83 can indicate one of the following: "approved," "under review," or "rejected." "Rejected" indicates that remote operation has been rejected by the MFP1 user via the license confirmation screen 71, which will be described later. "Under review" indicates that the license confirmation screen 71 is displayed and the system is waiting for user input. "Approved" indicates that remote operation has been approved by the MFP1 user via the license confirmation screen 71. If the status is none of the above, the selected status information 83 will not indicate "approved," "under review," or "rejected." In this case, for example, the selected status information 83 may indicate "before review" or "initial."

[0067] If the selection status information 83 is determined to indicate "rejection" (S111: YES), the CPU 11 decides not to start providing the requested remote operation (S102) and terminates the start determination process. When remote operation is rejected by the user of MFP1, MFP1 will not accept either cloud remote operation or local remote operation. Note that the decisions in S101 and S111 may be made in reverse order.

[0068] If the CPU determines that the selected status information 83 does not indicate "rejection" (S111: NO), the CPU 11 changes the status information of the remote operation for which the start command was received to "not unused" and sets the selected status information 83 to indicate "under review" (S112). If the CPU 11 is executing this start determination process because it has received a request to start a cloud remote operation from PC 5, etc., the CPU 11 changes the cloud status information 81 to "not unused". If the CPU 11 is executing this start determination process because it has received a request to start a local remote operation from PC 7, etc., the CPU 11 changes the local status information 82 to "not unused".

[0069] The CPU 11 then displays a license agreement confirmation screen on the user interface 13's touch panel 131 (S114). The CPU 11 displays a license agreement confirmation screen 71 on the touch panel 131, which includes a message asking whether to approve or reject the remote control request, a "Yes" button 711 indicating approval of the remote control, and a "No" button 712 indicating rejection of the remote control, as shown in Figure 7, and accepts input indicating the user's selection.

[0070] The CPU 11 waits until it receives user input. Once it receives user input, the CPU 11 determines whether the user's selection is an approval by pressing the "Yes" button 711 or a rejection by pressing the "No" button 712 (S115). If it determines that the user's selection is a rejection (S115: rejection), the CPU 11 changes the status information, which was changed to "not unused" in S112, to "unused" and sets the selected status information 83 to "rejection" (S116). Furthermore, the CPU 11 returns the display of the touch panel 131 to the state before the license confirmation screen 71 was displayed in S114, decides not to start providing the requested remote control (S102), and terminates the start determination process.

[0071] Subsequently, if a remote operation start request is received while the selection status information 83 is set to "rejected," the CPU 11 determines YES in S111 and does not start providing remote operation. During remote operation provision, the MFP1 is remotely operated by the users of PC5 and PC7. In this configuration, the MFP1 displays a license agreement confirmation screen 71, so users using the MFP1 can ask the remote operation to wait to start if they are displaying a screen they do not want others to see or if they are printing. On the other hand, repeatedly asking for license agreements is troublesome for the MFP1 user. If any remote operation start request is rejected, the MFP1 will treat other remote operation start requests as rejected for the time being, thus avoiding complicated operations.

[0072] Furthermore, if MFP1 accepts a user's rejection and sets the selection status information 83 to "rejected," it automatically deletes the "rejected" information from the selection status information 83 from memory 12 after a certain period, for example, one minute. As a result, if it subsequently receives a request to start remote operation, MFP1 will again accept the user's choice of whether or not to approve the remote operation. It should also be possible for MFP1 to accept a user's instruction to delete the "rejected" information from the selection status information 83.

[0073] On the other hand, if the CPU determines that the user's selection result is approval (S115: Approval), the CPU 11 changes the selection status information 83 to "Approval" (S117) and returns the display on the touch panel 131 to the state before the license confirmation screen 71 was displayed in S114. Furthermore, the CPU 11 decides to start the requested remote operation (S119) and terminates the start determination process.

[0074] Remote operation of the MFP1 when the user is not present may reduce the security of the MFP1. In this configuration, the MFP1 initiates remote operation only when it receives an "Yes" button 711 on the license agreement confirmation screen 71, thus ensuring high security for remote operation. Furthermore, if the user does not input after a predetermined time has elapsed since the license agreement confirmation screen 71 was displayed, the MFP1 may not initiate remote operation. Additionally, the MFP1 may be configured to skip accepting input on the license agreement confirmation screen 71, and if this setting is accepted, approval may be automatically selected. Moreover, the MFP1 may be configured to initiate remote operation if the user does not input after a predetermined time has elapsed.

[0075] Furthermore, in the cloud remote operation start request, as mentioned above, the password entered in A01 of Figure 4 is stored as parameter 2025 shown in Figure 5(A). MFP1 compares the password included in the instruction data downloaded in A22 of Figure 4 with the password 252 of the identification information 25 stored in memory 12 to determine whether it is a correct password, for example, whether the two passwords correspond. If MFP1 determines that it is not a correct password, it is preferable to decide not to start the cloud remote operation without executing the start determination process.

[0076] Note that password 252 may also be the password required to operate MFP1 locally. For example, if a branch office user needs to log in to MFP1 in order to operate it, password 252 may be the password that needs to be entered via user IF13 to log in to MFP1. Also, password 252 may be a password for initiating cloud remote operation and may be different from the password used when using MFP1.

[0077] Regarding the password required to operate MFP1 locally, if local login authentication, which involves password entry, fails more than a predetermined number of times, MFP1 may enter an account lock state. If MFP1 is in an account lock state, CPU11 may decide not to start remote operation without comparing the password stored in record R1 with password 252. Furthermore, if the CPU11 repeatedly determines that the password is incorrect in requests to start cloud remote operation more than a predetermined number of times, CPU11 may also put MFP1 into an account lock state. MFP1 may also automatically release the account lock state based on predetermined triggers, such as the expiration of a predetermined period of time.

[0078] Thus, in the remote control system 100, the MFP1 stores a password 252 for providing cloud remote control, and the MFP1 performs authentication using that password 252, thus providing high security for cloud remote control. On the other hand, in the case of a request to start local remote control, access is accepted using the access information of the MFP1 to the EWS80, so the MFP1 does not need to request a password.

[0079] Returning to the explanation of the start instruction procedure in Figure 4, MFP1 downloads instruction data indicating the start of cloud remote operation at A22. If it decides to start cloud remote operation at S119 of the start decision process in Figure 6 (alt:[Start]), it requests the unstructured storage 201 and structured storage 202 to delete data related to cloud remote operation (A31). For example, if data uploaded by the previous cloud remote operation remains, it may cause confusion with the instruction for the current cloud remote operation. When starting cloud remote operation, MFP1 deletes any data remaining in unstructured storage 201 and structured storage 202, thus avoiding the impact of the previous cloud remote operation.

[0080] Subsequently, MFP1 creates a folder 2011 (see Figure 2) for its own device to be used in this cloud remote control operation in the unstructured storage 201 (A32). For example, MFP1 creates a folder 2011 that includes its own device ID 251 (see Figure 2) in the folder name. If folder 2011 already exists, MFP1 may delete all data in that folder in A31 and skip A32. Furthermore, as mentioned above, MFP1 creates access information 243 for its own folder in the unstructured storage 201 to access folder 2011 created in A32 (A33). The access information 243 for its own folder is in URL format. MFP1 may store the created access information 243 for its own folder in memory 12. From here on, without further explanation, MFP1 accesses folder 2011 created in the unstructured storage 201 using the access information 243 for its own folder.

[0081] Then, MFP1 and PC5 execute a screen preparation procedure (A35) to display a screen that virtually replicates the user interface 13 of MFP1, which accepts cloud remote control, on the user interface 53 of PC5. The screen preparation procedure will be explained with reference to the sequence diagram in Figure 8.

[0082] MFP1 uses the access information 243 for its own folder created in A33 to upload the executable file 26 (see Figure 1) to folder 2011, which was created in the unstructured storage 201 in A32 of Figure 4 (B11). In other words, MFP1 places the executable file 26 in folder 2011.

[0083] The executable file 26 is, for example, an HTML file written in HTML, which includes CSS261 and a browser program (hereinafter referred to as "Browser Program P") 262, as shown in Figure 9. CSS261 is information that sets the layout of the web page to be displayed, such as the color and size of the text, background, and placement, in order to display an image that virtually reproduces the user IF13 of MFP1 in the browser 62 of PC5. Browser Program P262 is, for example, a program written in JavaScript (registered trademark), which is operable in the browser 62 and causes the browser 62 to execute processing for cloud remote control. Browser Program P262 is an example of a first remote control program. B11 is an example of program upload processing.

[0084] Furthermore, the browser P262 of the executable file 26 has structured access information 263 and hard key image data 264 embedded within it. The structured access information 263 is information used by the browser P262 to access the area of ​​the structured storage 202 that is available to the MFP1, and it is in the form of a URL. The structured access information 263 embedded in the executable file 26 may be the same as the structured storage access information 242 stored in the MFP1, or it may be partially different. Note that the structured access information 263 may also be information generated by the CPU 11 of the MFP1 based on the account name, token, etc., included in the structured storage access information 242.

[0085] The image data 264 for the hard keys consists of multiple image files corresponding to each hard key 132 included in the user interface 13, and is described, for example, in GIF format. The image data 264 for the hard keys includes, for example, icon images representing each hard key 132.

[0086] By uploading the executable file 26, which includes structured access information 263 for accessing structured storage 202, PC5, which downloads this executable file 26 in a later step, can obtain browser P262 and structured access information 263. Therefore, based on the processing performed by browser P262, PC5 can access structured storage 202 using structured access information 263.

[0087] Furthermore, the executable file 26 includes, for example, a panel display element 266, a hard key display element 267, and image data 268 for the exit button, as shown in Figure 9. The panel display element 266 is an image embedding element that causes the browser P262 to display a panel image that virtually reproduces the image displayed on the touch panel 131 of the MFP1. The hard key display element 267 is an image embedding element that causes the browser P262 to display key images that virtually reproduce the hard keys 132 of the MFP1. The image data 268 for the exit button is image data that causes the browser P262 to display the exit button 533, which will be described later.

[0088] Furthermore, MFP1 performs image storage processing (B12). The procedure for image storage processing will be explained with reference to the flowchart in Figure 10. Image storage processing is the process of uploading panel image data representing the currently displayed panel image to the cloud storage service 200 in order to display a panel image that virtually reproduces the screen displayed on the touch panel 131 of the MFP1's user interface 13 to the user interface 53 of PC5. Note that image storage processing is executed by the CPU 11 of MFP1 when it is necessary to upload panel image data. The screen displayed on the touch panel 131 is an example of a main unit operation screen and an example of a main unit panel screen.

[0089] The MFP1 can display a screen containing multiple icons on its touch panel 131, and can accept instructions to execute processes indicated by icons by operating on the area where the icons are displayed. For example, if the MFP1 receives an operation on the copy icon included in the standby screen displayed on the touch panel 131, it will display a settings screen that includes an icon for instructing a change in the copy process settings, an input field for the setting value to be changed, and an icon for instructing the execution of the copy process. When the MFP1 receives an instruction operation regarding a setting change, it changes the settings of the copy process. Also, when the MFP1 receives an operation on the icon for instructing the execution of the copy process, it starts the execution of the copy process. Each icon displayed on the touch panel 131 is an example of an operator. Operations on the copy icon, operations on the icon for instructing a setting change, input of a setting value into the input field, operations on the icon for instructing the execution of the copy process, etc., are examples of operations related to image formation, and the copy process is an example of an image formation process.

[0090] Furthermore, when the MFP1 receives an operation on a hard key 132 included in the user interface 13, it can execute processing based on the received hard key 132. For example, if it receives an operation on the back key, the MFP1 can cancel the processing of the previously received operation and return to the state before the operation was received. For example, if the copy processing settings screen is displayed and an operation on the back key is received, the CPU 11 will display the previous screen, for example, the standby screen, on the touch panel 131.

[0091] The CPU 11 first acquires bitmap data representing the screen currently displayed on the touch panel 131 (S201). Here, we will provide a supplementary explanation of bitmap data. When the MFP 1 displays various screens on the touch panel 131, it generates bitmap data representing the screen to be displayed. The MFP 1 stores image data that constitutes the components of each screen in memory 12, for example, as display image data 22 (see Figure 1). The MFP 1 reads image data that constitutes the components of the screen to be displayed, such as image data representing the background image of each screen and image data representing icons, from the display image data 22, and uses the read image data to generate image data representing the screen and bitmap data.

[0092] Furthermore, screens with significantly different roles, such as the standby screen and the copy settings screen, are each assigned a different screen ID, allowing the MFP1 to identify the role of the currently displayed screen based on its screen ID. For example, while the standby screen is displayed, the MFP1 temporarily stores the standby screen ID in memory 12 as the screen ID of the currently displayed screen. When switching to the copy settings screen, the MFP1 generates bitmap data indicating the copy settings screen and temporarily stores the screen ID indicating the copy settings screen in memory 12 as the screen ID of the currently displayed screen.

[0093] Furthermore, if input is received into the setting input field while the copy screen is displayed, MFP1 generates bitmap data showing the copy settings screen with the input field content changed to reflect the entered setting value, without changing the screen ID of the currently displayed screen. In this way, when changing the content of a screen while it is being displayed, for example, by changing the appearance of an icon or changing some text display, MFP1 generates bitmap data showing the screen with the changed content without changing the screen ID of the currently displayed screen. Note that screens with significantly different roles, i.e., different screen IDs, often have significantly different overall layouts, but there are also screens with similar overall layouts even if they have different screen IDs, such as the copy settings screen and the scan settings screen.

[0094] CPU11 generates panel image data (S202), which is image data to be displayed on the PC5's user interface 53 in a later step, based on the bitmap data acquired in S201. For example, CPU11 compresses the acquired bitmap data to generate PNG data.

[0095] Furthermore, the data representing the screen currently displayed on the touch panel 131 is not limited to bitmap data; for example, it may be PNG data or JPEG data. Also, the panel image data is not limited to compressed bitmap data or other screen-representing data; it may be the screen-representing data itself. In addition, instead of creating the panel image data from the screen-representing data, the CPU 11 may acquire image data of components included in the screen currently displayed on the touch panel 131 and create the panel image data using that acquired image data.

[0096] Furthermore, if the displayed screen contains video or animation, the CPU 11 may generate panel image data that does not include the video or animation, or it may generate panel image data with a still image of one scene from the video or animation added. Videos, which tend to have a large amount of data, place a high communication load on the system, and communication delays can easily cause differences in the displayed content between the PC 5 and the MFP 1. In this configuration, the MFP 1 generates panel image data that does not include video, so the user interface 53 on the PC 5 displays an image without video, making display discrepancies less likely.

[0097] The CPU 11 then determines whether the data size of the generated panel image data is smaller than a predetermined size (S205). The panel image data to be uploaded varies depending on the type of screen, for example, in terms of how easily it can be compressed, and the compressed data size may be large or small. For example, screens containing only icons and menus, such as a waiting screen, tend to be able to reduce in size by compression, while screens that include print preview images or scanned image results tend to have a larger data size. In addition, there is a limit set by the cloud storage service 200 on the data size that can be stored in each record of the structured storage 202.

[0098] If the CPU determines that the image data is smaller than a predetermined size (S205: YES), the CPU 11 encodes the panel image data into text data format according to a predetermined rule (S211). Since text data format is suitable for the data stored in each record in structured storage 202, the CPU 11 encodes the panel image data into text data format. However, if the cloud storage service 200 specifies a data format other than text data format that is suitable for storage in structured storage 202, the CPU 11 may encode the panel image data into that data format. Also, if the format of the panel image data is a data format suitable for storage in structured storage 202, the CPU 11 may choose not to encode it.

[0099] Then, CPU 11 uploads the encoded panel image data to structured storage 202 (S212, B13 in Figure 8). Specifically, MFP1 uploads each piece of information to structured storage 202 so that, for example, it becomes record R21 as shown in Figure 11. In other words, MFP1 uploads each piece of information to structured storage 202 so that the first key 2021 stores information indicating that it is a panel image, the second key 2022 stores information indicating that it is image data, the device ID of MFP1 is stored in device ID 2024 of the record, and the panel image data encoded in text data format is stored in binary 2026. S212 and B13 in Figure 8 are examples of image upload processing. The panel image data uploaded in S212 and B13 in Figure 8 are examples of first image data.

[0100] On the other hand, if the CPU determines that the data size of the panel image data is not smaller than a predetermined size, that is, that it is too large to be stored in a record in the structured storage 202 (S205: NO), the CPU 11 uses the aforementioned access information 243 for its own folder (see Figure 2) to access folder 2011 in the unstructured storage 201 and uploads the panel image data to the unstructured storage 201 (S213, B14 in Figure 8). S213 and B14 in Figure 8 are examples of the image upload process. The panel image data uploaded in S213 and B14 in Figure 8 is an example of the first image data.

[0101] In other words, MFP1 creates a file containing panel image data and uploads it to folder 2011, which was created in A32 of Figure 4. Folder 2011 is a folder whose name includes the device ID of MFP1. In this case, MFP1 does not upload the panel image data to structured storage 202. That is, no panel image data is stored in record R21.

[0102] Panel image data uploaded to structured storage 202 or unstructured storage 201 is read by the browser 62 on PC5 using the procedure described later. The panel image data includes image data that virtually reproduces the screen displayed on the user interface 13 of MFP1 on the user interface 53 of PC5. When using the cloud storage service 200, the upload destination is selected according to the size of the image data, so even large panel image data can be passed to the browser 62 via the cloud storage service 200.

[0103] Furthermore, the CPU 11 generates image information, which is information about the uploaded panel image data, and uploads it to the structured storage 202 (S215, B15 in Figure 8). The MFP 1 uploads the image information to the structured storage 202 so that, for example, it becomes record R22 as shown in Figure 11. In other words, the MFP 1 uploads each piece of information to the structured storage 202 so that the first key 2021 stores information indicating that it is a panel image, the second key 2022 stores information indicating that it is image information, the device ID of the MFP 1 is stored in the device ID 2024 of the record, and the parameters 2025 store information indicating the number of times the panel image data has been changed, information indicating the storage location of the panel image data, and information indicating the file name of the panel image data.

[0104] The information indicating the storage location of the panel image data includes information indicating whether the panel image data is stored in record R21 or whether the panel image data has been uploaded to folder 2011 in unstructured storage 201. For convenience, below, the information stored in parameter 2025 of record R22, namely the information indicating the number of times the panel image data has been modified, the information indicating the storage location of the panel image data, and the information indicating the file name of the panel image data, will be referred to as image information.

[0105] The panel image data uploaded in B13 or B14 in Figure 8 is the first panel image data for this cloud remote operation. Therefore, MFP1 sets the number of changes included in the image information uploaded in B15 to "1". As will be explained in more detail later, MFP1 may upload the panel image data again in subsequent steps, and each time it uploads, it stores the number of changes, increased by 1, in the image information record.

[0106] Record R21, which stores the panel image data, and record R22, which stores the image information, are a pair of records for uploading one panel image file. Although not shown in the diagram, structured storage 202 has another pair of records in addition to the record pair of records R21 and R22. When MFP1 uploads information about the panel image data in subsequent uploads, it uploads the information about the panel image data to the record in the pair of records that has fewer stored modification counts. In other words, MFP1 uploads to the two pairs of records alternately. Alternatively, MFP1 could upload the information about the panel image data to the record pair with the older stored timestamp.

[0107] Furthermore, there may be three or more pairs of records that store information about the panel image data. In this case, MFP1 should upload the information about the panel image data to the record pair with the fewest number of changes stored. Alternatively, MFP1 may upload the information about the panel image data to the record pair with the oldest timestamp stored.

[0108] Furthermore, if MFP1 determines that the panel image data is too large to be stored in a record in structured storage 202, it may, instead of uploading it to unstructured storage 201, split it into multiple data files and upload them to multiple records in structured storage 202. For example, MFP1 may split the panel image data into data showing the upper half of the screen and data showing the lower half of the screen, and upload them so that they are stored in two separate records. Alternatively, MFP1 may determine the number of divisions according to the data size of the panel image data. In this case, MFP1 may encode each of the divided panel image data into text data format and store them in multiple records, and store information indicating the number of divisions and information indicating the order of each record as image information. Even in this way, by uploading the divided data according to the data size of the panel image data, large image data can be passed to the browser 62 via the cloud storage service 200 in a later step.

[0109] After S215, the CPU 11 finishes the image storage process shown in Figure 10 and returns to the screen preparation procedure shown in Figure 8. In Figure 8, the procedure for uploading panel image data to structured storage 202 (S212 in Figure 10) in the image storage process B12 is shown as B13, and the procedure for uploading it to unstructured storage 201 (S213 in Figure 10) is shown as B14. In addition, the procedure for uploading image information to structured storage 202 (S215 in Figure 10) is shown as B15.

[0110] Returning to the explanation of the screen preparation procedure in Figure 8, after uploading the executable file 26 in B11 and the panel image data and image information in B12, MFP1 uploads response data to the structured storage 202 (B17) indicating that processing is complete for the instruction data acquired in A22 of Figure 4. After B17, MFP1 remains in a cloud remote operation provision state until the termination conditions for ending the cloud remote operation are met. The cloud remote operation provision state is an example of the first remote operation provision state. Note that MFP1 may also execute the image storage process in B12 after uploading the response data.

[0111] MFP1 uploads response data, for example, to store the response data in record R1 (see Figure 5), which contains the instruction data. Specifically, as shown in Figure 5(C), MFP1 uploads each piece of information so that record R1 stores information indicating "completed" as the progress status, information indicating "success" as the result, and access information for downloading the executable file 26 uploaded in B11 as additional information 2027.

[0112] The access information stored in the additional information 2027 of record R1 is information for accessing folder 2011, which is created in unstructured storage 201 and whose folder name includes the device ID of MFP1. The access information stored in additional information 2027 is in URL format. The access information stored in additional information 2027 may be the same as the access information 243 for the local folder created in A33, or it may be partially different. MFP1 may store the access information 243 for the local folder as is in additional information 2027. Alternatively, MFP1 may create the access information stored in additional information 2027 by combining, for example, the unstructured storage access information 241 stored in memory 12 with the device ID 251.

[0113] Meanwhile, as mentioned above, the PC5 startup application 63 uploads record R1 containing instruction data in A11, and then periodically accesses structured storage 202 to monitor whether there is a response to the instruction in record R1 (A12). If the progress information contained in record R1 changes from "running" to "finished," and if information indicating "success" is stored as a result, the startup application 63 can start cloud remote control.

[0114] The startup application 63 retrieves access information from the additional information 2027 of record R1, which contains information indicating "success" (B21). The startup application 63 then launches a browser 62 and passes the access information retrieved from the additional information 2027 to the launched browser 62 (B22).

[0115] Browser 62 downloads the executable file 26 from folder 2011 in unstructured storage 201 using non-anonymous access with access information passed from startup application 63 (B31). As mentioned above, the executable file 26 contains CSS261, and browser 62 configures the display settings of the web page based on the CSS261 of the obtained executable file 26. The executable file 26 also contains browser P262, so browser 62 starts executing processing based on browser P262 (B32).

[0116] Furthermore, the executable file 26 contains structured access information 263, so the browser 62 uses the structured access information 263 to access the structured storage 202 and download the image information (B33). The image information is the information uploaded by MFP1 in B15. In B33, the browser 62 downloads the image information from the record that stores the image information with a modification count of "1". As mentioned above, MFP1 uploaded record R22 in B15 of Figure 8 so that it stores the image information with a modification count of "1". The browser 62 downloads the image information stored in this record R22.

[0117] In other words, by downloading the executable file 26, the browser 62 can obtain the structured access information 263, and use that structured access information 263 to access the structured storage 202. Therefore, there is no need for the startup application 63 to prepare the structured access information 263. Alternatively, the startup application 63 could generate the structured access information 263 and pass it to the browser 62.

[0118] Browser 62 downloads panel image data from structured storage 202 or unstructured storage 201 based on the information stored in parameter 2025 of image information record R22 (B34 or B35). If the panel image data is stored in structured storage 202 (alt:[structured storage]), Browser 62 downloads the panel image data stored in record R21 of structured storage 202 (B34). If the panel image data is stored in unstructured storage 201 (alt:[unstructured storage]), Browser 62 downloads the panel image data stored in folder 2011 of unstructured storage 201 (B35).

[0119] As mentioned above, if the data size of the panel image data is large, MFP1 may, instead of uploading it to unstructured storage 201, split the panel image data and upload it to structured storage 202. In that case, the browser 62 can obtain the panel image data by downloading the multiple split panel image data from structured storage 202 based on the image information and integrating them based on the image information.

[0120] Based on the downloaded panel image data, the browser 62 displays the panel image 531 at the position corresponding to the panel display element 266 on the user interface 53, for example, as shown in Figure 12 (B37). The panel image 531 is an example of the first remote operation screen, and B37 is an example of the screen display process. The panel image 531 displayed on the user interface 53 of the PC 5 is an image that virtually reproduces the screen currently displayed on the touch panel 131 of the user interface 13 of the MFP 1.

[0121] Furthermore, based on the image data 264 for the hard keys, the browser 62 displays the key image 532 at the position corresponding to the hard key display element 267 on the user interface 53, for example, as shown in Figure 12 (B38). The three key images 532 displayed on the user interface 53 of PC 5 are images corresponding to the three hard keys 132 included in the user interface 13 of MFP 1. As a result, an image that virtually reproduces the user interface 13 of MFP 1, including the touch panel 131 and the hard keys 132, is displayed on the user interface 53 of PC 5.

[0122] Note that while Figure 12 shows an example of displaying a standby screen, the screen displayed in B37 is a virtual reproduction of the screen currently displayed on the MFP1's touch panel 131, and is not limited to the standby screen. For example, if an error occurs in the MFP1 and an error screen is displayed on the touch panel 131, the error screen will also be displayed on the PC5's user interface 53. Also, as mentioned above, if the copy settings screen is displayed on the touch panel 131, the copy settings screen will also be displayed on the PC5's user interface 53.

[0123] The screen displayed to the user interface 53 by the browser 62 includes a panel image 531 that virtually reproduces the screen displayed on the touch panel 131 of the MFP1, and a key image 532 that virtually reproduces the hard keys 132. In other words, since the user interface 53 of the PC5 displays an image that virtually reproduces the user interface 13 of the MFP1, the user of the PC5 can perform operations on the user interface 53 in the following steps that are the same as operations on the MFP1, making cloud remote control easier. It is desirable that the arrangement and shape of the key image 532 displayed on the user interface 53 be the same as that of the user interface 13 of the MFP1, but it is not required to be the same.

[0124] Additionally, the user interface 53 on PC5 displays an exit button 533. The exit button 533 is not included in the user interface 13 of MFP1, and is used by a user performing cloud remote control on PC5 to terminate the cloud remote control.

[0125] This completes the screen preparation procedure shown in Figure 8 (A35 in Figure 4). Let's return to the explanation of the start instruction procedure in Figure 4. Next, we will explain the procedure when it is decided not to start cloud remote operation in the start decision process at A23 in Figure 4. Note that the procedure after the screen preparation procedure when it is decided to start cloud remote operation at A23 in Figure 4 will be described later.

[0126] Even if MFP1 downloads instruction data indicating a request from PC5 to start cloud remote operation, it will decide not to start cloud remote operation if, for example, it determines that the password included in the instruction data is incorrect. Also, as shown in Figure 6, if it is already accepting remote operation requests from other PCs, etc. (NO in S101), or if the user of MFP1 rejects the start of remote operation (YES in S111, or rejected in S115), MFP1 will decide not to start cloud remote operation (S102).

[0127] Then, if the A23 start decision process decides not to start the cloud remote operation (alt:[Do not start]), MFP1 does not upload the executable file 26 and uploads information indicating that it will not start to the structured storage 202 (A41). As shown in Figure 5(D), MFP1 uploads the information such that record R1 stores information indicating "Completed" as the progress status, and information indicating "Failed" as the result. MFP1 may also store information indicating the reason for not starting the cloud remote operation in record R14.

[0128] The PC5 startup application 63 monitors for the presence of response data to the instruction in record R1 (A12), and if the progress of record R1 changes from "running" to "finished", it downloads the response data (A43). If the response data contains information indicating "failure", the startup application 63 does not start the cloud remote operation, for example, it displays a message to user IF53 indicating that it has been rejected (A44). Furthermore, the startup application 63 deletes the instruction data uploaded in A11 from the structured storage 202 (A45). In this case, the cloud remote operation is not started. A user who wants to perform a cloud remote operation can, for example, wait for a predetermined amount of time to elapse, or perform an operation such as re-entering their password, and then re-enter the instruction to start the cloud remote operation in the startup application 63.

[0129] Next, the remote operation procedure, which is the operation procedure after cloud remote operation has started, will be explained with reference to the sequence diagram in Figure 13. The remote operation procedure is executed collaboratively by PC5 and MFP1 after, as shown in Figure 12, an image that virtually reproduces the user IF13 of MFP1 is displayed on the user IF53 of PC5, as shown in B37 and B38 of Figure 8. The browser 62 of PC5 executes the browser image change process shown in Figure 14 based on the description of browser P262 (see Figure 9) included in the executable file 26 downloaded in B31 of Figure 8. MFP1 executes the device image change process shown in Figure 15 based on the remote operation program 23.

[0130] First, the procedure for the browser image change process performed by browser 62 will be explained with reference to the flowchart in Figure 14, and, if necessary, to the sequence diagram in Figure 13. During the execution of the browser image change process, browser 62 periodically accesses structured storage 202 (C03 in Figure 13), checks record R22 (see Figure 11) where the aforementioned image information is stored, and repeatedly determines whether the number of changes included in parameter 2025 has been updated from the number of changes stored by browser 62.

[0131] In the browser image change process, the browser 62 determines whether it has received an operation on any of the panel image 531, key image 532, or exit button 533 displayed to the user IF53 (S311). If it determines that it has received an operation on the user IF53 (S311: YES, C11 in Figure 13), the browser 62 outputs an effect (S312). Specifically, the browser 62 outputs an effect to indicate that an operation has been received by briefly displaying a small animation or still image at the operation location, for example, a mark such as "○", changing the display color of the operation location, or outputting a sound effect. By outputting an effect for the operation, the user who performed the operation on the user IF53 can easily feel that they have performed an operation.

[0132] Furthermore, if the user IF53, who is viewing the panel image 531 shown in Figure 12, receives an operation within the area of ​​the panel image 531, within the area of ​​either the key image 532, or within the area of ​​the exit button 533, the browser 62 will determine YES in S311. For example, if an operation is received at any other location, the browser 62 may ignore the operation or execute other processes that the browser 62 has.

[0133] Browser 62 determines whether the received instruction is an operation to the end button 533, i.e., an instruction to end the cloud remote operation (S321). If it determines that the received instruction is not an instruction to end the cloud remote operation (S321: NO), Browser 62 uploads the operation information indicating the received operation to structured storage 202 so that it is stored in a record indicating the operation content described later (S322, C12 in Figure 13).

[0134] Browser 62 can upload records containing operation information to structured storage 202 using structured access information 263 (see Figure 9) contained in the executable file 26 downloaded at B31 in Figure 8. S322 and C12 in Figure 13 are examples of operation upload processing. The data uploaded to S322 and C12 in Figure 13 are examples of first operation data.

[0135] Furthermore, the browser 62 increments the stored operation number (S323). The operation number also indicates the order of operations. The browser 62 stores the operation number as 1 at the start of the browser image change process, that is, at the start of the remote operation procedure.

[0136] Operation information is information indicating the content of the operation received by PC5. Browser 62 uploads one of several types of records depending on the operation, for example as shown in Figure 11. For example, in records R31~R34 and R41 shown in Figure 11, the first key 2021 of the record that stores operation information stores information indicating that it is cloud remote operation information, and the second key 2022 stores information indicating the operation number. Browser 62 uploads the operation information to structured storage 202 so that the operation information is stored in the record in which the second key contains information corresponding to the stored operation number, for example.

[0137] For example, if the stored operation number is 1 and a click is received on the icon displayed at position A of panel image 531, the browser 62 first uploads operation information corresponding to the click operation as soon as it receives the operation of pressing a mouse button or the like (S322). Specifically, the browser 62 uploads operation information such that the first key 2021 contains information indicating remote operation, the second key contains operation number 1, the device ID of MFP1 is stored in device ID 2024, and the parameter 2025 contains information indicating the operation position and information indicating that it is a downward operation (press operation). Then, the browser 62 adds 1 to the stored operation number, making it 2 (S323). Parameter 2025 in records R31~R34 and R41 is an example of first operation data indicating the content of the operation.

[0138] Next, when the browser 62 receives an operation in which the finger is released from a button or the like during a click, it uploads information corresponding to the release operation (S322). Specifically, the browser 62 uploads operation information such that the first key 2021 contains information indicating remote operation, the second key contains operation number 2, the device ID of MFP1 is stored in device ID 2024, and the parameter 2025 contains the same operation position information as in record R31 and information indicating that it is an upward operation (release operation). Then, the browser 62 adds 1 to the stored operation number, making it 3 (S323).

[0139] Thus, records R31 and R32 store information indicating that a click, a press operation and a release operation were performed at the same operation location, as well as timestamps for the press operation and the release operation, respectively. As will be described later, MFP1 downloads the operation information uploaded by PC5 and processes it according to the downloaded operation information. At that time, MFP1 can distinguish between, for example, a short press and a long press of an icon by comparing the timestamps 2023 stored in records R31 and R32. The information indicating the operation location may be coordinate information within the panel image 531 or within the corresponding touch panel 131 of MFP1, or it may be information identifying the operated button or icon.

[0140] Furthermore, for example, if the stored operation number is 7 and the browser 62 receives an operation request for the key image representing the back key, which is one of the key images 532, the browser 62 uploads the operation information to record R41, which includes information indicating remote operation in the first key 2021 and operation number 7 in the second key, stores the device ID of MFP1 in device ID 2024, and stores information indicating the back key in parameter 2025 (S322). Then, the browser 62 adds 1 to the stored operation number, making it 8 (S323).

[0141] In the case of operations on a key image, the browser 62 uploads operation information such that parameter 2025 stores information indicating the key that was operated, rather than information indicating the direction of operation. The information indicating the key may be information that identifies one of the hard keys 132 of the MFP1 corresponding to the key image 532 that received the operation, or it may be information indicating the coordinates of the corresponding hard key 132. In addition, in order to enable distinction between short presses and long presses for operations on the hard keys 132 of the MFP1, the browser 62 may also upload multiple operation information, including information on the direction of operation, for operations on the key image 532.

[0142] After S323, or if it determines that no operation has been accepted by user IF53 (S311: NO), browser 62 decides whether or not to update the panel image 531 currently displayed to user IF53 (S341). As mentioned above, browser 62 periodically accesses structured storage 202 (C03 in Figure 13), and decides to update panel image 531 if the number of changes stored in the image information has increased from the number of changes stored by browser 62.

[0143] As will be explained in more detail later, the image displayed on the MFP1's touch panel 131 may be changed by remote cloud control from PC5 or by user operation on the MFP1. When the image displayed on the MFP1's touch panel 131 is changed, the MFP1 uploads the updated image information to the structured storage 202 so that the updated image information is stored in record R22 in Figure 11, or in another pair of image information records. The updated image information includes information on the number of changes added by the MFP1.

[0144] Furthermore, the number of changes stored by the browser 62 is the number of changes included in the image information used to display the panel image 531 currently displayed in the browser 62. In other words, if the number of changes stored in the image information is greater than the number of changes stored, it means that the panel image data corresponding to that image information is newer than the panel image data used to display the panel image 531 currently displayed in the browser 62. Note that the browser 62 stores the number of changes as 1 at the start of the browser image change process, that is, at the start of the remote control procedure.

[0145] If it is determined to update the panel image 531 (S341: YES), the browser 62 retrieves the panel image data (S342). Specifically, as shown in C21-C23 of Figure 13, the browser 62 first downloads the image information from record R22 or another pair of image information records (C21), and then, based on the parameters of the image information, downloads the panel image data from a record in structured storage 202 or from folder 2011 in unstructured storage 201 (C22 or C23). C21-C23 in Figure 13 is the same process as B33-B35 in Figure 8.

[0146] If there are three or more pairs of records storing information about the panel image data, the browser 62 may download the information about the panel image data stored in the record with the most recent change count. Alternatively, the browser 62 may download the information about the panel image data stored in the record with the most recent timestamp.

[0147] Furthermore, the browser 62 displays the updated panel image 531 to the user interface 53 based on the downloaded panel image data (S343, C25 in Figure 13). When the content of the screen displayed on the touch panel 131 of the MFP1 is changed, the panel image 531 displayed on the user interface 53 of the PC5 is also changed accordingly, thus synchronizing the screens of the MFP1 and the PC5. Note that only the panel image 531 is subject to change, and the browser 62 does not change the displayed key image 532 or the exit button 533. In addition, the browser 62 stores the number of changes included in the image information (S344).

[0148] In other words, when an event occurs that changes the content of the screen displayed on the touch panel 131, the MFP1 uploads panel image data and updates the image information. Then, with the image information updated, the browser 62 downloads the panel image data and updates the panel image 531. This allows the image displayed on the MFP1's touch panel 131 and the image displayed on the PC5's user interface 53 to be synchronized.

[0149] If browser 62 fails to display panel image 531 based on the information downloaded in C21-C23, it will maintain the display of the panel image 531 that was originally displayed. In this case, if necessary to maintain the display, browser 62 can re-download the information downloaded in the previous steps B33-B35 or C21-C23.

[0150] After S344, or if it is determined that the panel image 531 will not be updated (S341: NO), the browser 62 proceeds to S311 and repeats the process of determining whether or not it has accepted the operation from user IF53 (S311) and whether or not to update the panel image 531 (S341) (loop in Figure 13).

[0151] On the other hand, if the browser 62 receives an operation from user IF53 (S311:YES), and further determines that the instruction from the received operation is an instruction to terminate the cloud remote operation (S321:YES), that is, if it receives an operation to the termination button 533 shown in Figure 12 (C31 in Figure 13), the browser 62 uploads the termination information to the structured storage 202 (S331, C32 in Figure 13).

[0152] Specifically, browser 62 uploads termination information, for example, as shown in Figure 11, by storing information indicating that the termination instruction for cloud remote operation of MFP1 has been received in parameter 2025 of record R51, which is a record in which the first key 2021 stores information indicating that it is operation information for cloud remote operation and the second key 2022 stores information indicating a termination instruction. The termination information may also be information indicating an operation to the termination button 533. S331 is an example of the operation upload process. After that, PC5 terminates the browser image change process and terminates the cloud remote operation of MFP1 (C33 in Figure 13, break).

[0153] After the cloud remote control is complete, the MFP1 will not upload panel image data. PC5 may close browser 62, or it may stop executing the process according to browser P262 and display another web page, or it may display a blank screen. The downloaded executable file 26 may be deleted when browser 62 is closed, or it may be deleted voluntarily upon completion of execution.

[0154] Next, the procedure for the device image modification process executed by MFP1 when cloud remote control is initiated will be explained with reference to the flowchart in Figure 15, and, if necessary, to the sequence diagram in Figure 13. As mentioned above, the device image modification process is executed by the CPU 11 of MFP1 after the executable file 26 and image data are uploaded and response data indicating "success" is uploaded to the structured storage 202 (B11~B17 in Figure 8).

[0155] During the device image change process, the CPU 11 determines whether it is time for a periodic check (S401). While cloud remote operation is in progress, the MFP 1 periodically accesses the structured storage 202, similar to the browser 62 on the PC 5 (C04 in Figure 13). If it is not time for a periodic check (S401: NO), the CPU 11 repeats the determination in S401. Note that the timing of the check by the MFP 1 and the timing of the check by the browser 62 do not have to be the same.

[0156] If it determines that it is time for a periodic check (S401:YES), CPU11 checks whether termination information containing its own device ID has been uploaded to structured storage 202 (S402). If it determines that termination information has not been uploaded (S402:NO), CPU11 checks whether operation information containing its own device ID has been uploaded to structured storage 202 (S403). Operation information is uploaded to structured storage 202 via browser image change processing (S322 in Figure 14, C12 in Figure 13) according to the procedure described above when an operation is received by the user of PC5.

[0157] If the CPU 11 determines that the operation information has been uploaded to the structured storage 202 (S403: YES), it performs operation information processing (S411). The procedure for operation information processing will be explained with reference to the flowchart in Figure 16, and, if necessary, to the sequence diagram in Figure 13.

[0158] In operation information processing, the CPU 11 obtains up to N operation information items uploaded to the structured storage 202 by downloading them (S501, C31 in Figure 13). N is, for example, an integer between 2 and 20, and is pre-set in the MFP1. If there are fewer than N operation information items uploaded, the CPU 11 obtains all uploaded operation information. As mentioned above, when the browser 62 receives multiple operations, it stores each operation as one piece of operation information. For example, if it receives a click on an icon, two consecutive pieces of operation information are stored. Since the MFP1 obtains a certain number of operation information items at once, the number of accesses to the cloud storage service 200 does not increase.

[0159] CPU 11 processes the downloaded operation information in chronological order, based on the operation number or timestamp. CPU 11 first analyzes the first operation information (S511) and determines whether the operation is an instruction to start maintenance mode (S512). If it determines that it is not an instruction to start maintenance mode (S512: NO), CPU 11 executes the processing according to the operation information (S514, C32 in Figure 13). By processing in chronological order, MFP1 can execute the processing according to the procedure operated on PC 5.

[0160] As mentioned above, the MFP1 can display a screen containing one or more icon images on the touch panel 131, and when it receives an operation within the area of ​​the displayed icon image, it can execute the process indicated by that icon image. For example, if the operated icon image is an icon indicating an instruction to execute image processing, the MFP1 executes the instructed image processing. Then, the PC5 user IF53 displays a panel image 531 that virtually reproduces the touch panel 131. If the downloaded operation information is information indicating an operation on the panel image 531, the CPU 11 executes the process corresponding to the operation, just as if it had received an operation on the touch panel 131.

[0161] For example, if the operation information indicates an operation on the copy icon included in the panel image 531 which is a virtual representation of the MFP1's standby screen, the CPU 11 obtains information for displaying the copy processing settings screen. If the operation information indicates an operation related to changing settings on the panel image 531 which is a virtual representation of the copy processing settings screen, the CPU 11 changes the copy processing settings. If the operation information indicates an operation on an icon indicating an instruction to execute the image formation process, the CPU 11 starts executing the image formation process as the process corresponding to that operation. For example, if the operation information indicates an operation on an icon for instructing the execution of the copy process on the panel image 531 which is a virtual representation of the copy processing settings screen, the CPU 11 starts executing the copy process.

[0162] Furthermore, if the operation information indicates a key operation, the CPU 11 executes the corresponding processing, similar to when it receives an operation on the hard key 132. Then, the CPU 11 changes the panel image according to the execution of the processing and displays it on the touch panel 131 (S515).

[0163] For example, if information is obtained to display the copy process settings screen, the CPU 11 displays a settings screen that includes an icon to instruct the user to change the settings, an input field for the setting value to be changed, and an icon to instruct the user to execute the copy process. After the copy process settings are changed, the CPU 11 displays the updated copy process settings screen to show the changed settings. After changing the copy process settings, the CPU 11 may display a waiting screen. After starting the execution of the image formation process, the CPU 11 displays the execution screen for the image formation process, for example, the execution screen for the copy process.

[0164] On the other hand, if the CPU 11 determines that it is an instruction to start maintenance mode (S512: YES), it starts maintenance mode (S516). Maintenance mode is a mode used by maintenance personnel for maintenance of MFP1, for example, and is a mode in which instructions such as printing reports and checking the operating status can be received. MFP1 can receive an instruction to start maintenance mode, for example, by a specific combination of operations on a specific key.

[0165] When the MFP1 receives a command to start maintenance mode, it displays a dedicated maintenance mode screen on the touch panel 131, which is a screen that accepts commands that can be executed in maintenance mode (S517). The CPU 11 then calculates the checksum of the displayed maintenance mode screen and stores it in the memory 12 (S518).

[0166] Furthermore, when MFP1 is in maintenance mode, it displays a maintenance mode screen and can only accept maintenance mode execution commands, for example, those that specify the type of maintenance mode by number. When MFP1 is in maintenance mode, it does not accept print commands from external devices, for example. The screen data of the maintenance mode screen is, for example, bitmap data and does not have a screen ID set. Therefore, CPU11 uses a checksum to determine whether or not the screen has been changed.

[0167] Note that a screen ID may also be set for the maintenance mode screen. In that case, MFP1 may store the screen ID in S518 instead of calculating a checksum. Also, there may be screens other than the maintenance mode screen that do not have a screen ID set. In that case, even when displaying a screen without a screen ID set, MFP1 should calculate and store a checksum.

[0168] After S515 or S518, the CPU 11 deletes the processed operation information from the structured storage 202 (S521). Then, the CPU 11 determines whether the processing of the N operation information obtained in S511 is complete (S522). If it determines that it is not complete (S522: NO), the CPU 11 proceeds to S511 and processes the next item. If it determines that the processing of all N operation information items is complete (S522: YES), the CPU 11 terminates the operation information processing and returns to the device image change processing in Figure 15. Alternatively, instead of deleting each item one by one in S521, the CPU 11 may delete all N operation information items at once after the processing of all N items is complete (YES in S522).

[0169] Returning to the explanation of the device image change process in Figure 15, after the operation information processing in S411, the CPU 11 executes the image storage process (S431). The image storage process is the process shown in Figure 10, which uploads the panel image data representing the current panel image to the cloud storage service 200.

[0170] During the image storage process, as shown in C33-C35 of Figure 13, panel image data is uploaded to structured storage 202 or unstructured storage 201, and image information is uploaded to structured storage 202. C33-C35 in Figure 13 is the same process as B13-B15 in Figure 8. The information uploaded by C33-C35 is downloaded by browser 62 in C21-C23 and processed by browser 62, as described above.

[0171] For example, MFP1 may change the screen displayed on the touch panel 131 when it performs processing based on acquired operation information. In the operation information processing in S411, for example, if the panel image displayed on the touch panel 131 is updated (S515) or if maintenance mode is started and the maintenance mode screen is displayed (S517), MFP1 uploads the panel image data and image information by performing image storage processing.

[0172] As a result, after uploading operation information indicating an operation on the panel image 531, the browser 62 can download new panel image data and update the panel image 531 in response to the MFP1 updating the image information after executing processing according to the operation information. This allows the screen displayed on the MFP1's touch panel 131 to change, and the image showing the changed screen to be displayed on the PC5's user interface 53. Therefore, the screens of the MFP1 and PC5 can be synchronized. After the image storage process in S431, the CPU 11 returns to S401 to determine whether it is time for the next confirmation.

[0173] On the other hand, if it is determined that no operation information has been stored (S403: NO), the CPU 11 determines whether or not it is in maintenance mode (S421). If it is determined that it is not in maintenance mode (S421: NO), the CPU 11 executes the image storage process (S431). The MFP1 may change the screen displayed on the touch panel 131 even if it has not executed processing based on acquired operation information, such as when it has received an operation from the user IF 13. If cloud remote control is being performed, and the MFP1 is not in maintenance mode, it executes the image storage process and uploads panel image data and image information.

[0174] In other words, MFP1 repeatedly uploads panel image data showing the screen currently displayed on the touch panel 131. If the displayed screen changes, the updated panel image data and image information are uploaded in the image storage process immediately following the change. As a result, even if the screen displayed on the touch panel 131 changes, not triggered by an operation on the panel image 531, but for example triggered by an operation on the user interface 13 of MFP1, the browser 62 can download new panel image data in response to the update of the image information by MFP1, and update and display the panel image 531. Therefore, the screen of MFP1 and the screen of PC5 can be synchronized. After the image storage process in S431, the CPU 11 returns to S401 to determine whether it is time for the next confirmation.

[0175] On the other hand, if it is determined that the system is in maintenance mode (S421: YES), the CPU 11 obtains the checksum of the currently displayed panel image (S422). When the system receives a command to start maintenance mode and starts maintenance mode (S516 in Figure 16), the MFP1 is in maintenance mode. The CPU 11 compares the obtained checksum with the stored checksum to determine whether the checksum has been changed (S423). For example, the CPU 11 compares it with the checksum stored in S518 of the operation information processing (Figure 16) when maintenance mode was started to determine whether it has been changed. If it is determined that the checksum has been changed (S423: YES), the CPU 11 stores the checksum of the current panel image (S424) and executes the image storage process (S431). In the image storage process, the modified panel image data and image information are uploaded. In the next S423, the CPU 11 compares the checksum stored in the current S424 with the checksum of the currently displayed panel image. After the image storage process in S431, CPU 11 returns to S401 to determine whether or not it is time for the next check.

[0176] Furthermore, if the system is in maintenance mode and determines that the checksum has not been changed (S423: NO), the CPU 11 returns to S401 to determine whether it is time for the next check. If the system is in maintenance mode and the checksum has not been changed, the CPU 11 does not perform the image storage process.

[0177] Thus, during maintenance mode, MFP1 determines whether the displayed screen has been changed, and if it has not been changed, it does not upload the panel image data, thereby reducing communication volume. Furthermore, MFP1 may also determine whether the displayed screen has been changed and not upload the panel image data even when not in maintenance mode. For example, MFP1 may determine whether the screen has been changed based on the screen ID of the displayed screen. Not uploading the panel image data when the screen has not been changed reduces the communication load.

[0178] Furthermore, if it is determined that termination information has been uploaded (S402: YES), the CPU 11 executes the operation termination process (S441). In addition, the CPU 11 changes the cloud status information 81 (see Figure 3) to information indicating "unused" (S442). S402 is an example of the termination conditions for the first remote operation, and if termination information is stored (YES in S402), the CPU 11 determines that the termination conditions for the first remote operation have been met. Note that if termination information has been uploaded, the CPU 11 will not download the operation information, even if operation information that has not been downloaded has been uploaded. The procedure for the operation termination process will be explained with reference to the sequence diagram in Figure 17.

[0179] MFP1 downloads termination information from structured storage 202 (D01). Termination information is information stored in structured storage 202 by browser 62 at C32 in Figure 13, as described above. Termination information includes information indicating that a termination command was received at C31 in Figure 13, for example, as shown in record R51 in Figure 11.

[0180] Upon downloading the termination information, MFP1 instructs the structured storage 202 and unstructured storage 201 to delete various information uploaded to them. Specifically, MFP1 instructs structured storage 202 to delete the termination information for this cloud remote operation (D11). MFP1 instructs structured storage 202 to delete the operation information for this cloud remote operation (D12). Upon receiving the termination command, MFP1 deletes all remaining operation information, even if any is still pending.

[0181] Furthermore, MFP1 instructs structured storage 202 to delete the uploaded panel image data and image information (D13, D14). If there is any panel image data uploaded to unstructured storage 201, MFP1 instructs it to delete that panel image data (D21). As PC5 receives the termination command, various data uploaded to structured storage 202 and unstructured storage 201 are deleted, thereby reducing the memory load on structured storage 202 and unstructured storage 201.

[0182] Furthermore, MFP1 instructs the deletion of the executable file 26 that was uploaded to the unstructured storage 201 at the start of cloud remote operation (D22). In addition, MFP1 instructs the deletion of the folder created in A32 of Figure 4 (D23). Note that the order in which each data is deleted is not limited to the order described above; any order is acceptable.

[0183] Once all data has been deleted, MFP1 displays a message on the touch panel 131 indicating that the cloud remote operation is complete (D31), and then terminates the cloud remote operation. MFP1 then displays a standby screen on the touch panel 131, for example, and becomes available to accept user input.

[0184] Next, the local operation procedure, which is the process when a request to start local remote operation is received, will be explained with reference to the sequence diagram in Figure 18. A user of PC7 that can communicate locally with MFP1 launches browser 92 on PC7 and issues a local remote operation start command by specifying the address information that points to the EWS80 of MFP1 (E01). Based on the specified address information, browser 92 accesses MFP1 via local communication and sends a request to start local remote operation (E02).

[0185] When MFP1 receives a request to start local remote operation, it executes the start determination process shown in Figure 6 (E03) to decide whether or not to start local remote operation. For example, if MFP1 is accepting cloud remote operation from PC5, it will not start local remote operation even if it receives a command to start local remote operation. If the start determination process decides not to start local remote operation, MFP1 responds with information indicating that it cannot start and terminates the procedure shown in Figure 18.

[0186] If it decides to start local remote control, MFP1 sends an HTML file indicating the web page for local remote control to PC7 in response to the local remote control start request received in E02 (E10). The HTML file includes, for example, a panel display element 901, a hard key display element 902, image data for the exit button 903, image data for the hard keys 904, a link to the CSS file 905, and a link to the browser program 84 (see Figure 3) 906, as shown in Figure 19. The HTML file 90 is stored, for example, in memory 12 (see Figure 1). MFP1 may also dynamically generate the HTML file 90.

[0187] The panel display element 901, the hard key display element 902, and the hard key image data 904 contain the same information as the panel display element 266, the hard key display element 267, and the hard key image data 264 (see Figure 9) included in the executable file 26 used for cloud remote control. Note that this data may be exactly the same as the information included in the executable file 26, or it may be partially different.

[0188] As mentioned above, if MFP1 decides to start a local remote operation in response to a request to start one, it sets the local status information 82 to "In Use". During a local remote operation, MFP1 can pass various information to the PC7's browser 92 via local communication that does not go through the cloud storage service 200. After sending the HTML file in E10, MFP1 remains in a provision state, indicating that it is providing the local remote operation, until it meets the termination conditions for ending the local remote operation. The provision state of the local remote operation is an example of the second remote operation provision state.

[0189] Browser 92 obtains the browser program 84 and the CSS file from MFP1 based on the links contained in the acquired HTML file 90 (E11~E12). Browser program 84 is a program that can run on browser 92 and is similar to, but slightly different from, the browser program P262 for cloud remote control (see Figure 9). Details of the differences will be described later. Browser program 84 is an example of a second remote control program. E11 is an example of the program transmission process.

[0190] The CSS file contains the same information as CSS261 (see Figure 9) included in executable file 26 (see Figure 9) used for cloud remote control. The CSS may be exactly the same as CSS261 included in executable file 26, or it may differ in some respects. The CSS for local remote control may have slightly different display content from the CSS for cloud remote control, for example, in terms of display colors, frame shapes, etc.

[0191] Note that HTML file 90 may contain a link to hard key image data 904 instead of the hard key image data 904 itself. In this case, browser 92 may retrieve hard key image data 904 from MFP1 based on the link, similar to browser program 84 and CSS files (E11-E12).

[0192] Furthermore, instead of the HTML file 90 shown in Figure 19, MFP1 may pass to the browser 92 an HTML file with embedded CSS and a browser program 84, such as the executable file 26 (see Figure 9) used for cloud remote control. Additionally, the image data for the hardware keys may be embedded in the browser program 84. Similarly, the image data for the exit button may be included in the HTML file as an exit button display element, and the image data for the exit button may be embedded in the browser program 84.

[0193] The browser 92 then acquires panel image data from the MFP1 (E14) and displays the panel image on the PC7's user interface 93 (see Figure 3) based on the acquired panel image data (E15). The panel image data acquired from the MFP1 in E14 is data used to virtually reproduce the screen displayed on the MFP1's touch panel 131, and is, for example, bitmap data or PNG data which is compressed bitmap data. E14 is an example of image transmission processing, and the panel image data transmitted from the MFP1 in E14 is an example of second image data. E15 is an example of local screen display processing, and the panel image displayed on the PC7's user interface 93 in E15 is an example of a second remote operation screen.

[0194] As a result, the user interface 93 of PC7 displays a panel image that virtually replicates the user interface 13 of MFP1, similar to the example shown in Figure 12. The panel image displayed on the user interface 93 of PC7 also includes an exit button, similar to the panel image 531 displayed on the user interface 53 of PC5. In addition, MFP1 may transmit panel image data that does not include videos or animations via E14, as in the case of cloud remote control.

[0195] A user using PC7 can perform operations on user IF93, on which the panel image is displayed (E21). When browser 92 receives an operation on the displayed panel image, if the user's operation is not an operation on the exit button, it sends operation data indicating the content of the operation to MFP1 via local communication (E22). The operation data sent in E22 is an example of second operation data, and E22 is an example of operation transmission processing.

[0196] MFP1 executes processing corresponding to the operation content based on the received operation data (E23). If the screen displayed on the touch panel 131 is updated by the processing in E23, MFP1 sends panel image data showing the updated screen to PC7 (E24). E24 is an example of image transmission processing. The browser 92 on PC7 updates the panel image based on the received panel image data (E25). The browser 92 repeats E21 to E25 each time it receives user input (loop).

[0197] When Browser 92 receives an input to the exit button (E31), it stops repeating E21-E25 (break) and sends exit data indicating the termination instruction to MFP1 (E32). Then, PC7 terminates the local remote operation (E33). PC7 may also terminate Browser 92.

[0198] MFP1 terminates the provision of local remote control upon receiving termination data (E35). MFP1 may, for example, display an indication that local remote control has been terminated. When MFP1 receives termination data for local remote control from a PC that is providing local remote control, it determines that the termination conditions for the second remote control have been met. MFP1 terminates local remote control and changes the local status information 82 to information indicating "unused" (E36).

[0199] As explained in detail above, with this form of remote control system 100, the MFP1 can provide both cloud remote control via the cloud storage service 200 by the PC5 and local remote control using local communication by the PC7. When the MFP1 receives a command to start cloud remote control (A22 in Figure 4) and decides to start cloud remote control (S119 in Figure 6), it uploads the executable file 26 (see Figure 1), etc. (B11 in Figure 8), and enters the cloud remote control provision state. In the cloud remote control provision state, the MFP1 will not start providing local remote control until the termination conditions for cloud remote control are met. This avoids confusion in operations when cloud remote control is provided.

[0200] Furthermore, according to this configuration of the remote control system 100, when MFP1 receives a local remote control start instruction (E02 in Figure 18), and the start determination process determines that local remote control should be started (S119 in Figure 6), it sends an HTML file (E10 in Figure 18) and enters the local remote control provision state. In the local remote control provision state, MFP1 will not start providing cloud remote control until the local remote control termination conditions are met. This prevents confusion in operations even in the local remote control provision state.

[0201] Furthermore, communication between different networks presents security challenges and is often subject to communication restrictions. Therefore, PC5 may not be able to directly transfer data necessary for remote operation to MFP1, which is not on the same network. As in the present invention, by using a cloud storage service 200 on the internet to transfer data between PC5 and MFP1, communication restrictions are less likely to occur, increasing the possibility of enabling remote operation of MFP1, which is not on the same network.

[0202] This embodiment is merely illustrative and does not limit the present invention in any way. Therefore, the technology disclosed herein can naturally be improved and modified in various ways without departing from its essence. For example, the device to be remotely controlled locally or via cloud remote control may be any device that has a communication interface and is connectable to the Internet, and has image processing capabilities. It is not limited to MFPs, but may also be a printer with only printing functions, a copier, a scanner, a fax machine, or a machine tool capable of image-based machining. Furthermore, the device that performs local remote control or cloud remote control may be a mobile terminal such as a smartphone or tablet computer, not limited to a PC. In addition, the remote control system 100 may include one or more devices that perform remote control and devices that are the target of remote control, and the number and arrangement of these devices are not limited to the illustrated examples.

[0203] The configuration of the user interface 13 of the MFP1 is not limited to those exemplified in the embodiment. For example, the MFP1 may have a screen with only display functionality instead of the touch panel 131, and may accept operation input by cursor operation or the like. Also, the number of hard keys 132 is not limited to three, and may be omitted.

[0204] Furthermore, for example, MFP1 may be capable of accepting both cloud remote control from the same PC and local remote control from the same PC. For example, a PC that can communicate with MFP1 over a LAN may receive a command to start cloud remote control via the start application 63, and then be able to perform local remote control via EWS80. In this case, both remote controls may be performed by displaying multiple web pages in a single browser on the PC, or separate browsers may be launched in different processes, and remote controls may be performed using each browser. In this case, the PC is an example of both the first and second information processing devices.

[0205] Furthermore, for example, in the embodiment, when a request to start remote operation is received, the license confirmation screen 71 is displayed to obtain the user's approval. However, this procedure may be omitted, and subsequent processing may be performed in the same manner as when approval is obtained. Also, for example, if a command to start cloud remote operation or local remote operation is received and rejected by the user, and then another command to start remote operation is received, the remote operation is not provided without displaying the license confirmation screen 71. However, the MFP1 may display the license confirmation screen 71 again. In that case as well, if the user rejects it, the MFP1 will not provide the remote operation that was received later. On the other hand, if the user approves, the MFP1 may start providing the remote operation that was received later.

[0206] Furthermore, for example, if MFP1 sets the selection status information 83 to "rejected," it is stated that it will automatically change it to "approved" after a certain period of time, but this is not required. MFP1 may, for example, continue to store the "rejected" information until the power is turned off, or it may be possible to delete it through user operation. Also, the deletion procedure may differ depending on whether cloud remote control is rejected or local remote control is rejected.

[0207] Furthermore, in this embodiment, when providing cloud remote control, PC5 uploads a record containing instruction data to structured storage 202, and MFP1 obtains the instruction data by downloading it. However, this is not limited to this. For example, PC5 may directly send the instruction data to MFP1. Similarly, MFP1 may directly send the response data to PC5.

[0208] Furthermore, in this embodiment, when providing cloud remote control, the MFP1 uploads response data including access information 243 for its own folder to structured storage 202 and passes its URL to PC5, but it is not limited to this. For example, the startup application 63 of PC5 may have in advance a URL indicating the folder used by the MFP that is the target of cloud remote control.

[0209] Furthermore, in the embodiment, when cloud remote control is provided, the image information includes information indicating the storage location of the panel image data, specifically information indicating structured storage 202 or unstructured storage 201, but this is not limited to this. For example, the information indicating the storage location may be included only when the panel image data is uploaded to structured storage 202, or it may be included only when the panel image data is uploaded to unstructured storage 201.

[0210] Furthermore, while the embodiment described receiving an operation on the termination button 533 (see Figure 12) displayed on a PC or the like as a trigger for terminating local remote operation or cloud remote operation, it is not limited to this. For example, if PC5 or PC7 is performing remote operation and does not accept any operation to the browser 62 for a certain period of time, it may upload or send termination information in the same way as when a termination instruction is received, and terminate the remote operation. Alternatively, for example, the remote operation may be terminated when MFP1 receives a specific operation, such as a long press of the cancel button. Also, MFP1 may determine that a termination instruction has been received if no new operation information is uploaded for a certain period of time. In that case, MFP1 may upload information indicating a termination instruction to structured storage 202.

[0211] Furthermore, while the embodiment described an example of obtaining one token each for accessing the unstructured storage 201 and the structured storage 202, it is also possible to obtain multiple tokens for a single area. In that case, for example, multiple tokens may be stored as access information 24 of MFP1, one of which may be used by the MFP1 itself, while the other is passed to PC5. Also, for example, different tokens may be used for uploading and downloading. Furthermore, different tokens may be available for each MFP.

[0212] Furthermore, in any flowchart or sequence diagram disclosed in the embodiments, the execution order of any multiple processes in any multiple steps can be arbitrarily changed or executed in parallel, as long as no inconsistencies arise in the processing content.

[0213] Furthermore, the processes disclosed in the embodiments may be executed by a single CPU, multiple CPUs, hardware such as an ASIC, or a combination thereof. Also, the processes disclosed in the embodiments can be implemented in various forms, such as a recording medium or method that stores a program for executing the process. [Explanation of Symbols]

[0214] 1 MFP 13 User Interface 131 Touch Panel 132 hard keys 5, 7 PC 53, 93 User IF 100 Remote Control Systems 200 Cloud Storage Services

Claims

1. A system comprising an image forming apparatus, a first information processing apparatus, and a second information processing apparatus, wherein the first information processing apparatus, the second information processing apparatus, and the image forming apparatus are connected to the Internet. The first information processing device and the second information processing device are capable of installing a browser. When the image forming apparatus receives a command to start a first remote operation from the first information processing device via the cloud storage connected to the Internet, it executes a program upload process to upload a first remote operation program, which can be run in the browser, to the cloud storage. Once the first remote operation program is uploaded to the cloud storage, the browser installed on the first information processing device can cause the first information processing device to download the first remote operation program from the cloud storage. The image forming apparatus is further capable of performing an image upload process to upload first image data showing the first remote operation screen to the cloud storage after receiving the start instruction for the first remote operation. The first remote operation program can cause the first information processing device to download the first image data from the cloud storage and to perform a screen display process that displays the first remote operation screen shown in the first image data on the user interface of the first information processing device using the browser. The first remote operation program, upon receiving an operation on the first remote operation screen via the user interface of the first information processing device, further causes the first information processing device to execute an operation upload process to upload first operation data indicating the operation content to the cloud storage. When the first operation data is uploaded to the cloud storage, the image forming apparatus downloads the first operation data from the cloud storage, executes a process corresponding to the operation shown in the first operation data, and if the operation shown in the downloaded first operation data is an operation related to image forming, executes an image forming process as the process corresponding to that operation. When the image forming apparatus receives a command to start the second remote operation from the second information processing device via local communication without using the cloud storage, it executes a program transmission process to send a second remote operation program, which can be run in the browser, to the second information processing device without using the cloud storage, and the browser installed in the second information processing device acquires the second remote operation program transmitted from the image forming apparatus. The image forming apparatus is further capable of performing an image transmission process to send a second image data showing the second remote operation screen to the second information processing device without going through the cloud storage, after receiving the start instruction for the second remote operation. The second remote operation program, when the second information processing device receives the second image data, causes the second information processing device to execute a local screen display process that displays the second remote operation screen shown in the received second image data on the user interface of the second information processing device using the browser. The second remote operation program, upon receiving an operation on the second remote operation screen via the user interface of the second information processing device, causes the second information processing device to execute an operation transmission process that transmits second operation data indicating the operation content to the image forming apparatus without going through the cloud storage. When the image forming apparatus receives the second operation data from the second information processing device, it executes a process corresponding to the operation indicated in the received second operation data, and if the operation indicated in the received second operation data is an operation related to image forming, it executes an image forming process as the process corresponding to that operation. After the image forming apparatus receives the start instruction for the first remote operation, it starts providing the first remote operation by executing the program upload process, and remains in the first remote operation provision state until the termination conditions for the first remote operation are met. After receiving the start instruction for the second remote operation, the image forming apparatus starts providing the second remote operation by executing the program transmission process, and remains in the second remote operation provision state until the termination conditions for the second remote operation are met. When the image forming apparatus is not in the state of providing the first remote operation, if it receives the start instruction for the second remote operation from the second information processing device, it will be able to start providing the second remote operation. When the image forming apparatus is in the first remote operation provision state and receives the start instruction for the second remote operation from the second information processing device, it will not start providing the second remote operation. A system configured in such a way.

2. The system described in claim 1, When the image forming apparatus is not in the second remote operation provision state, and receives the start instruction for the first remote operation from the first information processing device, it enables the provision of the first remote operation. When the image forming apparatus is in the second remote operation provision state and receives the start instruction for the first remote operation from the first information processing device, it does not start providing the first remote operation. A system configured in such a way.

3. The system described in claim 1, The image forming apparatus includes a display, which displays the main unit operation screen, and in the image upload process, it uploads image data to the cloud storage as the first image data representing the first remote operation screen, which virtually reproduces the main unit operation screen displayed on the display. The first remote operation program can cause the first information processing device to execute the screen display process when the first image data is uploaded to the cloud storage, and in the screen display process, the first remote operation screen, which virtually reproduces the main unit operation screen based on the first image data downloaded from the cloud storage, is displayed on the user interface of the first information processing device using the browser. When the image forming apparatus is not in the first remote operation provision state, and receives the start instruction for the second remote operation from the second information processing device and starts providing the second remote operation, the image transmission process transmits image data for virtually reproducing the main unit operation screen displayed on the display to the second information processing device as the second image data showing the second remote operation screen. The second remote operation program, when the second information processing device receives the second image data, causes the second information processing device to execute the local screen display process, and in the local screen display process, displays the second remote operation screen, which virtually reproduces the main unit operation screen based on the received second image data, on the user interface of the second information processing device using the browser. A system configured in such a way.

4. The system described in claim 3, When the image forming apparatus is in the first remote operation provision state, if an event occurs that changes the content of the main unit operation screen displayed on the display, it executes the image upload process, and in the image upload process, it uploads the first image data for virtually reproducing the main unit operation screen after the change to the cloud storage. When the first remote operation program receives the first image data for virtually reproducing the modified main unit operation screen from the cloud storage, it causes the first information processing device to execute the screen display process, in which the screen display process downloads the first image data for virtually reproducing the modified main unit operation screen from the cloud storage and displays the first remote operation screen that virtually reproduces the modified main unit operation screen on the user interface of the first information processing device using the browser. When the image forming apparatus is not in the first remote operation provision state, if it receives the start instruction for the second remote operation from the second information processing device and starts providing the second remote operation, and then an event occurs that changes the content of the main unit operation screen displayed on the display, it executes the image transmission process, and in the image transmission process, it transmits image data for virtually reproducing the main unit operation screen after the change as the second image data showing the second remote operation screen to the second information processing device. The second remote operation program, upon receiving the second image data of the second information processing device, causes the second information processing device to execute the local screen display process, and in the local screen display process, displays the second remote operation screen, which virtually reproduces the modified main unit operation screen based on the received second image data, on the user interface of the second information processing device using the browser. A system configured in such a way.

5. The system described in claim 3, When the image forming apparatus is in the first remote operation provision state, it can change the content of the main unit operation screen in accordance with the execution of a process corresponding to an operation shown in the first operation data uploaded to the cloud storage, and when the content of the main unit operation screen is changed in accordance with the execution of the process corresponding to an operation shown in the first operation data, the first image data is uploaded to the cloud storage by executing the image upload process to virtually reproduce the main unit operation screen after the execution of the process corresponding to the operation shown in the first operation data. When the first remote operation program receives the first image data for virtually reproducing the modified main unit operation screen associated with the processing corresponding to the operation shown in the first operation data, it causes the first information processing device to execute the screen display process, and in the screen display process, it causes the first image data for virtually reproducing the modified main unit operation screen associated with the processing corresponding to the operation shown in the first operation data to be downloaded from the cloud storage, and displays the first remote operation screen that virtually reproduces the modified main unit operation screen associated with the processing corresponding to the operation shown in the first operation data on the user interface of the first information processing device using the browser. When the image forming apparatus is not in the first remote operation provision state, it receives the start instruction for the second remote operation from the second information processing device and starts providing the second remote operation. After that, it can change the content of the main unit operation screen in accordance with the execution of the process corresponding to the operation indicated in the second operation data transmitted from the second information processing device. When the content of the main unit operation screen is changed in accordance with the execution of the process corresponding to the operation indicated in the second operation data transmitted from the second information processing device, it executes the image transmission process to transmit the second image data to the second information processing device, thereby virtually reproducing the main unit operation screen after the execution of the process corresponding to the operation indicated in the second operation data. The second remote operation program, upon receiving the second image data of the second information processing device, causes the second information processing device to execute the local screen display process, and in the local screen display process, displays the second remote operation screen, which virtually reproduces the modified main unit operation screen based on the received second image data, on the user interface of the second information processing device using the browser. A system configured in such a way.

6. The system described in claim 1, The image forming apparatus includes a touch panel, displays a main panel screen including multiple controls on the touch panel, and when it receives an operation on one of the multiple controls, it executes processing based on the operation received by the control. The first remote operation program can cause the first information processing device to execute the screen display process, and in the screen display process, the first remote operation screen, which virtually reproduces the main unit panel screen based on the first image data downloaded from the cloud storage, is displayed on the user interface of the first information processing device using the browser. The first remote operation program, upon receiving an operation on one of the plurality of operators displayed on the first remote operation screen via the user interface of the first information processing device, causes the first information processing device to execute the operation upload process, in which the operation upload process uploads information identifying the operator that received the operation as first operation data to the cloud storage. When the first operation data is uploaded to the cloud storage, the image forming apparatus downloads the first operation data from the cloud storage and performs processing based on the operator identified by the first operation data. The second remote operation program can cause the second information processing device to execute the local screen display process, and in the local screen display process, the second remote operation screen, which virtually reproduces the main unit panel screen based on the second image data received by the second information processing device, is displayed on the user interface of the second information processing device using the browser. The second remote operation program, upon receiving an operation on one of the multiple operators displayed on the second remote operation screen via the user interface of the second information processing device, causes the second information processing device to execute the operation transmission process, in which the operation transmission process transmits information identifying the operator that received the operation as second operation data to the image forming apparatus. When the image forming apparatus receives the second operation data transmitted from the second information processing device, it performs processing based on the operator identified by the received second operation data. A system configured in such a way.

7. The system described in claim 6, The image forming apparatus includes an operation panel including a touch panel, the operation panel also includes a plurality of hardware keys, and when it receives an operation on one of the plurality of hardware keys, it executes processing based on the hardware key that received the operation. The first remote operation program can cause the first information processing device to execute the screen display process, and in the screen display process, the first remote operation screen, which virtually reproduces the control panel based on the first image data downloaded from the cloud storage, is displayed on the user interface of the first information processing device using the browser. The first remote operation program, upon receiving an operation on one of the multiple hardware keys displayed on the first remote operation screen via the user interface of the first information processing device, causes the first information processing device to execute the operation upload process, in which the operation upload process uploads information identifying the hardware key for which the operation was received as first operation data to the cloud storage. When the first operation data is uploaded to the cloud storage, the image forming apparatus downloads the first operation data from the cloud storage and performs processing based on the hardware key identified by the first operation data. The second remote operation program can cause the second information processing device to execute the local screen display process, and in the local screen display process, the second remote operation screen, which virtually reproduces the control panel based on the second image data received by the second information processing device, is displayed on the user interface of the second information processing device using the browser. The second remote operation program, upon receiving an operation on one of the multiple hardware keys displayed on the second remote operation screen via the user interface of the second information processing device, causes the second information processing device to execute the operation transmission process, in which the operation transmission process transmits information identifying the hardware key that received the operation as second operation data to the image forming apparatus. When the image forming apparatus receives the second operation data transmitted from the second information processing device, it performs processing based on the hardware key identified by the received second operation data. A system configured in such a way.

8. The system described in claim 1, The image forming apparatus, while in the state of providing the first remote operation, does not start providing the first remote operation even if it receives the start instruction for the first remote operation from an information processing device other than the first information processing device. A system configured in such a way.

9. The system according to claim 8, The image forming apparatus, while in the state of providing the second remote operation, will not start providing the second remote operation even if it receives the start instruction for the second remote operation from an information processing device other than the second information processing device. A system configured in such a way.

10. The system described in claim 3, The image forming apparatus, when it is neither in the first remote operation provision state nor the second remote operation provision state, receives the start instruction for the first remote operation from the first information processing device via the cloud storage, is configured to receive a first operation via the user interface of the image forming apparatus indicating whether or not to approve the provision of the first remote operation, and if it receives the first operation approving the provision of the first remote operation, it starts providing the first remote operation, and if it receives the first operation not approving the provision of the first remote operation, it does not start providing the first remote operation. When the image forming apparatus is in the first remote operation provision state or the second remote operation provision state, if it receives the start instruction for the first remote operation, it will not accept the first operation and will not start providing the first remote operation. The image forming apparatus, when it is neither in the first remote operation provision state nor the second remote operation provision state, receives the start instruction for the second remote operation from the second information processing device via local communication, is configured to receive a second operation via the user interface of the image forming apparatus indicating whether or not to approve the provision of the second remote operation, and if it receives the second operation approving the provision of the second remote operation, it starts providing the second remote operation, and if it receives the second operation not approving the provision of the second remote operation, it does not start providing the second remote operation. When the image forming apparatus is in the first remote operation provision state or the second remote operation provision state, if it receives the start instruction for the second remote operation, it will not accept the second operation and will not start providing the second remote operation. A system configured in such a way.

11. An image forming apparatus that can connect to the internet, When the image forming apparatus receives a command to start a first remote operation from the first information processing device via the cloud storage connected to the Internet, it executes a program upload process to upload a first remote operation program, which can be run in a browser, to the cloud storage. Once the first remote operation program is uploaded to the cloud storage, the browser installed on the first information processing device can cause the first information processing device to download the first remote operation program from the cloud storage. The image forming apparatus is further capable of performing an image upload process to upload first image data showing the first remote operation screen to the cloud storage after receiving the start instruction for the first remote operation, the first image data being downloaded to the first information processing device by the first remote operation program, and the first remote operation screen shown in the first image data being displayed on the user interface of the first information processing device by the browser. The image forming apparatus further downloads the first operation data from the cloud storage when the first operation data is uploaded to the cloud storage, executes a process corresponding to the operation shown in the first operation data, and if the operation shown in the first operation data is an operation related to image forming, executes an image forming process as the process corresponding to that operation, the first operation data is data indicating the content of the operation on the first remote operation screen, and when the first information processing device receives an operation on the first remote operation screen, the first operation data is uploaded to the cloud storage by the first remote operation program, When the image forming apparatus receives a command to start the second remote operation from the second information processing device via local communication without using the cloud storage, it executes a program transmission process to send a second remote operation program, which is executable in a browser, to the second information processing device without using the cloud storage, and the browser installed in the second information processing device acquires the second remote operation program transmitted from the image forming apparatus. The image forming apparatus is further capable of performing an image transmission process to send a second image data showing the second remote operation screen to the second information processing device without going through the cloud storage, after receiving the start instruction for the second remote operation, the second image data being downloaded to the second information processing device by the second remote operation program, and the second remote operation screen shown in the second image data being displayed on the user interface of the second information processing device by the browser. The image forming apparatus, upon receiving second operation data from the second information processing apparatus, executes a process corresponding to the operation indicated in the received second operation data. If the operation indicated in the received second operation data is an operation related to image forming, it executes an image forming process as the process corresponding to that operation. The second operation data is data indicating the content of an operation on the second remote operation screen. When the second information processing apparatus receives an operation on the second remote operation screen, it is transmitted to the image forming apparatus by the second remote operation program. After the image forming apparatus receives the start instruction for the first remote operation, it starts providing the first remote operation by executing the program upload process, and remains in the first remote operation provision state until the termination conditions for the first remote operation are met. After receiving the start instruction for the second remote operation, the image forming apparatus starts providing the second remote operation by executing the program transmission process, and remains in the second remote operation provision state until the termination conditions for the second remote operation are met. When the image forming apparatus is not in the state of providing the first remote operation, if it receives the start instruction for the second remote operation from the second information processing device, it will be able to start providing the second remote operation. When the image forming apparatus is in the first remote operation provision state and receives the start instruction for the second remote operation from the second information processing device, it will not start providing the second remote operation. An image forming apparatus configured as follows.

12. An image forming apparatus according to claim 11, When the image forming apparatus is not in the second remote operation provision state, and receives the start instruction for the first remote operation from the first information processing device, it enables the provision of the first remote operation. When the image forming apparatus is in the second remote operation provision state and receives the start instruction for the first remote operation from the first information processing device, it does not start providing the first remote operation. An image forming apparatus configured as follows.

13. An image forming apparatus according to claim 11, The image forming apparatus includes a display, which displays a main unit operation screen, and in the image upload process, it uploads image data to the cloud storage as the first image data representing the first remote operation screen, which virtually reproduces the main unit operation screen displayed on the display, and the uploaded image data is downloaded to the first information processing apparatus by the first remote operation program, and the first remote operation screen that virtually reproduces the main unit operation screen based on the first image data downloaded from the cloud storage is displayed on the user interface of the first information processing apparatus by the browser. When the image forming apparatus is not in the first remote operation provision state, and receives the start instruction for the second remote operation from the second information processing device and starts providing the second remote operation, the image transmission process transmits image data for virtually reproducing the main unit operation screen displayed on the display to the second information processing device as the second image data showing the second remote operation screen, and the second remote operation program displays the second remote operation screen, which virtually reproduces the main unit operation screen based on the second image data received by the second information processing device, on the user interface of the second information processing device via the browser. An image forming apparatus configured as follows.

14. An image forming apparatus according to claim 11, The image forming apparatus includes a touch panel, displays a main panel screen including multiple controls on the touch panel, and when it receives an operation on one of the multiple controls, it executes processing based on the operation received by the control. When the image forming apparatus is in the first remote operation provision state and receives an operation on one of the plurality of operators, it can execute the image upload process, and in the image upload process, image data for virtually reproducing the main unit panel screen displayed on the touch panel is uploaded to the cloud storage as the first image data showing the first remote operation screen, the first image data is downloaded to the first information processing device by the first remote operation program, and the first remote operation screen that virtually reproduces the main unit panel screen based on the first image data downloaded from the cloud storage is displayed on the user interface of the first information processing device by the browser, and when an operation on one of the plurality of operators displayed on the first remote operation screen is received, the first remote operation program uploads the first operation data to the cloud storage, and the uploaded first operation data is information that identifies the operator that received the operation among the plurality of operators displayed on the first remote operation screen. When the first operation data is uploaded to the cloud storage, the image forming apparatus downloads the first operation data from the cloud storage and performs processing based on the operator identified by the first operation data. When the image forming apparatus is in the second remote operation provision state and receives an operation to one of the plurality of operators, it can execute the image transmission process, and in the image transmission process, it transmits image data for virtually reproducing the main panel screen displayed on the touch panel as the second image data showing the second remote operation screen to the second information processing device, the second image data is received by the second remote operation program to the second information processing device, and the second remote operation screen that virtually reproduces the main panel screen based on the received second image data is displayed on the user interface of the second information processing device by the browser, and when an operation to one of the plurality of operators displayed on the second remote operation screen is received, the second remote operation program transmits the second operation data to the image forming apparatus, and the transmitted second operation data is information that identifies the operator that received the operation from among the plurality of operators displayed on the second remote operation screen. When the image forming apparatus receives the second operation data transmitted from the second information processing device, it performs processing based on the operator identified by the received second operation data. An image forming apparatus configured as follows.

15. An image forming apparatus according to claim 11, The image forming apparatus, while in the state of providing the first remote operation, does not start providing the first remote operation even if it receives the start instruction for the first remote operation from an information processing device other than the first information processing device. An image forming apparatus configured as follows.

16. An image forming apparatus according to claim 11, The image forming apparatus, when it is neither in the first remote operation provision state nor the second remote operation provision state, receives the start instruction for the first remote operation from the first information processing device via the cloud storage, is configured to receive a first operation via the user interface of the image forming apparatus indicating whether or not to approve the provision of the first remote operation, and if it receives the first operation approving the provision of the first remote operation, it starts providing the first remote operation, and if it receives the first operation not approving the provision of the first remote operation, it does not start providing the first remote operation. When the image forming apparatus is in the first remote operation provision state or the second remote operation provision state, if it receives the start instruction for the first remote operation from the first information processing device, it will not accept the first operation and will not start providing the first remote operation. The image forming apparatus, when it is neither in the first remote operation provision state nor the second remote operation provision state, receives the start instruction for the second remote operation from the second information processing device via local communication, is configured to receive a second operation via the user interface of the image forming apparatus indicating whether or not to approve the provision of the second remote operation, and if it receives the second operation approving the provision of the second remote operation, it starts providing the second remote operation, and if it receives the second operation not approving the provision of the second remote operation, it does not start providing the second remote operation. When the image forming apparatus is in the first remote operation provision state or the second remote operation provision state, if it receives the start instruction for the second remote operation from the second information processing device, it will not accept the second operation and will not start providing the second remote operation. An image forming apparatus configured as follows.

Images