System and image forming apparatus

Abstract

To provide a new technique to control an image forming apparatus through remote control.SOLUTION: An MFP 1 can upload panel image data for virtually reproducing a screen being displayed on a touch panel 131 to a structured storage 202, a browser 62 installed in a PC 5 can acquire a P 262 for a browser that can operate in the browser 62, and the P 262 for a browser causes the PC 5 to download the panel image data and causes the browser 62 to display a panel image 531. When the P 262 for a browser receives an operation for the panel image 531, it causes the PC 5 to upload operation information to the structured storage 202, the MFP 1 downloads the operation information, executes processing corresponding to the operation, and when changing the screen being displayed, uploads panel image data of the screen after change, and the P 262 for a browser causes the PC to download the panel image data, and causes the browser 62 to display the panel image 531.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technical field disclosed in this specification relates to a system for controlling an image forming apparatus from an information processing apparatus. [Background technology]

[0002] There is known a technology for an image forming device capable of performing image formation such as printing and scanning, which accepts various input operations via an operation panel. For example, Patent Document 1 discloses a function executing device that includes a printing unit, an image reading unit, and an operation panel, in which the operation panel includes hardware keys and a touch panel, and accepts input operations via the hardware keys and icons or buttons displayed on the touch panel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-73543 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been an increasing demand for remote operation of an image forming apparatus, which allows an information processing device to input data to an operation panel of the image forming apparatus without touching the operation panel. Patent Document 1 does not disclose a technology for remotely controlling an image forming apparatus, and there is room for improvement. [Means for solving the problem]

[0005] A system made to solve this problem includes an image forming apparatus and an information processing apparatus, wherein the information processing apparatus and the image forming apparatus are connected to the Internet, the image forming apparatus has a display and is capable of displaying a main body operation screen on the display, the information processing apparatus is capable of installing a browser, and the browser is capable of causing the information processing apparatus to acquire a remote operation program operable on the browser, the image forming apparatus is capable of executing an image upload process for uploading image data for virtually reproducing the main body operation screen to be displayed on the display to cloud storage connected to the Internet, and the remote operation program causes the information processing apparatus to download the image data from the cloud storage and execute a screen display process for displaying a remote operation screen, which virtually reproduces the main body operation screen based on the image data, on a user interface of the information processing apparatus by the browser, and the remote operation program The program further causes the information processing device to execute an operation upload process for uploading operation data indicating the operation content to the cloud storage when the program accepts an operation on the remote operation screen via the user interface, and the image forming device, when the operation data is uploaded to the cloud storage, downloads the operation data from the cloud storage and executes a process corresponding to the operation indicated in the operation data, and if the operation indicated in the operation data is an operation related to image formation, executes an image formation process as a process corresponding to the operation, and when an event occurs that changes the content of the main body operation screen, the image forming device further executes the image upload process to upload the image data for virtually reproducing the main body operation screen after the change to the cloud storage, and the remote operation program causes the information processing device to execute the screen display process when the image data for virtually reproducing the main body operation screen after the change is uploaded to the cloud storage, and in the screen display process,The image data for virtually reproducing the changed main body operation screen is downloaded from the cloud storage, and the remote operation screen, which is a virtual reproduction of the changed main body operation screen based on the downloaded image data, is displayed on the user interface using the browser.

[0006] According to the system disclosed in this specification, a browser installed on an information processing device can acquire a remote operation program, and image data for virtually reproducing the main body operation screen displayed on the display of the image forming device is passed from the image forming device to the remote operation program via cloud storage. This makes it possible to display a remote operation screen on the information processing device that can accept operations equivalent to those on the main body operation screen, enabling remote operation to be achieved with the same ease of use as the image forming device itself.

[0007] A control method for realizing the functions of the above system, an image forming apparatus included in the system, a computer program executed by each apparatus, and a computer-readable storage medium storing the computer program are also novel and useful. [Effects of the Invention]

[0008] The technology disclosed in this specification realizes a new technology for remotely controlling an image forming apparatus. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an explanatory diagram showing an overview of a remote control system according to an embodiment of the present invention; [Figure 2] 1 is an explanatory diagram showing an overview of a remote control system according to an embodiment of the present invention; [Figure 3] FIG. 10 is a sequence diagram illustrating an example of a start instruction procedure. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of a data structure. [Figure 5] 10 is a flowchart illustrating an example of a procedure for a start determination process. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a usage permission confirmation screen. [Figure 7] FIG. 10 is a sequence diagram illustrating an example of a screen preparation procedure. [Figure 8] FIG. 10 is an explanatory diagram showing an example of an execution file. [Figure 9] 10 is a flowchart illustrating an example of a procedure for an image storage process. [Figure 10] FIG. 2 is an explanatory diagram illustrating an example of information stored in structured storage. [Figure 11] FIG. 10 is an explanatory diagram showing an example of a state in which screens are synchronized. [Figure 12] FIG. 10 is a sequence diagram illustrating an example of a remote operation procedure. [Figure 13] 10 is a flowchart illustrating an example of a procedure for a browser image change process. [Figure 14] 10 is a flowchart illustrating an example of a procedure for device image change processing. [Figure 15] 10 is a flowchart illustrating an example of a procedure for operation information processing. [Figure 16] FIG. 10 is a sequence diagram illustrating an example of an operation ending procedure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, a detailed description will be given of an embodiment of a remote control system with reference to the accompanying drawings. This specification discloses a remote control system for remotely controlling a multifunction peripheral (hereinafter referred to as "MFP") having a printing function via the Internet.

[0011] 1, the remote control system 100 of this embodiment includes one or more MFPs 1, 2, and 3 to be operated, and a personal computer (hereinafter referred to as "PC") 5 used by a user who performs remote operation, and is a system that enables each MFP to be operated from the PC 5 at a remote location via a cloud storage service 200. The MFPs 1, 2, and 3 are examples of image forming apparatuses, and the PC 5 is an example of an information processing apparatus. The remote control system 100 is an example of a system. The cloud storage service 200 is a service that has a communication function via an Internet line and is capable of storing various types of information.

[0012] The PC 5 is located, for example, at a head office, a system management company, or a management organization that manages an operational organization, and the MFPs 1-3 are located, for example, at each branch office, a client company, or an operational organization that runs a business using the MFP 1. The MFPs 1-3 may be the same model or different models. A user who performs remote operation using the PC 5 specifies the MFP to be operated and operates it. Note that the following describes the operation procedure when remote operation is performed via the cloud storage service 200 targeting the MFP 1, but the same procedure applies when targeting other MFPs.

[0013] 1, the MFP 1 that is the target of operation by the remote operation system 100 includes a controller 10 that includes a CPU 11 and a memory 12. The MFP 1 also includes a user interface (hereinafter referred to as "user IF") 13, a communication interface (hereinafter referred to as "communication IF") 14, a print engine 15, and a scanner 16, which are electrically connected to the controller 10.

[0014] The CPU 11 of the MFP 1 executes various processes in accordance with programs read from the memory 12 and based on user operations. The memory 12 of the MFP 1 can store various programs and data, including an image processing program 21, display image data 22, a remote control program 23, access information 24, identification information 25, and an executable file 26. The memory 12 is also used as a work area when various processes are executed. The buffer provided in the CPU 11 is also an example of the memory 12.

[0015] The image processing program 21 is a program for causing the MFP 1 to perform various types of image processing. The display image data 22 is a data group representing images used as components of various screens displayed on the user IF 13 of the MFP 1. The remote operation program 23 is a program for accepting remote operations. The access information 24 is information for accessing the cloud storage service 200. The identification information 25 is unique information for identifying the MFP 1. The executable file 26 is a data file containing a program for causing a PC or the like to execute a remote operation. Details of the programs and data will be described later.

[0016] It should be noted that the access information 24 is not stored in the memory 12 when the MFP 1 is shipped from the factory. In addition, the remote operation program 23 and the executable file 26 may or may not be stored in the memory 12 when the MFP 1 is shipped from the factory.

[0017] Note that an example of memory 12 is not limited to a ROM, RAM, HDD, etc. built into MFP 1, but may also be a storage medium that is readable and writable by CPU 11. A computer-readable storage medium is a non-transitory medium. In addition to the above examples, non-transitory media also include recording media such as CD-ROMs and DVD-ROMs. Non-transitory media are also tangible media. On the other hand, an electrical signal carrying a program downloaded from a server on the Internet is a computer-readable signal medium, which is a type of computer-readable medium, but is not included in non-transitory computer-readable storage media.

[0018] The user IF 13 includes hardware that displays a screen for notifying the user of information and hardware that accepts operations by the user. The user IF 13 of the MFP 1 includes a touch panel 131 that has a screen display function and an operation acceptance function, and a plurality of hardware keys (hereinafter referred to as "hard keys") 132. The user IF 13 is an example of an operation panel, and the touch panel 131 is also an example of a display.

[0019] The communication IF 14 includes hardware for communicating with external devices. The communication IF 14 includes functions compatible with communication standards such as Wi-Fi (registered trademark), Ethernet (registered trademark), and USB. The communication IF 14 can communicate with servers on the Internet using the HTTPS protocol.

[0020] The print engine 15 includes a configuration for printing an image based on image data onto a print medium such as a sheet. The print method of the print engine 15 is, for example, an electrophotographic method or an inkjet method. The scanner 16 includes a configuration for reading an image of an original document and acquiring image data.

[0021] 1, the PC 5 includes a controller 50 including a CPU 51 and a memory 52. ​​The PC 5 also includes a user IF 53 and a communication IF 54, which are electrically connected to the controller 50. The user IF 53 includes hardware for displaying a screen for notifying the user of information, and hardware for accepting operations by the user. The user IF 53 may or may not include a touch panel. The communication IF 54 has a function for connecting to the Internet using the HTTPS protocol.

[0022] The memory 52 of the PC 5 can store various programs and data, including an operating system (hereinafter referred to as "OS") 61, a browser 62, and an application program (hereinafter referred to as "start application") 63 for assisting in starting remote operation. Details of the programs and data will be described later.

[0023] Cloud storage service 200 is, for example, a service operated by a third party other than the company that uses MFP1 or the vendor of MFP1, and includes a group of storages accessible via the Internet. Cloud storage service 200 includes a first service that includes an unstructured storage area in which various data, such as files containing binary data, can be placed, and a second service that includes an area in which structured text data can be stored. Cloud storage service 200 is, for example, the Azure (registered trademark) storage service provided by Microsoft (registered trademark).

[0024] The cloud storage service 200 has a storage function as a data storage location, as well as a web server function. For example, when the first service or the second service of the cloud storage service 200 receives HTTPS access, it acts as a web server and can return a response that can be displayed by a browser.

[0025] Furthermore, the cloud storage service 200 is not limited to Microsoft Azure, but may be, for example, Google (registered trademark) Cloud, Amazon (registered trademark) AWS (registered trademark), etc. Furthermore, the cloud storage service 200 may be, for example, a service operated by a vendor of an image forming apparatus. Furthermore, for example, the first service may be a BLOB storage service, and the second service may be a Table storage service.

[0026] For example, a system administrator of a company or the like that uses the remote operation system 100 of this embodiment makes preparations to use remote operation via the cloud storage service 200. Specifically, the system administrator makes a contract with the company that operates the cloud storage service 200 and obtains an account name for using the cloud storage service 200.

[0027] Furthermore, the system administrator sets up storage areas in cloud storage service 200 corresponding to the accounts indicated by the acquired account names, for each of the first service and the second service, and acquires tokens for each service. Hereinafter, the storage areas in cloud storage service 200 corresponding to the accounts used in remote operation system 100 and set up in the first service will be referred to as unstructured storage 201, and the storage area set up in the second service will be referred to as structured storage 202. Unstructured storage 201 and structured storage 202 are examples of cloud storage.

[0028] Furthermore, the system administrator sets at least the unstructured storage 201 to prohibit anonymous access. An area set to prohibit anonymous access only accepts access requests from URLs that have a valid token attached as a query, and does not accept access if no query is attached or if the attached query is invalid. Furthermore, even if a file written in an area set to prohibit anonymous access contains link information to other files, link access using that link information is prohibited. Each token is, for example, a SAS (Shared Access Signature) token.

[0029] 2, the system administrator causes each MFP to be operated by remote operation system 100 to store access information 24 including unstructured storage access information 241, which is access information for accessing unstructured storage 201, and structured storage access information 242, which is access information for accessing structured storage 202. The system administrator stores access information 24 in a nonvolatile area of ​​the memory of each MFP.

[0030] 2, the unstructured storage access information 241 includes, for example, an account name which is text indicating an account acquired by a system administrator, an unstructured storage name which is text indicating the unstructured storage 201, and an unstructured storage token which is an SAS token for accessing the unstructured storage 201. In other words, the URL for accessing the unstructured storage 201 includes text in which the information shown in Fig. 2 is combined according to a predetermined rule. The unstructured storage access information 241 may be information written in the form of a URL, or may be information including each piece of information and the rules for combining them.

[0031] The structured storage access information 242 includes, for example, an account name which is text indicating an account acquired by a system administrator, a structured storage name which is text indicating the structured storage 202, and a structured storage token which is an SAS token for accessing the structured storage 202. The URL for accessing the structured storage 202 includes text in which the information shown in Fig. 2 is combined according to a predetermined rule. The structured storage access information 242 may be information written in the form of a URL, or may be information including the information and the rules for combining it.

[0032] Alternatively, each MFP may acquire a token. For example, a system administrator may store an account name in each MFP, and the MFP may use the account name to access cloud storage service 200 and acquire each token. Each MFP stores the acquired token in a non-volatile area of ​​its memory, including the access information 24.

[0033] 2, each MFP stores identification information 25 including a device ID 251 and a password 252. The configuration of each storage and the information stored therein will be described later.

[0034] As will be described in detail later, when each MFP uses remote operation system 100, it creates a folder in unstructured storage 201, for example, whose folder name includes its own device ID. For example, MFP 1 creates folder 2011 at A32 in FIG. 3, which will be described later. Furthermore, MFP 1 combines unstructured storage access information 241 stored in memory with its own device ID 251 to create access information for non-anonymously accessing folder 2011 created in unstructured storage 201 (for example, A33 in FIG. 3). For convenience, this access information is referred to as self-folder access information 243. Note that the system administrator may create a folder for each MFP in unstructured storage 201 in advance and have each MFP store access information for accessing that folder.

[0035] MFP1 can create unstructured storage access information 241 and structured storage access information 242 based on stored information. For example, a system administrator may store an account name in each MFP, and each MFP may combine the account name, a token, etc. to create unstructured storage access information 241 and structured storage access information 242. In this case, the token may be one acquired by each MFP, or may be one stored in each MFP by the system administrator. For example, MFP1 can create unstructured storage access information 241 and structured storage access information 242 using the stored account name.

[0036] By creating a data storage area called a folder whose name includes a device ID for each MFP, even if HTTPS requests specifying different MFPs are made simultaneously from different browsers, unstructured storage 201 can return an appropriate response based on the data stored in the folder for the MFP corresponding to each request. In other words, as will be described in more detail later, different MFPs can be remotely operated from different PCs using cloud storage service 200.

[0037] Furthermore, the system administrator stores access information 631 for accessing structured storage 202 in start application 63 installed on a PC or the like. Access information 631 is information used by start application 63 to access an area of ​​structured storage 202 that MFP1 can access using structured storage access information 242. Access information 631 may be the same as structured storage access information 242 used by MFP1, or may be partially different information. Specifically, access information 631 is in the form of a URL.

[0038] Regarding the access information 631, instead of storing a URL in the start application 63, an account name, a token, or the like may be stored, and the start application 63 may create a URL. Alternatively, the start application 63 may obtain a token using an account name stored in the PC, and create the access information 631 using the obtained token.

[0039] Furthermore, the account name may be common to the entire company, or may be different for each country or region, each business division, etc. However, the start application 63 of the PC that performs the remote operation can execute the remote operation only for each MFP that uses the structured storage 202 that it can access.

[0040] Next, a description will be given of a procedure for remotely operating the MFP 1 via the cloud storage service 200. Note that, hereinafter, remote operation via the cloud storage service 200 will be referred to as "cloud remote operation."

[0041] Note that the following processes basically refer to CPU processing in accordance with instructions written in a program. That is, processes such as "judging," "extracting," "selecting," "calculating," "deciding," "identifying," "acquiring," "receiving," and "control" in the following description represent CPU processing. CPU processing also includes hardware control using an OS API. In this specification, the operation of each program will be described without mentioning the OS. That is, in the following description, a statement to the effect that "Program B controls Hardware C" may also mean that "Program B controls Hardware C using the OS API." Furthermore, CPU processing in accordance with instructions written in a program may be described in abbreviated terms. For example, it may be described as "performed by the CPU." Furthermore, CPU processing in accordance with instructions written in a program may be described in abbreviated terms, such as "performed by Program A."

[0042] Note that "obtaining" is used as a concept that does not require a request. In other words, the process of receiving data without the CPU requesting it is also included in the concept of "the CPU obtaining data." Furthermore, "data" in this specification is represented as a bit string that can be read by a computer. Data with the same substantial meaning but different formats will be treated as the same data. The same applies to "information" in this specification. Furthermore, "requesting" and "instructing" are concepts that indicate outputting information indicating a request or an instruction to the other party. Furthermore, information indicating a request or an instruction will also be simply referred to as a "request" or "instruction."

[0043] Furthermore, the process by a CPU to determine whether information A indicates event B is sometimes conceptually described as "determining whether event B is true from information A." The process by a CPU to determine whether information A indicates event B or event C is sometimes conceptually described as "determining whether event B or event C is true from information A."

[0044] A start instruction procedure for starting cloud remote control of MFP1 by remote control system 100 using PC 5 will be described with reference to the sequence diagram of Fig. 3. A user who wants to perform cloud remote control, for example, a system administrator, starts start application 63 on PC 5, specifies MFP1, which is the device to be operated, and performs an operation to instruct the start of cloud remote control (A01). The start application 63 pre-registers the device ID of each MFP that can be the target of cloud remote control. In A01, start application 63 can identify the device ID of the device to be operated based on the user's operation.

[0045] To start cloud remote control, a password included in the identification information of the device to be controlled is required. The user also enters the password for MFP1 at A01. Authentication using a password can increase the security of cloud remote control.

[0046] Start application 63 uploads instruction data, which is information indicating the input start instruction, to structured storage 202 (A11). As described above, start application 63 has access information 631 for accessing structured storage 202, and can execute the upload by accessing structured storage 202 using that access information 631. Hereinafter, without any particular explanation, start application 63 will access structured storage 202 using access information 631.

[0047] Access information 631 is an example of second access information. Note that this example of second access information is information necessary for PC 5 or MFP 1 to access the same area of ​​structured storage 202, and may be a URL or information for creating a URL. Furthermore, the second access information used by PC 5 and the second access information used by MFP 1 may be the same information, or may be partially different information. Similarly, an example of first access information, which will be described later, is information necessary for PC 5 or MFP 1 to access the same area of ​​unstructured storage 201 (folder 2011 created by MFP 1), and may be a URL or information for creating a URL.

[0048] Structured storage 202 stores a structured database having a structure including a plurality of records. Each record includes, for example, a first key 2021, a second key 2022, a timestamp 2023, a device ID 2024, and parameters 2025, as shown in FIG. 4 . A system administrator registers information indicating the structure of the structured database, including the structure of each record, in structured storage 202, which enables structured storage 202 to store the structured database. Starting application 63 may also register information indicating the structure of the structured database in structured storage 202.

[0049] The first key 2021 and the second key 2022 are information indicating the type of content of the record. For example, in A11, the start application 63 uploads each piece of information to the structured storage 202 so that the information is stored in record R1, as shown in FIG. 4(A). Uploading information may be rephrased as uploading data. For convenience, uploading information so that each record stores information may also be described as uploading a record. Downloading information may be rephrased as downloading data.

[0050] When each device such as PC 5 or MFP 1 uploads each piece of information by specifying first key 2021 and second key 2022, structured storage 202 stores the uploaded information in a record that stores the information specified in both first key 2021 and second key 2022. Furthermore, when each device specifies first key 2021 and second key 2022 and requests the download of information stored in the corresponding record, structured storage 202 transmits to the requestor the information stored in the record that stores both first key 2021 and second key 2022 specified in the request. In other words, first key 2021 and second key 2022 are also information that identifies the record.

[0051] That is, initiation application 63 uploads information to structured storage 202 at A11 so that the information becomes record R1 shown in FIG. 4A. As a result, the first key 2021 of record R1 stores information indicating that the record stores information categorized as an instruction, and the second key 2022 stores information indicating that the information categorized as an instruction is an instruction for start processing. Furthermore, the device ID 2024 of record R1 stores the device ID indicating MFP1. As a result, record R1 becomes a record indicating instruction data addressed to MFP1.

[0052] The timestamp 2023 is information indicating the date and time when the data was stored in that record. The device ID 2024 is information identifying the device that is the target of the data stored in that record, and record R1 stores the device ID of the device that is the target of cloud remote control. The parameters 2025 include various information necessary for cloud remote control. Each record in this data structure further includes additional information as necessary. For example, as shown in FIG. 4(A), the instruction data uploaded in A11 stores, in the parameters 2025, information indicating that the process to be started is cloud remote control and the password entered in A01.

[0053] Meanwhile, each MFP that can be a target of cloud remote operation periodically accesses structured storage 202 using structured storage access information stored in its own memory, and checks whether there are any instructions that the MFP itself must process by monitoring records in which its own device ID 251 (see FIG. 2) is stored in device ID 2024 (A21). For example, MFP1 periodically accesses structured storage 202 using structured storage access information 242 (see FIG. 2). From now on, without any particular explanation, MFP1 will access structured storage 202 using structured storage access information 242.

[0054] When record R1 shown in FIG. 4(A) is uploaded in A11, MFP1 can download the stored data from record R1 in structured storage 202 (A22). Having acquired instruction data indicating instructions to itself from record R1, MFP1 starts processing based on the received instructions. As shown in FIG. 4(B), MFP1 uploads information indicating "in progress" as the progress status to structured storage 202 so that it can be stored in record R1. Furthermore, because the instruction in record R1 is an instruction to start cloud remote control, MFP1 executes start determination processing to determine whether to approve the start of cloud remote control (A23).

[0055] Meanwhile, after uploading the instruction data indicating an instruction to start cloud remote control in A11, the start application 63 of the PC 5 periodically accesses record R1 of the structured storage 202 and monitors whether information indicating a response has been stored in record R1 (A12). For example, if information indicating "in progress" or "completed" is not stored even after a predetermined time has elapsed since uploading record R1, the start application 63 may cancel the cloud remote control. Note that "completed" is uploaded by the MFP 1 so that it is stored in record R1 when the MFP 1 completes processing based on the instruction it received.

[0056] The procedure of the start determination process will be described with reference to the flowchart in Fig. 5. This start determination process is executed by the CPU 11 of the MFP 1 based on the remote operation program 23 when information indicating a start instruction for cloud remote operation is acquired.

[0057] In the start determination process, the CPU 11 compares the password stored in the downloaded record R1 with the password 252 of the identification information 25 stored in the memory 12, and determines whether the password is correct, for example, whether the two passwords correspond (S101). Specifically, the correspondence of the two passwords may mean that the two passwords match. If it is determined that the password is not correct (S101: NO), the CPU 11 decides not to start the cloud remote operation in response to the start instruction (S102).

[0058] Password 252 may be a password required to locally operate MFP 1. For example, if a user at a branch office needs to log in to MFP 1 to operate MFP 1, password 252 may be a password that needs to be input via user IF 13 to log in to MFP 1. Password 252 is a password for starting cloud remote operation, and may be different from the password used when using MFP 1.

[0059] Regarding a password required for locally operating the MFP1, the MFP1 may enter an account-locked state if local login authentication involving password entry fails a predetermined number of times or more. When the MFP1 is in an account-locked state, the CPU 11 may determine NO in S101 without comparing the password stored in record R1 with password 252. Alternatively, if the CPU 11 determines in S101 that the password is incorrect a predetermined number of times or more, the MFP1 may enter an account-locked state. The MFP1 may automatically release the account-locked state when a predetermined trigger, such as the passage of a predetermined time, is triggered.

[0060] In this way, in the remote control system 100, the password 252 for providing cloud remote control is stored in the MFP 1, and the MFP 1 performs authentication using the password 252, so that the security of the cloud remote control is high.

[0061] If it is determined that the password is correct (S101: YES), the CPU 11 determines whether or not the MFP 1 is currently accepting cloud remote control from another PC or the like (S111). The MFP 1 does not accept cloud remote control from multiple PCs at the same time. If the MFP 1 has already accepted a cloud remote control start instruction from another PC and is currently accepting cloud remote control from that PC (S111: YES), the CPU 11 determines not to start cloud remote control in response to the current cloud remote control start instruction (S102). Accepting cloud remote control from multiple external devices simultaneously could cause confusion. Since the MFP 1 is currently accepting cloud remote control from another device and is currently accepting cloud remote control, it will not start further cloud remote control, so there is little chance of confusion in operation. The determinations in S101 and S111 may be reversed.

[0062] If cloud remote control is not being executed (S111: NO), the CPU 11 displays a license confirmation screen (S121) on the touch panel 131 of the user IF 13. For example, as shown in Fig. 6, the CPU 11 displays a license confirmation screen 71 on the touch panel 131, which includes a message inquiring whether or not to approve the cloud remote control, a "Yes" button 711 indicating approval of the cloud remote control, and a "No" button 712 indicating rejection of the cloud remote control, and receives an input indicating a user selection operation (S122).

[0063] The CPU 11 waits until it receives a user input (S122: NO), and when it receives a user input (S122: YES), it determines whether the received button is the "Yes" button 711 or the "No" button 712 (S123). When it determines that an operation on the "Yes" button 711 has been received (S123: Yes button), the CPU 11 permits the start of cloud remote control (S131).

[0064] Cloud remote operation without a user in front of the MFP1 may reduce security. The MFP1 of this embodiment starts cloud remote operation when it accepts an operation on the "Yes" button 711 on the license confirmation screen 71, thereby increasing the security of the cloud remote operation. Note that if no user input is accepted even after a predetermined time has passed since the license confirmation screen 71 was displayed, the MFP1 may not start cloud remote operation. Also, for example, the MFP1 may be configured to omit accepting an operation on the license confirmation screen 71. Also, the MFP1 may be configured to start cloud remote operation if no user input is accepted even after a predetermined time has passed.

[0065] On the other hand, if it is determined that an operation on the "No" button 712 has been accepted (S123: No button), the CPU 11 rejects the start of the cloud remote operation (S102). During the cloud remote operation, the user of the PC 5 virtually views the display on the touch panel 131 and operates the MFP 1. By displaying the usage permission confirmation screen 71, the user who is using the MFP 1 can, for example, when a screen that the user does not want to view is displayed, by operating the "No" button 712 to wait for the start of the cloud remote operation. After S131 or S102, the CPU 11 returns the display on the touch panel 131 to the state before the usage permission confirmation screen 71 was displayed in S121, ends the start determination process, and returns to the start instruction procedure of FIG. 3.

[0066] Returning to the explanation of the start instruction procedure in Figure 3, if the MFP1 decides to start cloud remote operation in S131 of Figure 5 (alt: [Start]), it requests the unstructured storage 201 and structured storage 202 to delete data related to the cloud remote operation (A31). For example, if data uploaded by the previous cloud remote operation remains, it could be confused with instructions related to the current cloud remote operation. When starting cloud remote operation, the MFP1 deletes the data remaining in the unstructured storage 201 and structured storage 202, thereby avoiding the effects of the previous cloud remote operation.

[0067] Thereafter, the MFP 1 creates a folder 2011 (see FIG. 2) for its own device in unstructured storage 201 to be used in this cloud remote operation (A32). The MFP 1 creates folder 2011, for example, with its own device ID 251 (see FIG. 2) included in the folder name. Note that if folder 2011 has already been created, the MFP 1 may delete all data in that folder in A31 and skip A32. Furthermore, as described above, the MFP 1 creates own-folder access information 243 for accessing folder 2011 created in A32 in unstructured storage 201 (A33). The own-folder access information 243 is in the form of a URL. The MFP 1 may store the created own-folder access information 243 in memory 12. From now on, without any particular explanation, the MFP 1 will access folder 2011 created in unstructured storage 201 using the own-folder access information 243.

[0068] Then, the MFP 1 and PC 5 execute a screen preparation procedure (A35) for displaying, on the user IF 53 of PC 5, a screen that virtually reproduces the user IF 13 of MFP 1 that accepts cloud remote control. The screen preparation procedure will be described with reference to the sequence diagram of FIG. 7.

[0069] The MFP1 uses the self-folder access information 243 created in A33 to upload (B11) the executable file 26 (see FIG. 1) to the folder 2011 created in the unstructured storage 201 in A32 of FIG. 3. In other words, the executable file 26 is placed in the folder 2011.

[0070] The executable file 26 is, for example, as shown in FIG. 8 , a single HTML file written in HTML that includes a CSS 261 and a browser program (hereinafter referred to as a "browser P") 262. The CSS 261 is information that sets the layout of the Web page to be displayed, such as the color, size, background, and placement of text, in order to display an image virtually reproducing the user IF 13 of the MFP 1 on the browser 62 of the PC 5. The browser P 262 is, for example, a program written in JavaScript (registered trademark), is operable on the browser 62, and causes the browser 62 to execute processing for cloud remote control. The browser P 262 is an example of a remote control program.

[0071] Structuring access information 263 and hard key image data 264 are embedded in browser P262 of executable file 26. Structuring access information 263 is information used by browser P262 to access an area of ​​structured storage 202 that can be used by MFP1, and is in the form of a URL. Structuring access information 263 embedded in executable file 26 may be the same as structured storage access information 242 stored in MFP1, or may be partially different information. Note that structuring access information 263 may be information generated by CPU 11 of MFP1 based on an account name, token, etc. included in structured storage access information 242.

[0072] The hard key image data 264 is a plurality of image data corresponding to each hard key 132 included in the user IF 13, and is written in, for example, gif format. The hard key image data 264 includes, for example, icon images representing each hard key 132.

[0073] By uploading executable file 26 including structuring access information 263 for accessing structured storage 202, PC 5, which downloads this executable file 26 in a later procedure, can obtain browser P262 and structuring access information 263. Therefore, based on processing by browser P262, PC 5 can access structured storage 202 using structuring access information 263.

[0074] 8, the execution file 26 further includes a panel display element 266, a hard key display element 267, and end button image data 268. The panel display element 266 is an image embedding element for causing the browser page 262 to display a panel image that virtually reproduces the image displayed on the touch panel 131 of the MFP1. The hard key display element 267 is an image embedding element for causing the browser page 262 to display key images that virtually reproduce the hard keys 132 of the MFP1. The end button image data 268 is image data for causing the browser page 262 to display an end button 533, which will be described later.

[0075] Furthermore, the MFP 1 executes image storage processing (B12). The steps of the image storage processing will be described with reference to the flowchart of FIG. 9. The image storage processing is processing for uploading panel image data representing the currently displayed panel image to the cloud storage service 200 in order to display, on the user IF 53 of the PC 5, a panel image that virtually reproduces the screen currently displayed on the touch panel 131 of the user IF 13 of the MFP 1. Note that the image storage processing is executed by the CPU 11 of the MFP 1 when it is necessary to upload the panel image data. The screen currently displayed on the touch panel 131 is an example of a main body operation screen and an example of a main body panel screen.

[0076] The MFP 1 can display, for example, a screen including a plurality of icons on its touch panel 131, and can accept an instruction to execute a process indicated by the icon by operating an area where the icon is displayed. For example, when an operation on a copy icon included in a standby screen displayed on the touch panel 131 is accepted, the MFP 1 displays a setting screen including an icon for instructing a change in settings for the copy process, an input field for setting values ​​to be changed, and an icon for instructing execution of the copy process. When the MFP 1 accepts an operation to instruct a change in settings, the MFP 1 changes the settings for the copy process. Furthermore, when the MFP 1 accepts an operation on an icon for instructing execution of the copy process, the MFP 1 starts execution of the copy process. The icons displayed on the touch panel 131 are examples of controls. An operation on the copy icon, an operation on an icon for instructing a change in settings, input of setting values ​​in an input field, an operation on an icon for instructing execution of the copy process, etc. are examples of operations related to image formation, and the copy process is an example of the image formation process.

[0077] Furthermore, when the MFP 1 receives an operation on a hard key 132 included in the user IF 13, it can execute a process based on the received hard key 132. For example, when a back key operation is received, the MFP 1 can cancel the processing of the previously received operation and return to the state before the operation was received. For example, when a back key operation is received while a copy processing setting screen is displayed, the CPU 11 causes the touch panel 131 to display the previous screen, for example, a standby screen.

[0078] First, the CPU 11 acquires bitmap data indicating the screen currently being displayed on the touch panel 131 (S201). Here, a supplementary explanation of bitmap data will be provided. When displaying various screens on the touch panel 131, the MFP 1 generates bitmap data indicating the screen to be displayed. The MFP 1 stores image data that will be components of each screen in the memory 12, for example, as display image data 22 (see FIG. 1). The MFP 1 reads image data that will be components of the screen to be displayed, such as image data that indicates the background image of each screen and image data that indicates icons, from the display image data 22, and uses the read image data to generate image data indicating the screen and bitmap data.

[0079] Note that different screen IDs are assigned to screens with significantly different roles, such as a standby screen and a copy setting screen, and the MFP 1 can identify the role of the currently displayed screen based on the screen ID. For example, while the standby screen is displayed, the MFP 1 temporarily stores the screen ID of the standby screen in memory 12 as the screen ID of the currently displayed screen. When switching the display to the copy setting screen, the MFP 1 generates bitmap data representing the copy setting screen and temporarily stores the screen ID representing the copy setting screen in memory 12 as the screen ID of the currently displayed screen. The screen ID is an example of an identifier.

[0080] Furthermore, when an input to an input field for a setting value is accepted while the copy screen is displayed, the MFP 1 generates bitmap data representing a copy setting screen in which the content of the input field has been changed to indicate the input setting value, without changing the screen ID of the currently displayed screen. Thus, when the content of a screen is changed while the screen is being displayed, for example, by changing the appearance of an icon or by changing some text display, the MFP 1 generates bitmap data representing a screen with the changed content, without changing the screen ID of the currently displayed screen. Note that screens with significantly different roles, i.e., different screen IDs, often have significantly different overall layouts, but there are also screens with similar overall layouts, such as a copy setting screen and a scan setting screen, even if they have different screen IDs.

[0081] Based on the bitmap data acquired in S201, the CPU 11 generates panel image data (S202), which is image data to be displayed on the user IF 53 of the PC 5 in a later procedure. For example, the CPU 11 compresses the acquired bitmap data to generate PNG data.

[0082] The data representing the screen currently displayed on touch panel 131 is not limited to bitmap data, but may be, for example, PNG data or JPEG data. Furthermore, panel image data is not limited to compressed data representing the screen, such as bitmap data, but may be the data representing the screen itself. Furthermore, instead of creating panel image data from data representing the screen, CPU 11 may obtain image data of components included in the screen currently displayed on touch panel 131 and create the panel image data using the obtained image data.

[0083] Furthermore, if the currently displayed screen contains a moving image or animation, the CPU 11 may generate panel image data that does not include the moving image or animation, or may generate panel image data that includes a still image of a scene from the moving image or animation. Moving images, which tend to have a large data volume, impose a heavy communication load, and communication delays can easily cause timing differences in the content displayed on the PC 5 and MFP 1. Since the MFP 1 of this embodiment generates panel image data that does not include a moving image, an image that does not include a moving image is displayed on the user IF 53 of the PC 5, reducing display misalignment. A currently displayed screen without the moving image is an example of a main unit operation screen without the moving image, and the panel image data without the moving image is an example of image data without the moving image.

[0084] Then, CPU 11 determines whether the data size of the generated panel image data is smaller than a predetermined size (S205). The panel image data to be uploaded may have different compression effectiveness depending on the type of screen, and the data size after compression may or may not be large. For example, a screen that contains only icons and menus, such as a standby screen, tends to be smaller in size through compression, while a screen that includes a print preview image or a scanned image tends to have a larger data size. Furthermore, the data size that can be stored in each record of structured storage 202 is subject to a limit set by cloud storage service 200.

[0085] If it is determined that the size is smaller than the predetermined size (S205: YES), CPU 11 encodes the panel image data into a text data format in accordance with a predetermined rule (S211). Because the text data format is suitable as a format for data to be stored in each record in structured storage 202, CPU 11 encodes the panel image data into the text data format. However, if a data format other than the text data format that is suitable for storage in structured storage 202 is defined in cloud storage service 200, CPU 11 may encode the panel image data into that data format. Furthermore, if the format of the panel image data is a data format that is suitable for storage in structured storage 202, CPU 11 may not encode the panel image data.

[0086] Then, CPU 11 uploads the encoded panel image data to structured storage 202 (S212, B13 in FIG. 7). Specifically, MFP 1 uploads each piece of information to structured storage 202 so that it becomes, for example, record R21 shown in FIG. 10. That is, MFP 1 uploads each piece of information to structured storage 202 so that, of a record in which first key 2021 is stored with information indicating a panel image and second key 2022 is stored with information indicating image data, the device ID of MFP 1 is stored in device ID 2024, and the panel image data encoded in text data format is stored in binary 2026. S212 and B13 in FIG. 7 are an example of image upload processing.

[0087] On the other hand, if it is determined that the data size of the panel image data is not smaller than the predetermined size, that is, that the size is too large to be stored in a record of structured storage 202 (S205: NO), CPU 11 uses the above-mentioned own folder access information 243 (see FIG. 2) to access folder 2011 of unstructured storage 201 and uploads the panel image data to unstructured storage 201 (S213, B14 in FIG. 7). S213 and B14 in FIG. 7 are an example of image upload processing.

[0088] That is, MFP1 creates a file containing panel image data and uploads it to folder 2011 generated in A32 of Fig. 3. Folder 2011 is a folder whose folder name includes the device ID of MFP1. Note that in this case, MFP1 does not upload the panel image data to structured storage 202. That is, the panel image data is not stored in record R21.

[0089] The panel image data uploaded to structured storage 202 or unstructured storage 201 is read by browser 62 of PC 5 in a procedure described below. The panel image data includes image data representing a screen that virtually reproduces, on user IF 53 of PC 5, a screen displayed on user IF 13 of MFP 1. When using cloud storage service 200, different upload destinations are used depending on the size of the image data, so even panel image data with a large data size can be passed to browser 62 via cloud storage service 200.

[0090] Furthermore, CPU 11 generates image information that is information related to the uploaded panel image data and uploads it to structured storage 202 (S215, B15 in FIG. 7). MFP 1 uploads the image information to structured storage 202 so that it becomes, for example, record R22 shown in FIG. 10. That is, MFP 1 stores the device ID of MFP 1 in device ID 2024 of a record in which first key 2021 stores information indicating that it is a panel image and second key 2022 stores information indicating that it is image information, and uploads each piece of information to structured storage 202 so that information indicating the number of times the panel image data has been changed, information indicating the storage location of the panel image data, and information indicating the file name of the panel image data are stored in parameters 2025. Note that the information indicating the storage location of the panel image data includes information indicating whether the panel image data is stored in record R21 or whether the panel image data has been uploaded to folder 2011 of unstructured storage 201. For convenience, below, the information stored in parameter 2025 of record R22, i.e., information indicating the number of times the panel image data has been changed, information indicating the storage location of the panel image data, and information indicating the file name of the panel image data, will be referred to as image information.

[0091] 7 is the first panel image data of this cloud remote control, the MFP1 sets the number of changes included in the image information to be uploaded in B15 to "1." Note that, as will be described in detail later, the MFP1 may upload the panel image data again in a subsequent procedure, and each time an upload is performed, the MFP1 increments the number of changes by 1 and stores the incremented number in the record of the image information.

[0092] Record R21, which stores panel image data, and record R22, which stores image information, are a pair of records for uploading one piece of panel image data. Although not shown, structured storage 202 also has another pair of records prepared, similar to the record pair of records R21 and R22. When uploading information related to panel image data from the next time onward, MFP 1 uploads the information related to panel image data to the record of the record pair with the fewer number of changes stored. In other words, MFP 1 uploads information related to the two record pairs alternately. Alternatively, MFP 1 may upload information related to panel image data to the record pair with the older stored timestamp.

[0093] There may be three or more record pairs that store information about panel image data. In this case, the MFP 1 uploads information about panel image data to the record of the record pair that has the fewest number of changes stored. Alternatively, the MFP 1 may upload information about panel image data to the record pair that has the oldest stored timestamp.

[0094] After S215, the CPU 11 enters a state in which cloud remote control is being provided, ends the image storage process, and returns to the screen preparation procedure of FIG.

[0095] If MFP 1 determines that the panel image data is too large to be stored in a record of structured storage 202, instead of uploading it to unstructured storage 201, MFP 1 may divide the panel image data into multiple data pieces and upload them to multiple records of structured storage 202. For example, MFP 1 may divide the panel image data into data representing the upper half of the screen and data representing the lower half of the screen, and upload the data so that each data piece is stored in two records. MFP 1 may also determine the number of divisions based on the data size of the panel image data. In this case, MFP 1 may encode each of the divided multiple pieces of panel image data into text data format and store them in multiple records, respectively, and store information indicating the number of divisions and information indicating the order of each record as image information. Even in this case, by uploading divided data based on the data size of the panel image data, large image data can be passed to browser 62 via cloud storage service 200.

[0096] Returning to the explanation of the screen preparation procedure in Fig. 7, in Fig. 7, in the image storage process of B12, the procedure for uploading panel image data to structured storage 202 (S212 in Fig. 9) is shown as B13, and the procedure for uploading to unstructured storage 201 (S213 in Fig. 9) is shown as B14. Also, the procedure for uploading image information to structured storage 202 (S215 in Fig. 9) is shown as B15. After uploading executable file 26 in B11 and uploading panel image data and image information in B12, MFP1 uploads response data indicating that processing has been completed to structured storage 202 in response to the instruction data acquired in A22 in Fig. 3 (B17). Note that MFP1 may execute the image storage process of B12 after uploading the response data.

[0097] The MFP1 uploads the response data so that it is stored in record R1 (see FIG. 4) that includes the instruction data. Specifically, the MFP1 uploads each piece of information so that, for example, as shown in FIG. 4(C), record R1 stores information indicating "completed" as the progress status, stores information indicating "success" as the result, and stores access information for downloading the executable file 26 uploaded in B11 as additional information 2027.

[0098] The access information stored in additional information 2027 of record R1 is information for accessing folder 2011, which was created in unstructured storage 201 and includes the device ID of MFP1 in its folder name. The access information stored in additional information 2027 of the response data is an example of first access information. The access information stored in additional information 2027 is in the form of a URL. The access information stored in additional information 2027 may be the same as self-folder access information 243 created in A33, or may be partially different information. MFP1 may store self-folder access information 243 directly in additional information 2027. Furthermore, MFP1 may create the access information to be stored in additional information 2027 by, for example, combining unstructured storage access information 241 and device ID 251 stored in memory 12.

[0099] Meanwhile, as described above, after uploading record R1 including instruction data in A11, start application 63 on PC 5 periodically accesses structured storage 202 and monitors whether there is a response to the instruction in record R1 (A12). When the progress status information included in record R1 changes from "in progress" to "completed," if information indicating "success" is stored as a result, start application 63 can start cloud remote control.

[0100] The start application 63 acquires access information from the additional information 2027 of the record R1 in which information indicating "success" is stored (B21). The start application 63 then launches the browser 62 and passes the access information acquired from the additional information 2027 to the launched browser 62 (B22).

[0101] The browser 62 downloads the executable file 26 from the folder 2011 of the unstructured storage 201 through non-anonymous access using the access information passed from the initiating application 63 (B31). As described above, the executable file 26 includes the CSS 261, and the browser 62 sets the display configuration of the web page based on the CSS 261 of the obtained executable file 26. In addition, the executable file 26 includes a browser P262, so the browser 62 starts executing processing based on the browser P262 (B32).

[0102] Furthermore, since executable file 26 includes structuring access information 263, browser 62 uses structuring access information 263 to access structuring storage 202 and download image information (B33). The image information is information uploaded by MFP1 in B15. In B33, browser 62 downloads image information from the record in which image information with the number of changes equal to "1" is stored. As described above, MFP1 uploaded record R22 in B15 of FIG. 7 so that image information with the number of changes equal to "1" is stored. Browser 62 downloads the image information stored in this record R22.

[0103] That is, by downloading the executable file 26, the browser 62 can acquire the structuring access information 263 and can access the structured storage 202 using the structuring access information 263. Therefore, the initiation application 63 does not need to prepare the structuring access information 263. Alternatively, the initiation application 63 may generate the structuring access information 263 and pass it to the browser 62.

[0104] Browser 62 downloads panel image data from structured storage 202 or unstructured storage 201 based on the information stored in parameter 2025 of record R22 of the image information (B34 or B35). If the storage location of the panel image data is structured storage 202 (alt:[structured storage]), browser 62 downloads the panel image data stored in record R21 of structured storage 202 (B34). If the storage location of the panel image data is unstructured storage 201 (alt:[unstructured storage]), browser 62 downloads the panel image data stored in folder 2011 of unstructured storage 201 (B35).

[0105] As described above, when the data size of the panel image data is large, MFP 1 may divide the panel image data and upload it to structured storage 202 instead of uploading it to unstructured storage 201. In this case, browser 62 can acquire the panel image data by downloading the divided pieces of panel image data from structured storage 202 based on the image information and integrating them based on the image information.

[0106] 11, for example, the browser 62 displays a panel image 531 on the user IF 53 at a position corresponding to the panel display element 266 (B37). The panel image 531 is an example of a remote operation screen, and B37 is an example of screen display processing. The panel image 531 displayed on the user IF 53 of the PC 5 is an image that virtually reproduces the screen currently being displayed on the touch panel 131 of the user IF 13 of the MFP 1.

[0107] 11, based on the hard key image data 264 (B38). The three-key images 532 displayed on the user IF 53 of the PC 5 correspond to the three hard keys 132 included in the user IF 13 of the MFP 1. As a result, an image virtually reproducing the user IF 13 of the MFP 1, including the touch panel 131 and the hard keys 132, is displayed on the user IF 53 of the PC 5.

[0108] 11 shows an example in which a standby screen is displayed, the screen displayed in B37 is a screen that virtually reproduces the screen currently displayed on touch panel 131 of MFP1, and is not limited to the standby screen. For example, if an error occurs in MFP1 and an error screen is displayed on touch panel 131, the error screen is also displayed on user IF 53 of PC 5. Furthermore, as described above, if a copy setting screen is displayed on touch panel 131, the copy setting screen is also displayed on user IF 53 of PC 5.

[0109] The screen displayed on the user IF 53 by the browser 62 includes a panel image 531 that virtually reproduces the screen displayed on the touch panel 131 of the MFP 1, and key images 532 that virtually reproduce the hard keys 132. In other words, because an image that virtually reproduces the user IF 13 of the MFP 1 is displayed on the user IF 53 of the PC 5, the user of the PC 5 can perform operations on the user IF 53 similar to those performed on the MFP 1 in subsequent procedures, facilitating cloud remote control. Note that the arrangement and shape of the key images 532 displayed on the user IF 53 are preferably the same as those on the user IF 13 of the MFP 1, but they do not have to be the same.

[0110] Furthermore, an end button 533 is displayed on the user IF 53 of the PC 5. The end button 533 is a button that is not included in the user IF 13 of the MFP 1, and is a button that is operated by the user who is performing cloud remote control on the PC 5 to instruct the end of the cloud remote control.

[0111] Now that the screen preparation procedure in Fig. 7 is complete, we return to the explanation of the start instruction procedure in Fig. 3. Next, we will explain the procedure when it is decided not to start cloud remote control in the start determination process of A23 in Fig. 3. The rest of the procedure when it is decided to start cloud remote control in A23 in Fig. 3 will be described later.

[0112] For example, as shown in FIG. 5, if the MFP1 determines that the password included in the instruction data is incorrect (NO in S101), if the MFP1 is already accepting cloud remote control from another PC or the like (YES in S111), or if the MFP1 has accepted operation of the "No" button 712 on the usage permission confirmation screen 71 (see FIG. 6) (No button in S123), the MFP1 decides not to start cloud remote control (S102).

[0113] Then, if it is determined in the start determination process of A23 that cloud remote control will not be started (alt: [Do not start]), the MFP1 does not upload the executable file 26 and uploads information indicating that the process will not be started to structured storage 202 (A41). As shown in FIG. 4(D), the MFP1 uploads each piece of information so that record R1 stores information indicating "Ended," which indicates that the process has been completed, as the progress status, and information indicating "Failed," as the result. Note that the MFP1 may also store information indicating the reason for not starting cloud remote control in record R14.

[0114] The start application 63 of the PC 5 monitors whether or not there is response data to the instruction in record R1 (A12), and when the progress status of record R1 changes from "running" to "completed," it downloads the response data (A43). If information indicating "failure" is stored as the response data, the start application 63 displays a message indicating that the cloud remote control has been rejected on the user IF 53 (A44). Furthermore, the start application 63 deletes the instruction data uploaded in A11 from the structured storage 202 (A45). In this case, the cloud remote control does not start. A user who wants to perform cloud remote control performs an operation such as re-entering a password, and then inputs a new instruction to start the cloud remote control to the start application 63.

[0115] Next, the remote operation procedure, which is the operation procedure after the start of cloud remote operation, will be described with reference to the sequence diagram of Fig. 12. The remote operation procedure is executed by cooperation between PC 5 and MFP 1 after an image virtually reproducing the user IF 13 of MFP 1 is displayed on the user IF 53 of PC 5 by B37 and B38 of Fig. 7, as shown in Fig. 11. In PC 5, browser 62 executes browser image change processing based on the description of browser P262 (see Fig. 8) included in executable file 26 downloaded in B31 of Fig. 7. Furthermore, MFP 1 executes device image change processing based on remote operation program 23.

[0116] First, the procedure for the browser image change process executed by the browser 62 will be described with reference to the flowchart in Fig. 13 and, if necessary, the sequence diagram in Fig. 12. During the execution of the browser image change process, the browser 62 periodically accesses the structured storage 202 (C03 in Fig. 12), checks the record R22 (see Fig. 10) in which the above-mentioned image information is stored, and repeatedly determines whether the number of changes included in the parameter 2025 has been updated from the number of changes stored by the browser 62.

[0117] In the browser image change process, the browser 62 determines whether an operation on any of the panel image 531, key image 532, or end button 533 currently displayed on the user IF 53 has been accepted (S311). If it is determined that an operation on the user IF 53 has been accepted (S311: YES, C11 in FIG. 12), the browser 62 outputs an effect (S312). Specifically, the browser 62 outputs an effect indicating that the operation has been accepted by briefly displaying a small animation or still image indicating the position of the operation, for example, a mark such as a circle, at the operation location, changing the display color of the operation location, outputting a sound effect, or the like. By outputting an effect in response to the operation, the user who has operated the user IF 53 can easily feel that the operation has been accepted.

[0118] 11, the browser 62 determines YES in S311 when it receives an operation within the range of the panel image 531, within the range showing one of the key images 532, or within the range showing the end button 533. For example, when it receives an operation at a position other than these, the browser 62 may ignore the operation or may execute another process that the browser 62 has.

[0119] Browser 62 determines whether the received instruction is an operation on end button 533, i.e., an instruction to end cloud remote control (S321). If it is determined that the received instruction is not an instruction to end cloud remote control (S321: NO), browser 62 uploads operation information indicating the received operation to structured storage 202 so that the operation information is stored in a record indicating the operation content, which will be described later (S322, C12 in FIG. 12). Browser 62 can upload a record storing the operation information to structured storage 202 using structuring access information 263 (see FIG. 8) included in executable file 26 downloaded in B31 in FIG. 7. S322 and C12 in FIG. 12 are an example of operation upload processing.

[0120] Furthermore, the browser 62 increments the stored operation number (S323). The operation number is also information indicating the operation order. Note that the browser 62 stores the operation number as 1 when the browser image change process starts, i.e., when the remote operation procedure starts.

[0121] The operation information is information indicating the operation content of the operation accepted by PC 5. Browser 62 uploads one of multiple types of records in accordance with the operation, for example, as shown in Fig. 10. For example, like records R31 to R34 and R41 shown in Fig. 10, the first key 2021 of the record storing the operation information stores information indicating that the operation information is for cloud remote operation, and the second key 2022 stores information indicating the operation number. Browser 62 uploads the operation information to structured storage 202 so that the operation information is stored in a record in which the information corresponding to the stored operation number is stored in the second key, for example.

[0122] For example, if the stored operation number is 1 and a click on an icon displayed at position A on panel image 531 is received, browser 62 first uploads operation information corresponding to the pressing operation at the time when the operation of pressing a mouse button or the like among the clicks is received (S322). Specifically, browser 62 uploads the operation information so that record R31, which includes information indicating a remote operation in first key 2021 and operation number 1 in second key, stores the device ID of MFP1 in device ID 2024 and stores information indicating the operation position and information indicating a downward operation (pressing operation) in parameters 2025. Then, browser 62 increments the stored operation number by 1 to make it 2 (S323). The parameters 2025 of each of records R31 to R34 and R41 are an example of operation data.

[0123] Next, when an operation of releasing the finger from a button or the like among the clicked items is received, the browser 62 uploads information corresponding to the releasing operation (S322). Specifically, the browser 62 uploads the operation information so that the device ID of the MFP1 is stored in the device ID 2024 of record R32, which includes information indicating remote operation in the first key 2021 and operation number 2 in the second key, and the same operation position information as record R31 and information indicating an operation in the upward direction (releasing operation) are stored in the parameter 2025. The browser 62 then increments the stored operation number by 1 to make it 3 (S323).

[0124] In this way, records R31 and R32 store information indicating that a click operation and a release operation were performed at the same operation position, as well as timestamps for the click operation and the release operation, respectively. As will be described later, MFP 1 downloads operation information uploaded by PC 5 and performs processing in accordance with the downloaded operation information. In this case, MFP 1 can distinguish, for example, between a short press and a long press of an icon by, for example, comparing timestamps 2023 stored in records R31 and R32. Note that the information indicating the operation position may be information indicating coordinates within panel image 531 or within touch panel 131 of the corresponding MFP 1, or may be information identifying the operated button or icon.

[0125] Furthermore, for example, when an operation on a key image indicating a back key, which is one of key images 532, is accepted as an operation when the stored operation number is 7, browser 62 uploads operation information so that record R41, which includes information indicating a remote operation in first key 2021 and operation number 7 in second key, stores the device ID of MFP1 in device ID 2024 and stores information indicating the back key in parameter 2025 (S322). Then, browser 62 adds 1 to the stored operation number to make it 8 (S323).

[0126] In the case of an operation on a key image, the browser 62 uploads operation information so as to store information indicating the operated key, rather than information indicating the operation direction, in the parameter 2025. The information indicating the key may be information specifying one of the hard keys 132 of the MFP 1 corresponding to the key image 532 that accepted the operation, or may be information indicating the coordinates of the corresponding hard key 132. Note that in order to be able to distinguish between a short press and a long press for operations on the hard keys 132 of the MFP 1, the browser 62 may also upload multiple pieces of operation information including information on the operation direction for the operations on the key image 532.

[0127] After S323, or when it is determined that no operation on the user IF 53 has been accepted (S311: NO), the browser 62 determines whether or not to update the panel image 531 being displayed on the user IF 53 (S341). As described above, the browser 62 periodically accesses the structured storage 202 (C03 in FIG. 12), and determines to update the panel image 531 if the number of changes stored in the image information has increased from the number of changes stored by the browser 62.

[0128] As will be described in detail later, the image being displayed on touch panel 131 of MFP1 may be changed by cloud remote control from PC 5 or by a user operation on MFP1. When the image being displayed on touch panel 131 is changed, MFP1 uploads the image information to structured storage 202 so that the updated image information is stored in record R22 in Fig. 10 or in another record of a pair of image information. The updated image information includes information on the number of changes added by MFP1.

[0129] The number of changes stored by the browser 62 is the number of changes included in the image information used to display the panel image 531 currently being displayed on the browser 62. In other words, if an increased number of changes than the stored number of changes is stored in the image information, the panel image data corresponding to that image information is newer than the panel image data used to display the panel image 531 currently being displayed on the browser 62. At the start of the browser image change process, i.e., at the start of the remote operation procedure, the browser 62 stores the number of changes as 1.

[0130] If it is determined that panel image 531 should be updated (S341: YES), browser 62 acquires panel image data (S342). Specifically, as shown in C21 to C23 in Fig. 12, browser 62 first downloads image information from record R22 or another pair of image information records (C21), and then downloads panel image data from the record in structured storage 202 or from folder 2011 in unstructured storage 201 based on the parameters of the image information (C22 or C23). C21 to C23 in Fig. 12 are the same processes as B33 to B35 in Fig. 7.

[0131] If there are three or more pairs of records storing information related to panel image data, the browser 62 may download the information related to panel image data stored in the record with the largest number of changes stored. Alternatively, the browser 62 may download the information related to panel image data stored in the record with the most recent timestamp stored.

[0132] Furthermore, browser 62 displays updated panel image 531 on user IF 53 based on the downloaded panel image data (S343, C25 in FIG. 12). When the content of the screen being displayed on touch panel 131 of MFP 1 is changed, panel image 531 displayed on user IF 53 of PC 5 is also changed in the same way, so that the screen of MFP 1 and the screen of PC 5 can be synchronized. Note that the image that is subject to change is only the range of panel image 531, and browser 62 does not change the displayed key image 532 or end button 533. Furthermore, browser 62 stores the number of changes included in the image information (S344).

[0133] That is, when an event occurs that changes the content of the screen being displayed on touch panel 131, MFP 1 uploads the panel image data and updates the image information. Then, as the image information is updated, browser 62 downloads the panel image data and updates panel image 531. This makes it possible to synchronize the image displayed on touch panel 131 of MFP 1 with the image displayed on user IF 53 of PC 5.

[0134] If browser 62 fails to display panel image 531 based on the information downloaded in C21 to C23, browser 62 maintains the display of originally displayed panel image 531. At this time, if necessary to maintain the display, browser 62 can simply re-download the information previously downloaded in B33 to B35 or C21 to C23.

[0135] After S344, or if it is determined that the panel image 531 should not be updated (S341: NO), the browser 62 proceeds to S311, and repeats the determination of whether an operation on the user IF 53 has been accepted (S311) and the determination of whether the panel image 531 should be updated (S341) (loop in FIG. 12).

[0136] On the other hand, if it is determined that the received instruction is an instruction to end cloud remote control (S321: YES), that is, if an operation on end button 533 shown in FIG. 11 is received (C31 in FIG. 12), browser 62 uploads end information to structured storage 202 (S331, C32 in FIG. 12). Specifically, for example, as shown in FIG. 10, browser 62 uploads the end information so that information indicating that an instruction to end cloud remote control of MFP1 has been received is stored in parameter 2025 of record R51, which is a record in which first key 2021 stores information indicating operation information for cloud remote control and second key 2022 stores information indicating an end instruction. The end information may be information indicating an operation on end button 533. S331 is an example of operation upload processing. Thereafter, PC 5 ends the browser image change processing and ends cloud remote control of MFP1 (C33, break in FIG. 12).

[0137] After the cloud remote control ends, the MFP 1 does not upload panel image data. The PC 5 may terminate the browser 62, terminate the execution of the process according to the browser P262 and display another web page, or display a blank screen. The downloaded executable file 26 may be deleted when the browser 62 ends, or may be deleted automatically when execution ends.

[0138] Next, the procedure for the device image change process executed by MFP1 when cloud remote control is started will be described with reference to the flowchart in Fig. 14 and, if necessary, the sequence diagram in Fig. 12. As described above, the device image change process is executed by CPU 11 of MFP1 after uploading executable file 26 and image data and uploading response data indicating "success" to structured storage 202 (B11 to B17 in Fig. 7).

[0139] In the device image change process, the CPU 11 determines whether or not it is time for a periodic check (S401). While cloud remote operation is being performed, the MFP 1 periodically accesses the structured storage 202, just like the browser 62 (C04 in FIG. 12). If it is not time for a periodic check (S401: NO), the CPU 11 repeats the determination of S401. Note that the check timing by the MFP 1 and the check timing by the browser 62 do not have to be the same.

[0140] If it is determined that it is time for a periodic check (S401: YES), CPU 11 checks whether or not termination information including its own device ID has been uploaded to structured storage 202 (S402). If it is determined that termination information has not been uploaded (S402: NO), CPU 11 determines whether or not operation information including its own device ID has been uploaded to structured storage 202 (S403). When an operation by the user of PC 5 is accepted, the operation information is uploaded to structured storage 202 in the browser image change process (S322 in FIG. 13, C12 in FIG. 12) according to the procedure described above.

[0141] When it is determined that the operation information has been uploaded to structured storage 202 (S403: YES), CPU 11 executes operation information processing (S411). The procedure of the operation information processing will be described with reference to the flowchart in Fig. 15 and, if necessary, the sequence diagram in Fig. 12.

[0142] In the operation information processing, CPU 11 acquires up to N pieces of operation information uploaded to structured storage 202 by downloading them (S501, C31 in FIG. 12). N is, for example, an integer between 2 and 20, and is set in advance in MFP 1. If the number of pieces of uploaded operation information is less than N, CPU 11 acquires all of the uploaded operation information. As described above, when browser 62 receives multiple operations, it stores each operation as one piece of operation information. For example, when a click on an icon is received, two consecutive pieces of operation information are stored. Because MFP 1 acquires a certain number of pieces of operation information together, the number of times that it accesses cloud storage service 200 does not increase.

[0143] The CPU 11 processes the downloaded operation information in order of oldest to newest operation based on the operation number or timestamp. The CPU 11 first analyzes the first operation information (S511) and determines whether the operation is an instruction to start maintenance mode (S512). If it is determined that the operation is not an instruction to start maintenance mode (S512: NO), the CPU 11 executes processing in accordance with the operation information (S514, C32 in FIG. 12). By processing in order of oldest to newest, the MFP 1 can execute processing in accordance with the procedure operated on the PC 5.

[0144] As described above, MFP 1 can display a screen including one or more icon images on touch panel 131, and when an operation within the area of ​​a displayed icon image is accepted, MFP 1 can execute the processing indicated by that icon image. For example, if the operated icon image is an icon indicating an instruction to execute image processing, MFP 1 executes the instructed image processing. Then, panel image 531, which virtually reproduces touch panel 131, is displayed on user IF 53 of PC 5. If the downloaded operation information is information indicating an operation on panel image 531, CPU 11 executes the processing corresponding to the operation, in the same manner as when an operation on touch panel 131 is accepted.

[0145] For example, if the operation information is information indicating an operation on a copy icon included in panel image 531 which is a virtualized standby screen of MFP 1, CPU 11 acquires information for displaying a setting screen for copy processing. If the operation information is information indicating an operation related to a setting change on panel image 531 which is a virtualized setting screen for copy processing, CPU 11 changes the setting of the copy processing. If the operation information is information indicating an operation on an icon indicating an instruction to execute image formation processing, CPU 11 starts execution of the image formation processing as processing corresponding to that operation. For example, if the operation information is information indicating an operation on an icon for instructing execution of copy processing on panel image 531 which is a virtualized setting screen for copy processing, CPU 11 starts execution of the copy processing.

[0146] Furthermore, if the operation information is information indicating a key operation, the CPU 11 executes a process corresponding to the operation, similar to when an operation on the hard keys 132 is accepted. Then, the CPU 11 changes the panel image in accordance with the execution of the process and displays it on the touch panel 131 (S515).

[0147] For example, when information for displaying a copy processing setting screen is acquired, the CPU 11 displays a setting screen including an icon for instructing a setting change, an input field for setting values ​​to be changed, and an icon for instructing execution of the copy processing. After the copy processing setting is changed, the CPU 11 displays an updated copy processing setting screen showing the changed settings. After changing the copy processing setting, the CPU 11 may display a standby screen. After starting execution of the image formation processing, the CPU 11 displays an execution screen for the image formation processing, for example, an execution screen for the copy processing.

[0148] On the other hand, if it is determined that the instruction is to start the maintenance mode (S512: YES), the CPU 11 starts the maintenance mode (S516). The maintenance mode is, for example, a mode performed by a maintenance person for maintenance of the MFP 1, and is a mode in which, for example, an instruction to print a report or an instruction to check the operating status can be accepted. The MFP 1 can accept the instruction to start the maintenance mode by, for example, pressing a specific combination of specific keys.

[0149] When the MFP1 receives an instruction to start the maintenance mode, it displays on the touch panel 131 a dedicated maintenance mode screen, which is a screen that receives instructions that can be executed in the maintenance mode (S517). The maintenance mode is an example of a special mode, and an example of a mode for maintenance. The maintenance mode screen is an example of a main body operation screen that corresponds to the special mode. Then, the CPU 11 calculates a checksum of the displayed maintenance mode screen and stores it in the memory 12 (S518). The checksum is an example of a check value. S518 is an example of a calculation process.

[0150] During maintenance mode, the MFP 1 displays a maintenance mode screen and can only accept maintenance mode execution instructions that specify the type of maintenance mode by a number, for example. During maintenance mode, the MFP 1 does not accept print instructions from an external device, for example. The screen data of the maintenance mode screen is, for example, bitmap data, and no screen ID is set. Therefore, the CPU 11 uses a checksum to determine whether the screen has been changed.

[0151] A screen ID may also be set for the maintenance mode screen. In that case, the MFP 1 may store that screen ID in S518 instead of calculating a checksum. Furthermore, there may be screens other than the maintenance mode screen for which no screen ID is set. In that case, the MFP 1 may calculate and store a checksum even when a screen for which no screen ID is set is displayed.

[0152] After S515 or S518, the CPU 11 deletes the processed operation information from the structured storage 202 (S521). Then, the CPU 11 determines whether or not the processing of the N pieces of operation information acquired in S511 has been completed (S522). If it is determined that the processing has not been completed (S522: NO), the CPU 11 proceeds to S511 and processes the next piece of operation information. If it is determined that the processing of all N pieces of operation information has been completed (S522: YES), the CPU 11 ends the operation information processing and returns to the device image change processing of Fig. 14. Note that instead of deleting the pieces one by one in S521, the CPU 11 may delete all N pieces of operation information at once after the processing of N pieces of operation information has been completed (YES in S522).

[0153] Returning to the description of the device image change process in Fig. 14, after the operation information process in S411, the CPU 11 executes image storage process (S431). The image storage process is the process shown in Fig. 9, and is a process of uploading panel image data showing the current panel image to the cloud storage service 200.

[0154] As a result, as shown in C33 to C35 in Fig. 12, panel image data is uploaded to structured storage 202 or unstructured storage 201, and image information is uploaded to structured storage 202. C33 to C35 in Fig. 12 are the same processes as B13 to B15 in Fig. 7. Note that the information uploaded by C33 to C35 is downloaded by browser 62 at C21 to C23 and processed by browser 62, as described above.

[0155] For example, when the MFP 1 executes processing based on the acquired operation information, it may change the screen being displayed on the touch panel 131. During the operation information processing of S411, for example, when the MFP 1 updates the panel image displayed on the touch panel 131 (S515) or starts maintenance mode and displays the maintenance mode screen (S517), the MFP 1 executes image storage processing to upload the panel image data and image information.

[0156] As a result, browser 62 uploads operation information indicating an operation on panel image 531, and then downloads new panel image data in response to MFP 1 updating the image information after executing processing in accordance with the operation information, and updates and displays panel image 531. As a result, when the screen displayed on touch panel 131 of MFP 1 is changed, an image showing the screen after the change is displayed on user IF 53 of PC 5. Therefore, the screen of MFP 1 and the screen of PC 5 can be synchronized. After the image storage processing of S431, CPU 11 returns to S401 and determines whether the next confirmation timing has arrived.

[0157] On the other hand, if it is determined that operation information has not been stored (S403: NO), the CPU 11 determines whether or not the system is in maintenance mode (S421). If it is determined that the system is not in maintenance mode (S421: NO), the CPU 11 executes image storage processing (S431). The MFP 1 may change the screen being displayed on the touch panel 131 in cases other than when it executes processing based on acquired operation information, such as when it receives an operation on the user IF 13. If the MFP 1 is performing cloud remote control and is not in maintenance mode, the MFP 1 executes image storage processing and therefore uploads panel image data and image information.

[0158] In other words, since MFP1 repeatedly uploads panel image data representing the screen currently displayed on touch panel 131, if the currently displayed screen is changed, the changed panel image data and image information are uploaded in the image storage process immediately thereafter. As a result, even if the screen currently displayed on touch panel 131 is changed not triggered by an operation on panel image 531, but triggered by an operation on user IF 13 of MFP1, browser 62 can download new panel image data and update and display panel image 531 in response to MFP1 updating the image information. Therefore, the screen of MFP1 and the screen of PC 5 can be synchronized. After the image storage process of S431, CPU 11 returns to S401 and determines whether the next check timing has arrived.

[0159] On the other hand, if it is determined that the MFP 1 is in maintenance mode (S421: YES), the CPU 11 acquires the checksum of the currently displayed panel image (S422). When the maintenance mode is started in response to an instruction to start the maintenance mode (S516 in FIG. 15), the MFP 1 is in maintenance mode. The CPU 11 compares the acquired checksum with the stored checksum to determine whether the checksum has been changed (S423). For example, the CPU 11 compares the checksum with the checksum stored in S518 of the operation information processing (FIG. 15) when the maintenance mode was started to determine whether the checksum has been changed. If it is determined that the checksum has been changed (S423: YES), the CPU 11 stores the checksum of the current panel image (S424) and executes image storage processing (S431). In the image storage processing, the changed panel image data and image information are uploaded. Furthermore, in the next S423, the CPU 11 compares the checksum stored in S424 with the checksum of the currently displayed panel image. After the image storage process in S431, the CPU 11 returns to S401 and determines whether the next check timing has arrived.

[0160] If the CPU 11 determines that the checksum has not been changed during the maintenance mode (S423: NO), the CPU 11 returns to S401 and determines whether the next check timing has arrived. If the CPU 11 determines that the checksum has not been changed during the maintenance mode, the CPU 11 does not execute the image storage process.

[0161] In this way, the MFP 1 determines whether the displayed screen has changed during maintenance mode, and if not, does not upload panel image data, thereby reducing communication traffic. The MFP 1 may also determine whether the displayed screen has changed even when not in maintenance mode, and if not, not upload panel image data. The MFP 1 may determine whether the displayed screen has changed based on the screen ID of the displayed screen, for example. If panel image data is not uploaded when not changed, the communication load is small.

[0162] Furthermore, if it is determined that the termination information has been uploaded (S402: YES), the CPU 11 executes an operation termination process (S441). Note that if the termination information has been uploaded, the CPU 11 does not download the operation information even if the operation information that has not been downloaded has been uploaded. The procedure of the operation termination process will be described with reference to the sequence diagram of FIG.

[0163] The MFP 1 downloads the termination information from the structured storage 202 (D01). As described above, the termination information is information stored in the structured storage 202 by the browser 62 in C32 of Fig. 12. The termination information includes, for example, information indicating that a termination instruction has been received in C31 of Fig. 12, as shown in record R51 of Fig. 10.

[0164] Having downloaded the termination information, the MFP 1 deletes the various pieces of information that it uploaded to structured storage 202 and unstructured storage 201. Specifically, the MFP 1 instructs structured storage 202 to delete the stored termination information for the current cloud remote control (D11). The MFP 1 instructs structured storage 202 to delete the stored operation information for the current cloud remote control (D12). Having accepted the termination instruction, the MFP 1 deletes all unprocessed operation information, even if it remains.

[0165] Furthermore, MFP1 instructs structured storage 202 to delete the uploaded panel image data and image information (D13, D14). If there is panel image data uploaded to unstructured storage 201, MFP1 instructs that the panel image data be deleted (D21). By receiving the termination instruction at PC5, the various data uploaded to structured storage 202 and unstructured storage 201 are deleted, thereby reducing the memory load of structured storage 202 and unstructured storage 201.

[0166] Furthermore, the MFP 1 instructs the deletion of the executable file 26 uploaded to the unstructured storage 201 at the start of the cloud remote operation (D22). Furthermore, the MFP 1 instructs the deletion of the folder created in A32 of Fig. 3 (D23). Note that the order in which the data is deleted is not limited to the above order, and any order may be used.

[0167] When all data has been deleted, the MFP 1 displays on the touch panel 131 a message indicating that the cloud remote operation has ended (D31) and returns to normal operation. The MFP 1 then displays, for example, a standby screen on the touch panel 131, and becomes ready to accept user operations.

[0168] As described above in detail, according to remote operation system 100, when PC 5 starts cloud remote operation of MFP 1, initiation application 63 installed on PC 5 receives an instruction to start cloud remote operation and transmits instruction data to MFP 1. Upon receiving the instruction data, MFP 1 uploads browser P262 to unstructured storage 201. Once browser P262 has been uploaded to unstructured storage 201, initiation application 63 starts browser 62 and passes to browser 62 a URL required to download browser P262 from unstructured storage 201. By using the passed URL, browser 62 can download browser P262 and start cloud remote operation via cloud storage service 200 using browser P262.

[0169] Furthermore, according to remote control system 100, browser 62 installed in PC 5 can acquire browser P262, and panel image data showing panel image 531 including multiple icons is passed from MFP1 to browser P262 via structured storage 202 or unstructured storage 201. This enables PC 5 to accept multiple types of operations via panel image 531, and operation information specifying the icon for which the operation was accepted is passed from browser P262 to MFP1 via structured storage 202. Therefore, MFP1 can execute processing based on the icon, and can realize cloud remote control corresponding to multiple operations via cloud storage service 200.

[0170] Furthermore, according to remote control system 100, browser 62 installed on PC 5 can acquire browser P262, and panel image data for virtually reproducing the screen currently displayed on touch panel 131 of MFP1 is passed from MFP1 to browser P262 via structured storage 202 or unstructured storage 201. This makes it possible to display panel image 531 on PC 5 that can accept operations equivalent to those on the screen currently displayed on touch panel 131 of MFP1, enabling cloud remote control to be achieved with the same ease of use as on the MFP1 itself.

[0171] Furthermore, according to remote control system 100, when MFP 1 receives a start instruction from PC 5 and starts cloud remote control on PC 5, it uploads browser P262 and CSS 261 as a single executable file 26 to unstructured storage 201 in a state where anonymous access is prohibited. After uploading executable file 26, browser 62 of PC 5 accesses unstructured storage 201 via non-anonymous access and downloads the file. As a result, even when executable file 26 is uploaded to unstructured storage 201 in a state where anonymous access is prohibited, browser 62 can display an appropriate panel image 531 in accordance with browser P262.

[0172] Note that communication between different networks often involves security issues and communication restrictions. Therefore, the PC 5 may not be able to directly transfer data necessary for remote operation to an MFP 1 that is not on the same network. By using a cloud storage service 200 on the Internet to transfer data between the PC 5 and the MFP 1, as in the present invention, communication restrictions are reduced, increasing the possibility of remote operation of an MFP 1 that is not on the same network.

[0173] Note that this embodiment is merely illustrative and does not limit the present invention in any way. Therefore, the technology disclosed in this specification can naturally be improved and modified in various ways without departing from the spirit and scope of the present invention. For example, the device to be remotely controlled by cloud computing may be any device that has a communication interface, can connect to the Internet, and has an image processing function. This device is not limited to an MFP, but may also be a single-function printer, a copier, a scanner, a fax machine, or a machine tool capable of image-based processing. Furthermore, the device performing cloud remote control is not limited to a PC, but may also be a mobile terminal such as a smartphone or a tablet computer. Furthermore, the remote control system 100 may include at least one device performing cloud remote control and at least one device to be remotely controlled by cloud computing, and the number and arrangement of these devices are not limited to the illustrated example.

[0174] The configuration of user IF 13 of MFP 1 is not limited to that exemplified in the embodiment. For example, MFP 1 may have a screen with only a display function instead of touch panel 131, and may have a configuration that accepts operation input by cursor operation or the like. Furthermore, the number of hard keys 132 is not limited to three, and may be absent.

[0175] In the embodiment, the PC 5 uploads a record including the instruction data to the structured storage 202, and the MFP 1 downloads the instruction data to acquire the instruction data, but this is not limiting. For example, the PC 5 may send the instruction data directly to the MFP 1. The MFP 1 may also send the response data directly to the PC 5.

[0176] In the embodiment, MFP 1 uploads response data including self-folder access information 243 to structured storage 202, thereby passing the URL to PC 5, but this is not limiting. For example, start application 63 of PC 5 may previously store a URL indicating a folder used by the MFP that is the target of cloud remote control.

[0177] Furthermore, in the embodiment, MFP1 includes information indicating structured storage 202 or unstructured storage 201 as information indicating the storage location of panel image data in the image information, but this is not limited to this. For example, the information indicating the storage location may be information that is included only when panel image data is uploaded to structured storage 202, or may be information that is included only when panel image data is uploaded to unstructured storage 201.

[0178] In the embodiment, the trigger for terminating cloud remote control has been described as receiving an operation on the end button 533 of the PC 5 (see FIG. 11 ), but this is not limiting. For example, if the PC 5 is executing cloud remote control and no operation on the browser 62 is received for a certain period of time, the PC 5 may upload end information in the same way as when a termination instruction is received, and terminate the cloud remote control. Alternatively, for example, the cloud remote control may be terminated when a specific operation is received on the MFP 1, such as a long press of the cancel button. The MFP 1 may also determine that a termination instruction has been received when no new operation information is uploaded for a certain period of time. In this case, the MFP 1 may upload information indicating the termination instruction to the structured storage 202.

[0179] Furthermore, in the embodiment, an example has been described in which one token is acquired for accessing each of the unstructured storage 201 and the structured storage 202, but multiple tokens may be acquired for one area. In that case, for example, multiple tokens may be stored as the access information 24 of the MFP 1, and one of them may be used by the MFP 1 itself while the other is passed to the PC 5. Also, for example, different tokens may be used for uploading and downloading. Also, different tokens may be used for each MFP.

[0180] Furthermore, in any flowchart or sequence diagram disclosed in the embodiments, the execution order of multiple processes in any multiple steps can be changed or executed in parallel as desired, as long as no contradictions occur in the processing content.

[0181] The processes disclosed in the embodiments may be executed by hardware such as a single CPU, multiple CPUs, or ASIC, or a combination thereof. The processes disclosed in the embodiments may be realized in various ways, such as a recording medium on which a program for executing the processes is recorded, or a method. [Explanation of symbols]

[0182] 1, 2, 3 MFP 5 PC 13 User Interface 53 User Interface 100 Remote Control System 200 cloud storage services

Claims

1. A system including an image forming apparatus and an information processing apparatus, wherein the information processing apparatus and the image forming apparatus are connected to the Internet; the image forming apparatus is provided with a display and is capable of displaying a main body operation screen on the display; the information processing device is capable of installing a browser; the browser is capable of causing the information processing device to acquire a remote operation program operable on the browser, the image forming apparatus is capable of executing an image upload process of uploading image data for virtually reproducing the main body operation screen displayed on the display to a cloud storage connected to the Internet; the remote operation program causes the information processing device to download the image data from the cloud storage, and execute a screen display process to display a remote operation screen, which is a virtual reproduction of the main body operation screen based on the image data, on a user interface of the information processing device using the browser; the remote operation program further causes the information processing device to execute an operation upload process of uploading operation data indicating the operation content to the cloud storage when the remote operation screen is accepted via the user interface; When the operation data is uploaded to the cloud storage, the image forming device downloads the operation data from the cloud storage and executes a process corresponding to the operation indicated in the operation data, and if the operation indicated in the operation data is an operation related to image formation, executes an image formation process as a process corresponding to the operation; the image forming device further executes the image upload process when an event occurs that changes the content of the main body operation screen, thereby uploading the image data for virtually reproducing the changed main body operation screen to the cloud storage; When the image data for virtually reproducing the changed main body operation screen is uploaded to the cloud storage, the remote operation program causes the information processing device to execute the screen display process, and in the screen display process, the image data for virtually reproducing the changed main body operation screen is downloaded from the cloud storage, and the remote operation screen, which is a virtual reproduction of the changed main body operation screen based on the downloaded image data, is displayed on the user interface using the browser. A system configured as follows:

2. 2. The system of claim 1, the image forming device is capable of changing content of the main body operation screen in association with execution of a process corresponding to an operation indicated in the operation data uploaded to the cloud storage, and when the content of the main body operation screen is changed in association with execution of the process corresponding to the operation indicated in the operation data, the image forming device executes the image upload process to upload the image data for virtually reproducing the main body operation screen after the change in association with the process corresponding to the operation indicated in the operation data to the cloud storage; the remote operation program causes the information processing device to execute the screen display process when the image data for virtually reproducing the main body operation screen after the change associated with the processing corresponding to the operation indicated in the operation data is uploaded to the cloud storage, and in the screen display process, the image data for virtually reproducing the main body operation screen after the change associated with the processing corresponding to the operation indicated in the operation data is downloaded from the cloud storage, and the remote operation screen that virtually reproduces the main body operation screen after the change associated with the processing corresponding to the operation indicated in the operation data is displayed on the user interface using the browser. A system configured as follows:

3. 2. The system of claim 1, the image forming apparatus includes a touch panel, displays a main body panel screen including a plurality of operators on the touch panel, and, upon receiving an operation on one of the operators, executes a process based on the operator whose operation has been received; the remote operation program is capable of causing the information processing device to execute the screen display process, and in the screen display process, the remote operation screen, which virtually reproduces the main body panel screen based on the image data downloaded from the cloud storage, is displayed on the user interface using the browser; the remote operation program further causes the information processing device to execute the operation upload process when an operation on one of the plurality of operators displayed on the remote operation screen is accepted via the user interface, and in the operation upload process, uploads information identifying the operator whose operation has been accepted to the cloud storage as the operation data; When the operation data is uploaded to the cloud storage, the image forming device downloads the operation data from the cloud storage and executes a process based on an operator identified by the operation data. A system configured as follows:

4. 4. A system according to claim 3, the image forming apparatus includes an operation panel including the touch panel, the operation panel also including a plurality of hardware keys, and when an operation on one of the plurality of hardware keys is accepted, a process based on the hardware key that accepted the operation is executed; the remote operation program is capable of causing the information processing device to execute the screen display process, and in the screen display process, the remote operation screen that virtually reproduces the operation panel based on the image data downloaded from the cloud storage is displayed on the user interface using the browser; the remote operation program further causes the information processing device to execute the operation upload process when an operation on one of the plurality of hardware keys displayed on the remote operation screen is accepted via the user interface, and in the operation upload process, information identifying the hardware key for which the operation is accepted is uploaded to the cloud storage as the operation data; When the operation data is uploaded to the cloud storage, the image forming device downloads the operation data from the cloud storage and executes a process based on the hardware key identified by the operation data. A system configured as follows:

5. 2. The system of claim 1, the image forming apparatus repeatedly executes the image upload process, and uploads the image data for virtually reproducing the changed main body operation screen to the cloud storage by the image upload process executed immediately after an event that changes the content of the main body operation screen occurs. A system configured as follows:

6. 6. A system according to claim 5, When the operation data is uploaded to the cloud storage, the image forming apparatus downloads the operation data from the cloud storage, executes a process corresponding to the operation indicated in the operation data, and executes the image upload process regardless of whether or not the content of the main body operation screen is changed by the execution of the process corresponding to the operation indicated in the operation data. A system configured as follows:

7. 6. A system according to claim 5, the image forming apparatus is capable of displaying the main body operation screen corresponding to a special mode on the display, While the main body operation screen corresponding to the special mode is displayed on the display, the image forming device does not repeat the image upload process, but repeatedly executes a calculation process for calculating a check value using image data showing the main body operation screen corresponding to the displayed special mode, and if the check value calculated in the previous calculation process is different from the check value calculated in the current calculation process, executes the image upload process to upload the image data for virtually reproducing the main body operation screen corresponding to the check value calculated in the current calculation process to the cloud storage. A system configured as follows:

8. 8. A system according to claim 7, the special mode is a mode for maintenance of the image forming apparatus, When the operation indicated in the operation data downloaded from the cloud storage is an operation indicating the start of maintenance, the image forming apparatus displays the main body operation screen corresponding to the special mode on the display. A system configured as follows:

9. 2. The system of claim 1, an identifier is provided for each of the main body operation screens, When an event occurs that changes the identifier of the main body operation screen displayed on the display, the image forming device executes the image upload process to upload the image data for virtually reproducing the main body operation screen corresponding to the changed identifier to the cloud storage; When the image data for virtually reproducing the main body operation screen corresponding to the changed identifier is uploaded to the cloud storage, the remote operation program causes the information processing device to execute the screen display process, and in the screen display process, the image data for virtually reproducing the main body operation screen corresponding to the changed identifier is downloaded from the cloud storage, and the remote operation screen, which is a virtual reproduction of the main body operation screen corresponding to the changed identifier based on the downloaded image data, is displayed on the user interface using the browser. A system configured as follows:

10. 2. The system of claim 1, the image forming apparatus is capable of displaying the main body operation screen including a video on the display, the image forming device further uploads, to the cloud storage, image data without video for virtually reproducing a main body operation screen without video, which is the main body operation screen excluding the video, instead of the image data for virtually reproducing the main body operation screen while the main body operation screen including the video is being displayed on the display, by the image upload process; the remote operation program executes the screen display process even when the image data without video is downloaded from the cloud storage, and in the screen display process, the remote operation screen, which is a virtual reproduction of the main body operation screen without video based on the downloaded image data without video, is displayed on the user interface by the browser. A system configured as follows:

11. An image forming apparatus connectable to the Internet, the image forming apparatus is provided with a display and is capable of displaying a main body operation screen on the display; the image forming apparatus is capable of executing an image upload process for uploading image data for virtually reproducing the main body operation screen to be displayed on the display to a cloud storage connected to the Internet, the image data being downloaded to an information processing apparatus connected to the Internet by a remote operation program operable in a browser, and a remote operation screen that virtually reproduces the main body operation screen based on the image data being displayed on a user interface of the information processing apparatus by the browser; When operation data is uploaded to the cloud storage, the image forming device downloads the operation data from the cloud storage and executes a process corresponding to the operation indicated in the operation data, and if the operation indicated in the operation data is an operation related to image formation, executes an image formation process as a process corresponding to the operation, the operation data being data indicating the content of an operation on the remote operation screen, and when the information processing device accepts an operation on the remote operation screen, the operation data is uploaded to the cloud storage by the remote operation program, When an event occurs that changes the content of the remote operation screen displayed on the information processing device, the image forming device executes the image upload process to upload the image data for virtually reproducing the changed main body operation screen to the cloud storage, and when the image data showing the changed remote operation screen is uploaded to the cloud storage, the image data for virtually reproducing the changed main body operation screen is downloaded from the cloud storage to the information processing device by the remote operation program, and the remote operation screen that virtually reproduces the changed main body operation screen based on the downloaded image data is displayed on the user interface by the browser. The image forming apparatus is configured as follows.

12. 12. The image forming apparatus according to claim 11, the image forming device is capable of changing the content of the main body operation screen in conjunction with the execution of a process corresponding to the operation indicated in the operation data uploaded to the cloud storage, and when the content of the main body operation screen is changed in conjunction with the execution of the process corresponding to the operation indicated in the operation data, the image forming device executes the image upload process to upload the image data for virtually reproducing the main body operation screen after the change in conjunction with the process corresponding to the operation indicated in the operation data to the cloud storage, the image data is downloaded to the information processing device by the remote operation program, and the remote operation screen, which virtually reproduces the main body operation screen after the change in conjunction with the process corresponding to the operation indicated in the operation data based on the image data downloaded from the cloud storage, is displayed on the user interface by the browser. The image forming apparatus is configured as follows.

13. 12. The image forming apparatus according to claim 11, the image forming apparatus is provided with a touch panel, and displays a main body panel screen including a plurality of operators on the touch panel; when an operation on one of the plurality of operators is received, a process based on the operator whose operation is received is executed; in the image upload process, image data for virtually reproducing the main body panel screen displayed on the display is uploaded to the cloud storage as the image data representing the remote operation screen; the image data is downloaded to the information processing apparatus by the remote operation program; the remote operation screen, which virtually reproduces the main body panel screen based on the image data downloaded from the cloud storage, is displayed on the user interface by the browser; when an operation on one of the plurality of operators displayed on the remote operation screen is received, the operation data is uploaded to the cloud storage by the remote operation program; the uploaded operation data includes information identifying the operator whose operation is received among the plurality of operators displayed on the remote operation screen; When the operation data is uploaded to the cloud storage, the image forming device downloads the operation data from the cloud storage and executes a process based on the operator identified by the operation data. The image forming apparatus is configured as follows.

14. 12. The image forming apparatus according to claim 11, the image forming apparatus repeatedly executes the image upload process, and the image data for virtually reproducing the changed main body operation screen is uploaded to the cloud storage by the image upload process executed immediately after an event that changes the content of the main body operation screen occurs. The image forming apparatus is configured as follows.

Images