Image formation apparatus, terminal apparatus, control method of image formation apparatus, and control method of terminal apparatus

Abstract

To provide an image formation apparatus and the like which can securely resume communication with a terminal apparatus.SOLUTION: An image formation apparatus comprises: a control unit; a storage unit; and a communication unit. The control unit establishes connection with a terminal apparatus via the communication unit with QUIC, stores identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted to / received from the terminal apparatus in the storage unit, performs communication of the transmission / reception information with the terminal apparatus with HTTP / 3, establishes connection with the terminal apparatus by using the identification information and the key information when the connection with the terminal apparatus is cut, and resumes the communication of the transmission / reception information with the terminal apparatus.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image forming apparatus and the like. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a remote operation technique has been used to operate a so-called multifunction peripheral, which has functions such as a copy function, a scan function, and a document print function, from a terminal device such as a mobile terminal.

[0003] Furthermore, techniques for improving usability during remote operation have also been proposed. For example, an image processing device has been proposed that, when wireless communication with a mobile terminal is interrupted while receiving data to be processed transferred from the mobile terminal, stores interrupted job information including the mobile terminal's unique information (telephone number), and resumes wireless communication with the mobile terminal if the telephone number of the mobile terminal connected after the interruption matches the telephone number included in the interrupted job information (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-003175 Summary of the Invention [Problem to be solved by the invention]

[0005] To resume communication between a multifunction peripheral and a terminal device using the technology described in Patent Document 1, it is necessary to transmit mobile phone-specific information such as a telephone number and printing information (storage location of print data and print settings) from the terminal device to the multifunction peripheral and store them on the multifunction peripheral side. This poses a risk that personal information such as a telephone number and printing information may be leaked due to a malicious attack (e.g., interception of communications). In addition, mobile phone-specific information such as a telephone number may be known by a third party, raising concerns that print data may be leaked due to spoofing.

[0006] In view of the above-described problems, the present disclosure has an object to provide an image forming apparatus or the like that is capable of resuming communication with a terminal device securely. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the image forming apparatus of the present disclosure is equipped with a control unit, a memory unit, and a communication unit, and the control unit establishes a connection with a terminal device via the communication unit using QUIC, stores identification information that identifies the connection and key information used to encrypt transmission and reception information sent and received from the terminal device in the memory unit, communicates the transmission and reception information with the terminal device using HTTP / 3, and when the connection with the terminal device is disconnected, establishes a connection with the terminal device using the identification information and the key information and resumes communication of the transmission and reception information with the terminal device.

[0008] The terminal device of the present disclosure comprises a control unit, a memory unit, and a communication unit, and the control unit establishes a connection with an image forming device via the communication unit using QUIC, stores identification information that identifies the connection and key information used to encrypt transmission and reception information sent and received from the image forming device in the memory unit, communicates the transmission and reception information with the image forming device using HTTP / 3, and when the connection with the image forming device is disconnected, reconnects to the image forming device using the identification information and the key information and resumes communication of the transmission and reception information with the image forming device.

[0009] The control method for an image forming device according to the present disclosure is characterized by including the steps of: establishing a connection with a terminal device using QUIC; storing identification information for identifying the connection and key information used to encrypt transmission and reception information transmitted and received from the terminal device; communicating the transmission and reception information with the terminal device using HTTP / 3; and, when the connection with the terminal device is disconnected, establishing a connection with the terminal device using the identification information and the key information and resuming communication of the transmission and reception information with the terminal device.

[0010] The control method for a terminal device according to the present disclosure is characterized by including the steps of: establishing a connection with an image forming device using QUIC; storing identification information for identifying the connection and key information used to encrypt transmission and reception information sent and received from the image forming device; communicating the transmission and reception information with the image forming device using HTTP / 3; and, if the connection with the image forming device is disconnected, reconnecting to the image forming device using the identification information and the key information and resuming communication of the transmission and reception information with the image forming device. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide an image forming apparatus or the like that is capable of resuming communication with a terminal device securely. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining the overall configuration of a system in a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of a terminal device according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a data structure of multifunction device information in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating the functional configuration of the multifunction peripheral according to the first embodiment. [Figure 5] FIG. 4 is a diagram illustrating an example of a data structure of job detail information in the first embodiment. [Figure 6] FIG. 3 is a sequence diagram showing a processing flow in the first embodiment. [Figure 7] FIG. 4 is a flowchart showing the flow of processing of the terminal device in the first embodiment. [Figure 8] FIG. 3 is a flowchart showing the flow of processing in the multifunction peripheral according to the first embodiment. [Figure 9] FIG. 4 is a flowchart showing the flow of reconnection handling processing in the first embodiment. [Figure 10] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 12] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 13] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 14] FIG. 4 is a diagram illustrating an example of operation in the first embodiment. [Figure 15] FIG. 10 is a flowchart showing the flow of processing of a terminal device in the second embodiment. [Figure 16] FIG. 10 is a flowchart showing the flow of reconnection handling processing in the second embodiment. [Figure 17] FIG. 10 is a flowchart showing the flow of power-on processing in the second embodiment. [Figure 18] FIG. 11 is a sequence diagram showing a processing flow in the third embodiment. [Figure 19] FIG. 11 is a sequence diagram showing a processing flow in the third embodiment. [Figure 20] FIG. 11 is a sequence diagram showing a processing flow in the third embodiment. [Figure 21] FIG. 11 is a flowchart showing the flow of processing of a terminal device in the third embodiment. [Figure 22] FIG. 11 is a flowchart showing the flow of processing in a multifunction peripheral according to a third embodiment. [Figure 23] FIG. 11 is a flowchart showing the flow of reconnection handling processing in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment for carrying out the present disclosure will be described with reference to the drawings. Note that the following embodiment is an example for explaining the present disclosure, and the technical scope of the invention described in the claims is not limited to the following description.

[0014] [1. First embodiment] [1.1 Overall structure] The overall configuration of a system 1 in this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system 1 includes a terminal device 10 and a multifunction device 20. The terminal device 10 and the multifunction device 20 can be connected via a wireless LAN (Local Area Network), for example. The terminal device 10 and the multifunction device 20 may be connected directly or via a wireless access point.

[0015] The terminal device 10 is an information processing device carried by a user. The terminal device 10 is, for example, a smartphone, a tablet, or a PC (Personal Computer).

[0016] The multifunction device 20 is an information processing device having functions such as a copy function, a scan function, and a document printing function, and is also called an MFP (Multi-Function Printer / Peripheral). The multifunction device 20 may be an image processing device that processes images, or may be an image forming device that includes means for forming images on recording paper using means such as an electrophotographic method.

[0017] In this embodiment, when the terminal device 10 performs an operation on the multifunction peripheral 20 or executes a job (print job or transmission job), a connection is established using QUIC (Quick UDP Internet Connections) as the communication protocol (transport protocol) used for connection between the terminal device 10 and the multifunction peripheral 20. QUIC is a protocol that establishes a bidirectional connection between two devices using UDP (User Datagram Protocol). Two devices that have established a QUIC connection send a UDP packet that includes a QUIC packet in the UDP payload. The QUIC packet is composed of a QUIC packet header and a payload that includes predetermined information.

[0018] Although UDP is a connectionless protocol, the QUIC packet header contains identification information (e.g., a Connection ID) used to identify the connection, so two devices that have established a connection using QUIC can identify the connection based on that identification information. By identifying the connection based on the identification information, the two devices that have established the connection can maintain the connection between them even if the IP (Internet Protocol) addresses or port numbers of the devices that have established the connection change.

[0019] The payload in a QUIC packet is encrypted. A common key shared by the devices that establish a QUIC connection is used as key information to encrypt and decrypt the encrypted payload. The common key is generated by the two devices that establish the connection when the process to establish the QUIC connection is executed.

[0020] Furthermore, information (transmission / reception information) relating to remote operations and job execution is transferred (transmitted and received) between the terminal device 10 and the multifunction peripheral 20. For example, HTTP / 3, which is a type of HTTP (Hypertext Transfer Protocol), is used to transfer the transmission / reception information.

[0021] The transmission and reception information includes, for example, operation information and job information transmitted from the terminal device 10 to the multifunction device 20, and operation screen information (UI data) transmitted from the multifunction device 20 to the terminal device 10. The operation information is information indicating the content of an operation input by a user. The job information is information indicating the content of a job. The job information may include data (processing data) to be processed by the multifunction device 20 in a predetermined manner, such as printing or transmission, when the job is executed. The terminal device 10 may transmit only job information to the multifunction device 20, or may transmit both operation information and processing data. The operation screen information is information used to display a screen for accepting operations by the user. The transmission and reception information may be any information as long as it causes the multifunction device 20 to execute a predetermined job.

[0022] The terminal device 10 operates the multifunction device 20 by transmitting transmission / reception information such as operation information and job information to the multifunction device 20. Note that the operation performed by the terminal device 10 through communication with the multifunction device 20 is also referred to as remote operation.

[0023] [1.2 Functional Configuration] 1.2.1 Terminal Devices The functional configuration of the terminal device 10 will be described with reference to Fig. 2. As shown in Fig. 2, the terminal device 10 is configured to include a control unit 100, a display unit 140, an operation unit 150, a storage unit 160, and a communication unit 190.

[0024] The control unit 100 controls the entire terminal device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 160, and is configured with one or more arithmetic units (e.g., CPUs (Central Processing Units)). The control unit 100 may also be configured as an SoC (System on a Chip) having multiple functions among those described below.

[0025] The display unit 140 displays various types of information. The display unit 140 is configured by a display device such as an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or a micro LED (Light Emitting Diode) display.

[0026] The operation unit 150 accepts operations by a user of the terminal device 10. The operation unit 150 is configured with an input device such as a touch sensor. The touch sensor may detect input using any common detection method, such as a resistive film method, an infrared method, an electromagnetic induction method, or a capacitance method. The terminal device 10 may be equipped with a touch panel in which the display unit 140 and the operation unit 150 are integrally formed.

[0027] The storage unit 160 stores various programs and various data necessary for the operation of the terminal device 10. The storage unit 160 is configured by a storage device such as a semiconductor memory, such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0028] The storage unit 160 stores an operation application 162 and reserves a multifunction device information storage area 164 as a storage area.

[0029] The operation application 162 is an application for causing the control unit 100 to realize a function for remotely operating the multifunction peripheral 20. For example, the operation application 162 causes the control unit 100 to realize a function for displaying a screen used for remote operation on the display unit 140, and a function for connecting to the multifunction peripheral 20 and communicating information.

[0030] The multifunction device information storage area 164 stores information (multifunction device information) related to the multifunction device 20. For example, as shown in Fig. 3, the multifunction device information includes the multifunction device name of the multifunction device 20 (for example, "xxxx"), the address of the multifunction device 20 (for example, "192.168.0.11"), a terminal device Connection ID (for example, "9dc9001122334455") and a multifunction device Connection ID (for example, "8ba9001122334455") that identify the connection with the multifunction device 20, and key information (for example, "******") used to encrypt information sent to and received from the multifunction device 20.

[0031] The address is information used to identify the multifunction peripheral 20 connected to the network to which the terminal device 10 is connected, and is used to establish a connection with the multifunction peripheral 20. The address is, for example, an IP address.

[0032] The Connection ID is information (identification information) used to identify the connection between the terminal device 10 and the multifunction device 20. In this embodiment, a terminal device Connection ID and a multifunction device Connection ID are stored as the Connection ID. The terminal device Connection ID is used when the multifunction device 20 communicates with the terminal device 10. The multifunction device Connection ID is used when the terminal device 10 communicates with the multifunction device 20.

[0033] The key information is, for example, information about a common key (TLS key information) used when performing communication according to the TLS (Transport Layer Security) protocol.

[0034] The multifunction device information may include information such as the port number of the multifunction device 20 in addition to the information described above.

[0035] The communication unit 190 communicates with external devices such as the multifunction peripheral 20. The communication unit 190 is configured by, for example, a communication device having a function of connecting to a wireless LAN in accordance with the IEEE802.11b / g / n standard. For example, the communication unit 190 is configured by a network interface card (NIC) or a communication module that can connect to a Long Term Evolution (LTE) / LTE-Advanced (LTE-A) / License-Assisted Access using LTE (LAA) / 5G line.

[0036] [1.2.2 Multifunction device] The functional configuration of the multifunction device 20 will be described with reference to Fig. 4. As shown in Fig. 4, the multifunction device 20 is configured to include a control unit 200, an image input unit 220, an image forming unit 230, a display unit 240, an operation unit 250, a storage unit 260, and a communication unit 290.

[0037] The control unit 200 controls the entire multifunction device 20. The control unit 200 realizes various functions by reading and executing various programs stored in the storage unit 260, and is configured with one or more arithmetic units (e.g., CPUs). The control unit 200 may also be configured as an SoC having multiple functions among those described below.

[0038] The control unit 200 executes a program stored in the storage unit 260 to function as an image processing unit 202. The image processing unit 202 performs various image processing operations on the image data input by the image input unit 220. For example, the image processing unit 202 performs sharpening processing and color conversion processing on the image data.

[0039] The image input unit 220 reads an original document and inputs the data of the read image. For example, the image input unit 220 is configured with a scanning device or the like equipped with a device that converts optical information, such as a CIS (Contact Image Sensor) or a CCD (Charge Coupled Device), into an electrical signal, and reads an original document placed on the table of the multifunction device 20.

[0040] The image forming unit 230 forms (prints) an image on a recording medium such as recording paper, etc. The image forming unit 230 is configured by, for example, a laser printer or the like that uses an electrophotographic method.

[0041] The display unit 240 displays various types of information. The display unit 240 is configured by a display device such as an LCD, an organic EL display, or a micro LED display.

[0042] The operation unit 250 accepts operations from a user who uses the multifunction device 20. The operation unit 250 is configured with input devices such as hardware keys (physical keys, numeric keypad) and touch sensors. The multifunction device 20 may be equipped with a touch panel in which the display unit 240 and the operation unit 250 are integrally formed.

[0043] The storage unit 260 stores various programs and various data required for the operation of the multifunction peripheral 20. The storage unit 260 is configured by a storage device such as an SSD, which is a semiconductor memory, or an HDD, for example.

[0044] The storage unit 260 secures a processing data storage area 262 and a job detail information storage area 264 as storage areas.

[0045] The processing data storage area 262 stores data to be processed by the multifunction device 20. The processing data is, for example, data representing an image formed by the image forming unit 230, or data to be transmitted to another device via the communication unit 290.

[0046] The job detail information storage area 264 stores detailed information (job detail information) about jobs executed by the multifunction device 20. As shown in FIG. 5, the job detail information includes, for example, a job ID (e.g., "25") that identifies the job, the address of the terminal device 10 (e.g., "192.168.0.101"), a port number (e.g., "19999") for communicating with the terminal device 10, the terminal device Connection ID (e.g., "9dc9001122334455") and the multifunction device Connection ID (e.g., "8ba9001122334455") used for communicating with the terminal device 10, key information (e.g., "******") used to encrypt information to be sent to the terminal device 10, operation information and job information (e.g., "simple copy, paper: A4, color mode: color, processed data: 20210801170000") that are sent and received information, and the job status (e.g., "not executed").

[0047] The processing data may be stored in the job detail information, or may be stored in the processing data storage area 262. When the processing data is stored in the processing data storage area 262, the job detail information may store information that identifies the processing data stored in the processing data storage area 262 (for example, a data name).

[0048] The address is, for example, an IP (Internet Protocol) address. The job status stores information indicating the progress of the job, such as "not executed," "executing," or "execution completed." The job detail information may also include information other than the information described above.

[0049] The communication unit 290 communicates with external devices such as the terminal device 10. The communication unit 290 is configured by a communication device (communication module) similar to the communication unit 190.

[0050] [1.3 Processing flow] [1.3.1 Processing Overview] An overview of the processing executed by the terminal device 10 and the multifunction device 20 will be described with reference to Fig. 6. First, the terminal device 10 accepts a request to start remote operation of the multifunction device 20 (S1000). Next, a handshake based on the QUIC protocol (QUIC Handshake) is executed between the terminal device 10 and the multifunction device 20 (S1002). This starts a session between the terminal device 10 and the multifunction device 20. In S1002, the terminal device 10 and the multifunction device 20 execute the following processing. (1)Generating and sending a Connection ID The terminal device 10 generates a terminal device Connection ID and transmits it to the multifunction device 20. The multifunction device 20 generates a multifunction device Connection ID and transmits it to the terminal device 10. As a result, the Connection ID is shared between the terminal device 10 and the multifunction device 20. (2) Determining the TLS shared key The terminal device 10 and the multifunction device 20 each generate a common key using information transmitted and received during a handshake. When the terminal device 10 and the multifunction device 20 confirm that the same common key has been generated, they establish the common key as TLS key information. For example, if the terminal device 10 and the multifunction device 20 transmit data encrypted using the common key and can decrypt data received from the other device using the common key, they can determine that the same common key has been generated.

[0051] By performing the handshake, a connection is established between the terminal device 10 and the multifunction device 20, and a session is started (established). At the start of the session, the multifunction device 20 stores the Connection ID and TLS key information used in the QUIC protocol (S1004). Similarly, at the start of the session, the terminal device 10 stores the Connection ID and TLS key information used in the QUIC protocol (S1006).

[0052] Next, the terminal device 10 and the multifunction device 20 communicate transmission and reception information (Application Data) using the Connection ID and the TLS key information (S1008).

[0053] For example, the multifunction device 20 encrypts information (UI data) of an operation screen using TLS key information. Furthermore, the multifunction device 20 transmits the encrypted information and a Connection ID for identifying the connection with the terminal device 10 (for example, a terminal device Connection ID received from the terminal device 10) to the terminal device 10. The terminal device 10 decrypts the information received from the multifunction device 20 using the TLS key information corresponding to the multifunction device 20, displays an operation screen based on the received information on the display unit 140, and accepts operation inputs from the user.

[0054] Furthermore, the terminal device 10 encrypts the operation information and job information using TLS key information and transmits them to the multifunction device 20. The multifunction device 20 decrypts the operation information and job information using the TLS key information corresponding to the terminal device 10, and stores the decrypted operation information and job information in association with the Connection ID and TLS key information related to communication with the terminal device 10.

[0055] Next, a case will be described in which the connection between the terminal device 10 and the multifunction peripheral 20 is cut off, the session is interrupted, and communication is interrupted. The connection is cut off, for example, due to a problem with the wireless communication network on the terminal device 10 side, or due to a change in the IP address of the terminal device 10 as a result of a change in the base station to which the terminal device 10 is connected.

[0056] When communication between the terminal device 10 and the multifunction device 20 is interrupted, the multifunction device 20 sets a communication timeout time (S1010). The communication timeout time is the time during which a reconnection from the terminal device 10 is accepted. The multifunction device 20 maintains a state in which it is able to accept a reconnection from the terminal device 10 (a state in which communication is available) until the communication timeout time has elapsed. On the other hand, after the communication timeout time has elapsed, the multifunction device 20 rejects a reconnection from the terminal device 10 (does not reconnect with the terminal device 10), thereby reducing consumption of resources (for example, memory) of the multifunction device 20 and preventing another user from being unable to operate the multifunction device 20 for a long period of time.

[0057] Furthermore, when the change of the IP address is completed or the network is restored, the terminal device 10 reconnects to the multifunction device 20 (S1012). For example, the terminal device 10 makes a reconnection request by transmitting the multifunction device Connection ID and information obtained by encrypting predetermined information using TLS key information to the multifunction device 20 (S1014).

[0058] The multifunction peripheral 20 identifies a job awaiting reconnection from the Connection ID received from the terminal device 10 and the TLS key information of the terminal device 10 (S1016). The multifunction peripheral 20 also executes a reconnection process with the terminal device 10 (S1018). For example, as part of the reconnection process, the multifunction peripheral 20 may confirm that normal communication with the terminal device 10 is possible, transmit information indicating the operation information or job information that needs to be transmitted next to the terminal device 10, or transmit information on an operation screen for inputting the next operation to the terminal device 10. The reconnection process connects the terminal device 10 and the multifunction peripheral 20, and the session is resumed, thereby resuming communication between the terminal device 10 and the multifunction peripheral 20.

[0059] Next, the terminal device 10 and the multifunction device 20 use the Connection ID and the TLS key information to communicate (resume) transmission and reception information (Application Data) in the same manner as in S1008 (S1020), thereby enabling remote operation again in the terminal device 10 (S1022).

[0060] In this way, when the network connection of the terminal device 10 is restored after the wireless or other connection on the terminal device 10 side is once disconnected, the terminal device 10 can resume the interrupted communication. This allows the terminal device 10 to resume communication for transmitting operation information and job information that have not been transmitted to the multifunction peripheral 20.

[0061] When the terminal device 10 resumes communication with the multifunction device 20, it resumes communication by the QUIC protocol using the Connection ID and TLS key information used in the QUIC protocol. The terminal device 10 does not need to perform a process to re-establish a session with the multifunction device 20 (session reconnection process), such as a handshake, and therefore can efficiently reconnect with the multifunction device 20.

[0062] Furthermore, although the communication from the terminal device 10 is interrupted on the multifunction peripheral 20 side during communication, the multifunction peripheral 20 stores the Connection ID and TLS key information used in the QUIC protocol and maintains a state in which it can accept communication. Therefore, the multifunction peripheral 20 can resume communication with the terminal device 10 using the Connection ID and TLS key information. Furthermore, by using the QUIC protocol, the multifunction peripheral 20 can connect to the terminal device 10 when communication from the terminal device 10 is resumed, and resume the session with the terminal device 10 without performing any special session resumption processing. This allows the multifunction peripheral 20 to efficiently resume communication with the terminal device 10 from where the communication was interrupted.

[0063] In this way, when resuming a session between the terminal device 10 and the multifunction device 20, there is no need to perform the process (session processing) to resume the session again between the terminal device 10 and the multifunction device 20, and it is possible to continue sending and receiving information (data processing) from the point where the interrupted communication was last performed.

[0064] 1.3.2 Terminal Processing Next, the process executed by the terminal device 10 will be described with reference to Fig. 7. The process shown in Fig. 7 is executed by the control unit 100 that reads out the operation application 162. It is assumed that the multifunction device information storage area 164 has stored therein in advance multifunction device information including information on the multifunction device name and address of the multifunction device 20.

[0065] First, the control unit 100 receives a request to start remote operation (step S100). For example, the control unit 100 displays a screen (multifunction device selection screen) that displays a list of multifunction device names included in the multifunction device information stored in the multifunction device information storage area 164 on the display unit 140. When a user selects one multifunction device 20 and performs an operation to start connection, the control unit 100 receives a request to start remote operation of the selected multifunction device 20.

[0066] Next, the control unit 100 executes a handshake with the multifunction device 20 using the QUIC protocol via the communication unit 190 (step S102). For example, the control unit 100 reads out the multifunction device information corresponding to the multifunction device 20 selected in step S100 from the multifunction device information storage area 164. Furthermore, the control unit 100 transmits a packet for performing the handshake to the address included in the read multifunction device information.

[0067] At this time, the control unit 100 generates a terminal device Connection ID and transmits it to the multifunction device 20, and receives the multifunction device Connection ID from the multifunction device 20. Furthermore, the control unit 100 generates a common key using information transmitted and received during the handshake. Furthermore, if the generated common key and the common key generated by the multifunction device 20 are identical, the control unit 100 determines the common key as TLS key information. For example, if the control unit 100 receives encrypted information from the multifunction device 20 and can decrypt the information with the generated common key, it can determine that the generated common key and the common key generated by the multifunction device 20 are identical.

[0068] Next, the control unit 100 stores the terminal device Connection ID, the multifunction device Connection ID, and the TLS key information in the multifunction device information corresponding to the multifunction device 20 selected in step S100 (step S104).

[0069] Next, the control unit 100 starts the remote operation (step S106). For example, the control unit 100 displays on the display unit 140 a screen (operation screen) for performing the remote operation.

[0070] Next, the control unit 100 transmits the operation information and job information via the communication unit 190 (step S108). For example, the control unit 100 encrypts the operation information indicating the operation input by the user using TLS key information corresponding to the destination multifunction device 20. Furthermore, the control unit 100 transmits a packet including the encrypted operation information and the multifunction device Connection ID corresponding to the destination multifunction device 20 to the multifunction device 20.

[0071] The control unit 100 may encrypt job information based on a user's operation using TLS key information, and transmit a packet including the multifunction device Connection ID corresponding to the destination multifunction device 20 and the encrypted operation information to the multifunction device 20. The control unit 100 may also read out a file selected by the user and generate the job information as data (processing data) indicating the contents of the file.

[0072] Furthermore, when control unit 100 receives information from multifunction device 20, it decrypts the received information using the TLS key information determined in step S102. When control unit 100 receives information about an operation screen from multifunction device 20, it may display an operation screen based on the information on display unit 140.

[0073] Next, the control unit 100 determines whether the connection with the multifunction device 20 has been disconnected (step S110). For example, the control unit 100 monitors the communication unit 190 to detect whether the connection with the multifunction device 20 has been disconnected. When the connection with the multifunction device 20 has been disconnected, the control unit 100 may display a message on the display unit 140 indicating that the connection with the multifunction device 20 has been disconnected.

[0074] When the connection with the multifunction peripheral 20 is cut off, the control unit 100 periodically determines whether the network has been restored (step S110; Yes → step S112). For example, the control unit 100 determines that the network has been restored when connection to a base station becomes possible or when an IP address is determined. Note that when the network has not been restored, the control unit 100 repeatedly executes the process of step S112 (step S112; No).

[0075] When the network is restored, the control unit 100 reconnects to the multifunction device 20 via the communication unit 190 using the multifunction device Connection ID and TLS key information stored in step S104 (step S112; Yes → step S114). As a result, the control unit 100 makes a reconnection request to the multifunction device 20. Note that the control unit 100 may display a message on the display unit 140 indicating that the remote connection to the multifunction device 20 is being resumed.

[0076] Furthermore, the control unit 100 executes a reconnection process (step S116). For example, the control unit 100 receives, from the multifunction peripheral 20, information indicating the operation information or job information that needs to be sent next, or receives information on an operation screen for inputting the next operation, and displays an operation screen based on the information on the display unit 140. Furthermore, when the reconnection process is completed, the control unit 100 may display, on the display unit 140, a message indicating that communication with the multifunction peripheral 20 has resumed or that remote operation has resumed.

[0077] If the control unit 100 receives information from the multifunction peripheral 20 indicating that the reconnection is rejected in step S116, the control unit 100 may display a message indicating that the reconnection with the multifunction peripheral 20 was not successful on the display unit 140, or may return to step S100. If the control unit 100 determines in step S110 that the connection with the multifunction peripheral 20 has not been disconnected, the control unit 100 omits the processes from step S112 to step S116 (step S110; No).

[0078] Next, the control unit 100 determines whether the job has been executed (step S118). For example, when the control unit 100 receives information from the multifunction device 20 indicating that the job has been executed, the control unit 100 determines that the job has been executed.

[0079] If the job has been executed, the control unit 100 ends the remote operation (step S118; Yes→step S120). At this time, the control unit 100 may display a message indicating that the job has been executed or a multifunction device selection screen on the display unit 140. If the control unit 100 determines that the job has not been executed, it returns to step S108 (step S118; No→step S108).

[0080] [1.3.3 Multifunction Device Processing] Next, the process executed by the multifunction device 20 will be described with reference to Figures 8 and 9. The process shown in Figures 8 and 9 is executed by the control unit 200 that reads out a program stored in the storage unit 260.

[0081] First, the processing shown in Fig. 8 will be described. When the control unit 200 receives information (packets) for performing a handshake from the terminal device 10 via the communication unit 290, it executes a handshake with the terminal device 10 using the QUIC protocol (step S150). For example, the control unit 200 receives a terminal device Connection ID from the terminal device 10, or generates a multifunction device Connection ID and transmits it to the terminal device 10. The control unit 200 also generates a shared key using information transmitted and received when performing the handshake, and if the generated shared key is identical to the shared key generated by the terminal device 10, it determines the shared key as TLS key information.

[0082] In addition, when the control unit 200 executes the processing in step S150 and displays a screen on the display unit 240 that accepts operations via the operation unit 250, the control unit 200 may also display a message indicating that processing to start remote operation is being executed.

[0083] Next, the control unit 200 stores the terminal device Connection ID, the multifunction device Connection ID, and the confirmed TLS key information (step S152). For example, the control unit 200 issues a job ID, and stores job detail information including the job ID, the terminal device Connection ID, the multifunction device Connection ID, and the TLS key information in the job detail information storage area 264. Furthermore, the control unit 200 stores the address and port number of the terminal device 10 that performed the handshake in the job detail information, and sets the job status of the job detail information to "not executed."

[0084] Next, the control unit 200 starts accepting remote operations from the terminal device 10 (step S154). At this time, the control unit 200 may stop accepting operation input via the operation unit 250, or may display on the display unit 240 a screen (lock screen) indicating that a remote operation is being performed.

[0085] Next, the control unit 200 receives and stores operation information and job information from the terminal device 10 (step S156). For example, when the control unit 200 receives information from the terminal device 10, it decrypts the received information using the TLS key information corresponding to the terminal device 10. At this time, when the control unit 200 receives operation information and job information from the terminal device 10, it stores the operation information and job information received from the terminal device 10 in the job detail information corresponding to the terminal device 10.

[0086] Furthermore, when the control unit 200 receives processing data from the terminal device 10, it may store the processing data in the processing data storage area 262 and store the data name of the processing data in the job detail information.

[0087] The control unit 200 may also transmit information (UI data) of an operation screen for inputting the next operation to the terminal device 10. For example, the control unit 200 encrypts the information of the operation screen using TLS key information corresponding to the destination terminal device 10. Furthermore, the control unit 200 transmits a packet including the terminal device Connection ID corresponding to the destination terminal device 10 and the encrypted information of the operation screen to the terminal device 10.

[0088] Next, the control unit 200 determines whether the connection with the terminal device 10 has been disconnected (step S158). For example, the control unit 200 monitors the communication unit 290 and detects that the connection with the multifunction peripheral 20 has been disconnected.

[0089] When the connection with the terminal device 10 is cut off, the control unit 200 stops accepting remote operations from the terminal device 10 (step S158; Yes→step S160).

[0090] Furthermore, the control unit 200 sets a communication timeout period for the disconnected terminal device 10 (step S162). For example, the control unit 200 may store the communication timeout period in the storage unit 260 in association with the terminal device Connection ID of the destination terminal device 10 in the stored job information detailed information. Furthermore, the communication timeout period may be set in advance, or may be set by an administrator of the multifunction peripheral 20, etc.

[0091] At this time, the control unit 200 may display on the display unit 240 a screen for accepting an operation via the operation unit 250, and may also display a message indicating that the remote operation has been interrupted.

[0092] Next, the control unit 200 executes a process (reconnection handling process) to handle a reconnection from the terminal device 10 (step S164). The reconnection handling process will be described later. Furthermore, the control unit 200 resumes the connection with the terminal device 10 by executing the reconnection handling process.

[0093] Next, the control unit 200 determines whether or not the job based on the operation information and job information stored in step S156 is executable (step S166). If the job is not executable, the control unit 200 returns to step S156 (step S166; No→step S156).

[0094] On the other hand, if the job is executable, the control unit 200 executes the job based on the job information stored in step S156 (step S166; Yes → step S168). At this time, the control unit 200 may update the job status of the job detail information corresponding to the job to be executed to information such as "in progress" or "execution completed" depending on the progress of the job. Furthermore, when execution of the job is completed, the control unit 200 may transmit information indicating that execution of the job is completed to the terminal device 10.

[0095] Next, the control unit 200 ends the reception of the remote operation (step S170). For example, the control unit 200 may disconnect the connection with the terminal device 10 or display on the display unit 240 a screen for receiving an operation from the operation unit 250.

[0096] Next, the reconnection handling process will be described with reference to Fig. 9. The control unit 200 determines whether or not a reconnection has been requested from the terminal device 10 (step S180). If a reconnection has not been requested from the terminal device 10, the control unit 200 repeatedly executes the process of step S180 (step S180; No).

[0097] When a reconnection is made from the terminal device 10, the control unit 200 identifies a job awaiting reconnection from the Connection ID received from the terminal device 10 and the TLS key information of the terminal device 10 (step S180; Yes -> step S182). For example, the control unit 200 reads out job detail information including the same multifunction device Connection ID as the multifunction device Connection ID received from the terminal device 10 from the job detail information storage area 264. Furthermore, the control unit 200 decrypts the information received from the terminal device 10 using the TLS key information included in the read job detail information. If the decryption is successful, the control unit 200 identifies the job based on the operation information and job information included in the read job detail information as the job awaiting reconnection.

[0098] Next, the control unit 200 determines whether the communication timeout time set for the terminal device 10 that has reconnected has elapsed (step S184). If the communication timeout time has elapsed, the control unit 200 rejects the reconnection from the terminal device 10 and ends the process shown in FIG. 8 (step S184; No -> step S190). In this case, the control unit 200 does not reconnect to the destination terminal device 10 and ends the remote operation with the terminal device 10. Note that the control unit 200 may transmit information indicating that the reconnection is rejected to the terminal device 10. If the communication timeout time has elapsed, the terminal device 10 establishes a new connection by newly executing a handshake with the multifunction peripheral 20, and can execute a new remote operation.

[0099] On the other hand, if the communication timeout time has not elapsed, the control unit 200 executes a reconnection process (step S184; Yes -> step S186). For example, the control unit 200 checks whether communication with the terminal device 10 is possible. The control unit 200 also stores (updates) the new IP address and port number received from the terminal device 10 in step S180 in the job detail information read out in step S182. As a result, even if the IP address or port number of the terminal device 10 is changed, the control unit 200 can use the changed IP address and port number.

[0100] In addition, based on the content of the job (job information) identified in step S182, the control unit 200 may transmit to the terminal device 10 operation information or job information that needs to be transmitted next, or information on an operation screen for inputting the next operation.

[0101] Furthermore, while the control unit 200 is executing the reconnection process, the control unit 200 may display on the display unit 240 a message indicating that a reconnection request is being accepted from the terminal device 10. When the reconnection process is completed and remote operation from the terminal device 10 is resumed, the control unit 200 may display on the display unit 240 a screen (lock screen) indicating that remote operation is being performed.

[0102] [1.4 Example of operation] Next, with reference to the drawings, a description will be given of screens displayed on the display unit 140 of the terminal device 10 and the display unit 240 of the multifunction peripheral 20 when the processing of this embodiment is executed.

[0103] 10A and 10B are diagrams showing screens that are displayed when communication for remote operation (remote communication) is initiated. Fig. 10A is a diagram showing a multifunction device selection screen W100 that is displayed on the display unit 140. The multifunction device selection screen W100 includes an area E100 in which the multifunction device names included in the multifunction device information stored in the terminal device 10 are displayed in a selectable manner. The user can select one multifunction device 20 from the area E100 and select the connect button B100 to cause the terminal device 10 to start connecting to the selected multifunction device 20.

[0104] 10(b) is a diagram showing a display screen W110 displayed on the display unit 240. The display screen W110 is displayed, for example, when a handshake is being executed. A message M110 indicating that communication for remote operation has started is displayed on the display screen W110.

[0105] FIG. 11 shows a screen displayed when a remote operation is being performed. FIG. 11(a) is an example of an operation screen W120 displayed on the display unit 140. The operation screen W120 displays buttons and the like for setting a job to be executed by the multifunction peripheral 20. For example, as shown in FIG. 11(a), a button for selecting a job to be executed by the multifunction peripheral 20 is displayed. The operation screen W120 may also display buttons or lists for setting a job (e.g., the number of copies to be printed, color mode, paper size, destination to which processed data is to be sent, etc.) and selecting a file (processed data) to be sent to the multifunction peripheral 20. Operation information corresponding to an operation input by the user via the operation screen W120 and job information based on the user's operation are sent to the multifunction peripheral 20.

[0106] 11(b) is an example of a lock screen W130 displayed on the display unit 240. By displaying the lock screen, the user can understand that operations via the operation unit 250 are not accepted. The lock screen W130 may include an interrupt button B130 for performing operations via the operation unit 250 when a remote operation is being performed. When the interrupt button B130 is selected, the multifunction peripheral 20 may interrupt the remote operation and enter a state in which operations via the operation unit 250 can be input.

[0107] 12A and 12B are diagrams showing screens that are displayed when the connection between the terminal device 10 and the multifunction peripheral 20 is disconnected and communication (remote communication) is interrupted. FIG. 12A is a display example of a display screen W140 that is displayed on the display unit 140. The display screen W140 includes a message M140 indicating that the connection with the multifunction peripheral 20 has been disconnected. By checking the message M140, the user can understand that the connection with the multifunction peripheral 20 has been disconnected. Furthermore, by selecting a resume button B140, the user can instruct the terminal device 10 to resume communication with the multifunction peripheral 20.

[0108] 12(b) is a display example of a display screen W150 displayed on the display unit 240. The display screen W150 includes a message M150 indicating that the execution of a job based on a remote operation has been interrupted due to an interruption of remote communication. By checking the message M150, the user can understand that the remote operation has been interrupted and that there is a possibility that the remote operation will be resumed.

[0109] Fig. 13 shows a screen displayed when remote communication is resumed. Fig. 13(a) is a display example of a display screen W160 displayed on the display unit 140. A message M160 indicating that the connection between the terminal device 10 and the multifunction peripheral 20 is being resumed is displayed on the display screen W160.

[0110] 13(b) is a display example of a display screen W170 displayed on the display unit 240. A message M170 indicating that the connection between the terminal device 10 and the multifunction peripheral 20 is being resumed is displayed on the display screen W170. Note that the message M170 may include a cancel button B170 for canceling the resumption of the connection between the terminal device 10 and the multifunction peripheral 20. When the cancel button B170 is selected, the resumption of the connection between the terminal device 10 and the multifunction peripheral 20 is canceled, and a state may be established in which operations can be input via the operation unit 250.

[0111] Fig. 14 shows a screen displayed when remote communication is resumed. Fig. 14(a) is a display example of a display screen W180 displayed on the display unit 140. The display screen W180 displays a message M180 indicating that communication between the terminal device 10 and the multifunction peripheral 20 has resumed. Fig. 14(b) is a display example of a lock screen W190 displayed on the display unit 240. The lock screen W190 is the same screen as the lock screen W130 of Fig. 11(b).

[0112] In this way, the terminal device and multifunction peripheral of this embodiment can resume communication by restarting a session using the Connection ID and TLS key information used in the QUIC protocol. In particular, the multifunction peripheral of this embodiment resumes communication with the terminal device without using mobile-device-specific information such as a phone number, eliminating the need to send, receive, or store mobile-device-specific information. Therefore, even if communication is intercepted or the storage unit of the multifunction peripheral is illegally read, the mobile-device-specific information will not be leaked. Furthermore, because the multifunction peripheral of this embodiment does not associate mobile-device-specific information with job information, it is possible to prevent job information from being leaked due to spoofing.

[0113] Furthermore, the terminal device and multifunction peripheral of this embodiment encrypts and transmits / receives information. This reduces the risk of leakage of processing data such as print data and transmitted / received information (operation information and job information). In this way, this embodiment does not use mobile-specific information and uses a highly secure communication standard, thereby increasing security regarding the resumption of communication and reducing the risk of information leakage. As a result, the terminal device and multifunction peripheral of this embodiment can resume communication securely.

[0114] Furthermore, the terminal device and multifunction peripheral of this embodiment communicate using a communication protocol that can flexibly accommodate network interruptions and IP address changes, and retains communication information such as a Connection ID and TLS key information. This allows the user to continue operating the terminal device and multifunction peripheral of this embodiment even if a network failure or IP address change occurs on the terminal device side. In particular, if the multifunction peripheral of this embodiment loses connection with the terminal device, it reconnects to the terminal device using a Connection ID that identifies the connection and key information corresponding to the terminal device. The terminal device and multifunction peripheral of this embodiment do not require special processing to resume a session when reconnecting after a disconnection, so reconnection and communication can be efficiently executed.

[0115] [2. Second Embodiment] Next, a second embodiment will be described. In addition to the processing described in the first embodiment, the second embodiment switches the operation of the terminal device depending on the state of the multifunction device when communication between the terminal device and the multifunction device resumes. In this embodiment, Fig. 7 in the first embodiment is replaced with Fig. 15, and Fig. 9 with Fig. 16. Note that the same processing is assigned the same reference numerals, and the description thereof will be omitted.

[0116] When a failure occurs in the network on the terminal device 10 side, it is expected that the communication between the terminal device 10 and the multifunction device 20 will be interrupted for a long time because it will take time to restore the network. If the communication is interrupted for a long time, the state of the multifunction device 20 may change during the communication interruption. Therefore, the terminal device 10 of this embodiment operates in accordance with the state of the multifunction device 20 when communication with the multifunction device 20 is resumed.

[0117] [2.1 Overview of terminal device operation] In this embodiment, when communication between the terminal device 10 and the multifunction device 20 is interrupted, if the state of the multifunction device 20 becomes any of the following states (1) to (6), the terminal device 10 switches its operation depending on the state of the multifunction device 20.

[0118] (1) When the operation state of the multifunction device 20 shifts to the energy saving state The energy saving state is one of the operating states of the multifunction device 20, and is an operating state in which power supply is more limited than in the normal operating state. The multifunction device 20 can individually set the time (energy saving setting time) for the operating state to transition to the energy saving state as a setting for remote operation. The multifunction device 20 transitions to the energy saving state if communication with the terminal device 10 is cut off during remote operation and communication with the terminal device 10 is not resumed after the energy saving setting time has elapsed. Note that the multifunction device 20 may be configured to select whether or not to retain detailed job information, operation information, and job information.

[0119] Furthermore, the multifunction peripheral 20 returns from the energy saving state to the normal state (returns to energy saving state) when communication with the terminal device 10 is resumed. The normal state is an operating state in which power is supplied to each functional unit of the multifunction peripheral 20.

[0120] When the terminal device 10 receives a notification from the multifunction peripheral 20 indicating that the multifunction peripheral 20 is returning from energy saving mode, the terminal device 10 may display a message on the display unit 140 indicating that the multifunction peripheral 20 is returning from energy saving mode. Furthermore, the terminal device 10 may display a message on the display unit 140 indicating that it will take time to return from energy saving mode and therefore that it will take time to update the operation screen for inputting the next operation from the terminal device 10. Note that by storing UI data for operating the multifunction peripheral 20 in advance in the storage unit 160, the terminal device 10 may display an operation screen based on the UI data and accept the next operation.

[0121] (2) When the power of the multifunction device 20 is turned off When communication with the terminal device 10 is disconnected, the multifunction device 20 may be directly powered off by the user. In this case, since the terminal device 10 cannot resume communication with the multifunction device 20, the terminal device 10 displays a message urging the user to check the status of the multifunction device 20. Furthermore, the terminal device 10 may prepare in advance by storing a resume screen (for example, an operation screen for inputting the next operation) for resuming operation when the multifunction device 20 is powered on in the storage unit 160 or the like, and display the resume screen.

[0122] Furthermore, if the terminal device 10 is unable to continue the print / transmission job after the multifunction peripheral 20 is powered on, the terminal device 10 may display on the display unit 140 a message indicating that the job cannot be continued.

[0123] (3) When a communication timeout occurs from the destination When using the Scan To Email function or the like to send image data or the like to a destination specified by a user, the multifunction device 20 does not receive all of the job information from the terminal device 10, but rather receives the job information page by page from the terminal device 10 and can execute the job information sequentially.

[0124] When communication with the terminal device 10 is interrupted while the multifunction device 20 is executing a job based on job information for each page, a timeout may occur in communication with the destination. In this case, the multifunction device 20 stores the occurrence of a timeout in association with the job detail information of the terminal device 10 with which communication was interrupted. Furthermore, when a timeout occurs in communication with the destination, the terminal device 10 displays on the display unit 140 a message indicating that a timeout has occurred at the destination of the job information.

[0125] (4) If the job is canceled on the MFP 20 side When communication between the multifunction peripheral 20 and the terminal device 10 is disconnected, if it is possible to cancel an interrupted operation or job, the job related to the remote operation with the terminal device 10 may be canceled on the multifunction peripheral 20 side. In this case, the terminal device 10 displays a message on the display unit 140 indicating that the job has been canceled.

[0126] (5) If a problem occurs with the multifunction device 20 When communication between the multifunction device 20 and the terminal device 10 is disconnected, a problem may occur in the multifunction device 20. In this case, the terminal device 10 displays a message on the display unit 140 indicating that a problem has occurred in the multifunction device 20.

[0127] (6) When the settings of the multifunction device 20 are changed The settings or state of the multifunction device 20 may be changed while communication with the terminal device 10 is disconnected. For example, the size or orientation of paper in the paper tray or the contents of the address book (information on destinations for image data) may be changed in the multifunction device 20.

[0128] When communication with the terminal device 10 is resumed, the multifunction device 20 transmits information indicating that the settings of the multifunction device 20 have been changed to the terminal device 10. In this case, the terminal device 10 displays a message indicating that the settings of the multifunction device 20 have been changed on the display unit 140. The terminal device 10 may also ask the user whether or not to update the part of the job information that has been changed from the previous information (changed information).

[0129] [2.2 Processing flow] 2.2.1 Terminal Devices The process executed by the terminal device 10 of this embodiment will be described with reference to Fig. 15. In this embodiment, after reconnecting to the multifunction peripheral 20 (step S114), the control unit 100 determines whether or not there is a response from the multifunction peripheral 20 within a predetermined time period (step S200). The predetermined time period may be set in advance or may be set by the user.

[0130] If there is a response from the multifunction device 20 within a predetermined time, the control unit 100 acquires the status of the multifunction device 20 (step S200; Yes -> step S202). For example, the control unit 100 may acquire the status of the multifunction device 20 by inquiring about the status of the multifunction device 20. Alternatively, the control unit 100 may acquire the status of the multifunction device 20 based on the response from the multifunction device 20 by configuring the multifunction device 20 to make a response including information indicating the status of the multifunction device 20 when reconnected by the terminal device 10.

[0131] As described in the process overview, the following states are acquired as the states of the multifunction device 20. Energy saving recovery is performed A communication timeout occurred from the destination The job was canceled - A problem occurred on the multifunction device 20 side. - The settings on the multifunction device 20 have been changed.

[0132] The control unit 100 displays a message indicating the status of the multifunction device 20 according to the status of the multifunction device 20 (step S204). For example, when the multifunction device 20 is in the state of recovering from energy saving mode, the control unit 100 displays a message indicating that the multifunction device 20 is recovering from energy saving mode on the display unit 140. At this time, when the multifunction device 20 is in the state of recovering from energy saving mode, the control unit 100 may display an operation screen for inputting the next operation. Furthermore, when the status of the multifunction device 20 is a state in which settings have been changed on the multifunction device 20 side, the control unit 100 may ask the user whether to update the changed parts from the previous information.

[0133] The control unit 100 displays a message indicating the status of the multifunction device, and then executes the reconnection process (step S116). Note that if there is no change in the status of the multifunction device 20, the control unit 100 may omit the process of step S204 and execute the reconnection process.

[0134] On the other hand, if there is no response from the multifunction device 20 within the predetermined time, the control unit 100 displays a message on the display unit 140 prompting the user to check the multifunction device (step S200; No→step S206). At this time, the control unit 100 may display a resume screen on the display unit 140.

[0135] Furthermore, the control unit 100 may periodically reconnect to the multifunction device 20 so that the connection is resumed when the power of the multifunction device 20 is turned on (when the power is turned on). Here, when the connection with the multifunction device 20 is resumed, the control unit 100 receives information indicating whether the job can be continued (step S208), and determines whether the job cannot be continued (step S210).

[0136] If the job cannot be continued, the control unit 100 displays a message indicating that the job cannot be continued on the display unit 140 (step S210; Yes→step S212), and ends the processing shown in Fig. 15. On the other hand, if the job can be continued, the control unit 100 executes the processing of step S116 (step S210; No→step S116).

[0137] [2.2.2 Multifunction device] (Reconnection handling process) The reconnection handling process executed by the multifunction peripheral 20 of this embodiment will be described with reference to Fig. 16. The control unit 200 determines whether a change has occurred in the state of the multifunction peripheral 20 (step S280). Changes in the state of the multifunction peripheral 20 include transitioning to energy saving mode, a timeout occurring at the destination of job information, a job being canceled, a problem occurring, and a change in settings.

[0138] When a change occurs in the state of the multifunction device 20, the control unit 200 stores the changed state of the multifunction device 20 in association with the information of the terminal device 10 with which communication is interrupted (step S280; Yes → step S282). For example, the control unit 200 may store information indicating the state of the multifunction device 20 in the job detailed information whose status is "not executed" among the job detailed information, or may store information indicating the state of the multifunction device 20 separately from the job detailed information in association with the job detailed information.

[0139] If there is no change in the state of the multifunction device 20, the control unit 200 omits the process in step S282 (step S280; No).

[0140] Furthermore, before executing the reconnection process, the control unit 200 transmits information indicating the status of the multifunction peripheral 20 to the terminal device 10 that transmitted the information regarding the job waiting for reconnection (step S284). The control unit 200 may transmit the information indicating the status of the multifunction peripheral 20 in response to an inquiry from the terminal device 10, or may transmit the information by including it in a response to the reconnection request from the terminal device 10.

[0141] (Power-on processing) When the power is turned on, the control unit 200 executes the power-on process shown in Fig. 17. First, the control unit 200 identifies a job that is waiting for reconnection. For example, the control unit 200 identifies a job that corresponds to operation information and job information included in job detail information whose status is "not executed" as a job that is waiting for reconnection.

[0142] Next, the control unit 200 determines whether the job identified in step S290 can be continued or not, and transmits information indicating whether the job can be continued or not to the terminal device 10 that transmitted the job information corresponding to the job (step S292).

[0143] By executing the above-mentioned processing by the terminal device 10 and the multifunction device 20, the terminal device 10 can obtain the status of the multifunction device 20 after communication with the multifunction device 20 is resumed, and can switch processing depending on the status of the multifunction device 20.

[0144] In this way, when reconnecting with the multifunction device, the terminal device of this embodiment can display an appropriate message according to the status of the multifunction device and prompt the user to enter the next operation, thereby enabling the terminal device of this embodiment to realize highly convenient remote operation for the user.

[0145] 3. Third Embodiment Next, a third embodiment will be described. In addition to the processing described in the first embodiment, the third embodiment is an embodiment that corresponds to a case where a remote operation is requested by another user while a remote operation by a certain user is suspended. In this embodiment, Fig. 6 is replaced with Figs. 18 to 20, Fig. 7 with Fig. 21, Fig. 8 with Fig. 22, and Fig. 9 with Fig. 23. Note that the same processing is assigned the same reference numerals, and the description thereof will be omitted.

[0146] When communication with the terminal device 10 is interrupted, the multifunction peripheral 20 may execute an operation or job by another user (another terminal device 10) by remotely operating the multifunction peripheral 20. In this case, when the interrupted communication is resumed and the other user is performing a remote operation, the terminal device 10 that resumed the interrupted communication displays a message indicating that the other user is performing an operation.

[0147] Note that when the interrupted communication is resumed and another user is performing a remote operation, the multifunction peripheral 20 may reserve a remote operation from the terminal device 10 that resumed communication. Also, the terminal device 10 may store UI data for operating the multifunction peripheral 20 in advance in the terminal device 10, and when the multifunction peripheral 20 is being operated by another user, display an operation screen based on the UI data and accept the next operation. Also, when the remote operation by the other user ends, the multifunction peripheral 20 may notify the terminal device 10 that resumed communication that remote operation is now possible. When the terminal device 10 receives a notification from the multifunction peripheral 20 indicating that remote operation is now possible, it resumes the remote operation.

[0148] The multifunction peripheral 20 may be able to set whether to permit or prohibit remote operations or job execution by another user when communication with the terminal device 10 is interrupted. When prohibiting another user from performing another operation or job, the multifunction peripheral 20 may display a message on the display unit 240 to the other user indicating that the other user is prohibited from performing another operation or job.

[0149] When communication with the terminal device 10 is interrupted, the setting of whether to permit or prohibit remote operation or job execution by another user is set by, for example, an administrator of the multifunction peripheral 20 and stored in the storage unit 260.

[0150] [3.1 Processing Overview] Next, an overview of the processing of this embodiment will be described with reference to Fig. 18 to Fig. 20. Note that the description will be given assuming that the multifunction peripheral 20 can be connected to two terminal devices 10 (a first terminal device 10 and a second terminal device 10).

[0151] First, the first terminal apparatus 10 and the multifunction peripheral 20 execute the processes from S1000 to S1010. As a result, the multifunction peripheral 20 maintains a state in which it can accept a reconnection request from the first terminal apparatus 10 (a state in which communication can be accepted).

[0152] On the other hand, the second terminal device 10 receives the remote operation start request (S3000), and executes a handshake (QUIC Handshake) with the multifunction peripheral 20 according to the QUIC protocol to establish a connection (S3002).

[0153] The multifunction device 20 stores the Connection ID and TLS key information related to communication with the second terminal device 10 (S3004). Similarly, the second terminal device 10 stores the Connection ID and TLS key information (S3006). The processes from S3000 to S3006 are the same as the processes from S1000 to S1006.

[0154] Next, the multifunction device 20 determines whether the setting allows remote operation or job execution by another user while communication with the terminal device 10 is interrupted (S3008). If remote operation or job execution by another user while communication with the terminal device 10 is interrupted is not allowed, the multifunction device 20 sends a busy notification to the second terminal device 10 (S3008; No→S3010). The busy notification is a notification indicating that remote operation cannot be performed because another user is using the multifunction device 20. The second terminal device 10 that has received the busy notification displays a message on the display unit 140 indicating that another user is using the multifunction device 20 (S3012).

[0155] On the other hand, if remote operation or job execution by another user is permitted while communication with the terminal apparatus 10 is interrupted, the multifunction peripheral 20 starts communication with the second terminal apparatus 10 and uses the Connection ID and TLS key information to communicate transmission and reception information (Application Data) (S3008; Yes → S3014). This enables remote operation of the second terminal apparatus 10 (S3016). The second terminal apparatus 10 terminates the remote operation by performing termination processing on the multifunction peripheral 20. The termination processing is, for example, processing for transmitting information (Application Data) indicating that the remote operation is to be terminated (S3018). At this time, the second terminal apparatus 10 may disconnect the connection with the multifunction peripheral 20. This terminates use of the multifunction peripheral 20 by the user of the second terminal apparatus 10.

[0156] On the other hand, when the second terminal apparatus 10 is performing a remote operation, if the change of the IP address of the first terminal apparatus 10 is completed or the network of the first terminal apparatus 10 is restored, the first terminal apparatus 10 requests the multifunction peripheral 20 to reconnect (S1014). The multifunction peripheral 20 identifies a job waiting for reconnection from the multifunction peripheral Connection ID received from the first terminal apparatus 10 and the TLS key information corresponding to that multifunction peripheral Connection ID (S1016). This allows the multifunction peripheral 20 to identify that remote operation is being performed again from the first terminal apparatus 10 with which communication was interrupted.

[0157] If another user is using the multifunction peripheral 20, the multifunction peripheral 20 notifies the first terminal apparatus 10 that it is busy (S3018; Yes → S3020). The first terminal apparatus 10 that has received the busy notification displays a message on the display unit 140 indicating that another user is using the multifunction peripheral 20 (S3022). At this time, the multifunction peripheral 20 designates the user of the first terminal apparatus 10 that has reconnected when another user is using the multifunction peripheral 20 as a waiting user. As a result, the multifunction peripheral 20 puts the first terminal apparatus 10 into a state waiting for remote operation.

[0158] On the other hand, if another user is not using the multifunction device 20, the multifunction device 20 does not send a busy notification, but executes a reconnection process with the terminal device 10, and resumes communication of transmitted and received information (S3018; No → S1018 → S1020).

[0159] Furthermore, when the remote operation is completed, the multifunction device 20 determines whether or not there is a waiting user (S3024). If there is a waiting user, the multifunction device 20 executes a reconnection process with the terminal device 10 used by the waiting user (the first terminal device 10 in a remote waiting state) (S3024; Yes → S1018). At this time, the multifunction device 20 may notify the terminal device 10 that the operation by another user has been completed (operation completion notification). When the operation completion notification is sent from the multifunction device 20, the terminal device 10 and the multifunction device 20 may execute a reconnection process.

[0160] As a result, the first terminal device 10 and the multifunction device 20 resume communication of predetermined information (S1020). Also, the first terminal device 10 becomes remotely operable (S1022).

[0161] 3.2 Terminal Device Processing The flow of processing by the terminal device 10 in this embodiment will be described with reference to FIG.

[0162] In this embodiment, after executing the process in step S104, the control unit 100 determines whether or not a busy notification has been received from the multifunction peripheral 20 (step S300). If a busy notification has been received, the control unit 100 displays a message on the display unit 140 indicating that another user is using the multifunction peripheral 20 (step S300; Yes→step S302), and ends the process shown in FIG. 21. In this case, the control unit 100 may execute the process again from step S100. Note that if a busy notification has not been received, the control unit 100 starts remote operation (step S300; No→step S106).

[0163] After executing the process in step S114, the control unit 100 determines whether or not a busy notification has been received from the multifunction peripheral 20 (step S304). If a busy notification has been received, the control unit 100 displays a message on the display unit 140 indicating that another user is using the multifunction peripheral 20 (step S304; Yes→step S306). If a busy notification has not been received, the control unit 100 executes a reconnection process (step S304; No→step S116).

[0164] Next, the control unit 100 determines whether or not the control unit 100 has received a notification from the multifunction device 20 that the operation of the multifunction device 20 by another user has ended (step S308). If the control unit 100 has received an operation end notification from the multifunction device 20 indicating that the operation of the multifunction device 20 by another user has ended, the control unit 100 executes a reconnection process (step S308; Yes -> step S116). If the control unit 100 has not received an operation end notification, the control unit 100 repeats the process in step S308 until the control unit 100 receives an operation end notification (step S308; No).

[0165] By the above-described processing, if a busy notification is received when the control unit 100 reconnects to the multifunction device 20, the control unit 100 waits until an operation completion notification is received, and when the operation completion notification is received, the control unit 100 can execute the reconnection processing.

[0166] [3.3 Processing of multifunction devices] The flow of processing by the multifunction peripheral 20 in this embodiment will be described with reference to Figures 22 and 23. First, the processing shown in Figure 22 will be described.

[0167] After executing the process in step S152, the control unit 200 determines whether another user is using the multifunction peripheral 20 (step S350). If another user is using the multifunction peripheral 20, a busy notification is sent to the terminal device 10 that performed the handshake (step S350; Yes→step S356).

[0168] Furthermore, when another user is not using the device, the control unit 200 determines whether or not there is a user whose communication based on a remote operation is interrupted (step S350; No→step S352). For example, when a remote operation is interrupted, the control unit 200 stores information indicating that the remote operation has been interrupted in the storage unit 260. When information indicating that the remote operation has been interrupted is stored in the storage unit 260, the control unit 200 determines that there is a user whose communication is interrupted.

[0169] If there is a suspended user, the control unit 200 determines whether another user is permitted to perform a remote operation or execute a job (step S352; Yes→step S354). If another user is not permitted to perform a remote operation or execute a job, the control unit 200 sends a busy notification to the terminal device 10 that performed the handshake (step S354; No→step S356).

[0170] If there is no suspended user, the control unit 200 starts accepting remote operations from the terminal device 10 that performed the handshake (step S352; No→step S154). Even if there is a suspended user, the control unit 200 starts accepting remote operations from the terminal device 10 that performed the handshake if another user is permitted to perform remote operations or execute jobs during the suspension (step S354; Yes→step S154).

[0171] Next, the reconnection handling process executed by the multifunction peripheral 20 of this embodiment will be described with reference to Fig. 23. In the description of Fig. 23, the terminal device 10 whose connection has been interrupted will be referred to as the first terminal device 10, and another terminal device 10 different from the first terminal device 10 will be referred to as the second terminal device 10.

[0172] First, the control unit 200 determines whether a handshake has been requested by the second terminal device 10 (step S380). If a handshake has been requested, the multifunction peripheral 20 executes a handshake with the second terminal device (step S380; Yes→step S382).

[0173] Next, if there is a suspended user, the control unit 200 determines whether another user is permitted to perform remote operations or execute jobs (step S384). If another user is permitted to perform remote operations or execute jobs, the control unit 200 accepts remote operations from a user (another user) using the second terminal device (step S384; Yes → step S386). On the other hand, if another user is not permitted to perform remote operations or execute jobs, the control unit 200 sends a busy notification to the second terminal device (step S384; No → step S388).

[0174] If the control unit 200 determines in step S380 that a handshake has not been requested by the second terminal apparatus 10, it omits the processes from step S382 to step S388 (step S380; No).

[0175] Then, the control unit 200 determines whether or not a reconnection has been received from the first terminal apparatus 10 (step S390). If a reconnection has not been received from the first terminal apparatus 10, the control unit 200 returns to step S380 (step S390; No→step S380). On the other hand, if a reconnection has been received from the first terminal apparatus 10, the control unit 200 identifies a job waiting for reconnection from the Connection ID received from the first terminal apparatus 10 and the TLS key information of the first terminal apparatus 10 (step S390; Yes→step S182).

[0176] Next, the control unit 200 determines whether or not the communication timeout period set for the first terminal apparatus 10 that has reconnected has elapsed (step S184). If the timeout period has elapsed, the control unit 200 rejects the reconnection from the first terminal apparatus 10 (step S184; No→step S190). On the other hand, if the timeout period has not elapsed, the control unit 200 determines whether or not another user is using the multifunction peripheral 20 (step S184; Yes→step S392).

[0177] If another user is using the multifunction peripheral 20, the control unit 200 notifies the first terminal apparatus 10 that it is busy (step S392; Yes→step S394). The control unit 200 also determines whether or not the other user has finished using the multifunction peripheral 20 (step S396). If the other user has not finished using the multifunction peripheral 20, the control unit 200 repeats the process in step S396 (step S396; No). On the other hand, if the other user has finished using the multifunction peripheral 20, the control unit 200 notifies the first terminal apparatus 10 that the operation has been completed (step S396; Yes→step S398).

[0178] If the control unit 200 determines in step S392 that another user is not using the multifunction peripheral 20, it skips the processes from step S394 to step S398 described above and executes the reconnection process in step S186 (step S392; No -> step S186). In this case, the control unit 200 does not send a busy notification to the first terminal apparatus 10. As a result, when reconnection from the first terminal apparatus 10 to the multifunction peripheral 20 is performed, the reconnection process is executed between the first terminal apparatus 10 and the multifunction peripheral 20.

[0179] Through the above-described processing, when the multifunction device 20 is remotely operated via the first terminal device 10 and the second terminal device 10, the multifunction device 20 can start the remote operation or send a busy notification based on the state and settings of communication with the terminal device 10.

[0180] As described above, the terminal device and the multifunction peripheral according to this embodiment can appropriately perform remote operation when another user starts remote operation after communication has been interrupted.

[0181] [4. Modifications] The present invention is not limited to the above-described embodiments, and various modifications are possible. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of the gist of the present invention are also included in the technical scope of the present invention.

[0182] Although the above-described embodiments are described separately for convenience of explanation, they may be combined to the extent technically feasible. For example, the second embodiment and the third embodiment may be combined. In this case, when communication with the multifunction peripheral is resumed, the terminal device can execute processing according to the state of the multifunction peripheral. Furthermore, the terminal device and the multifunction peripheral can respond appropriately when remotely operated by multiple users.

[0183] In addition, the programs that run on each device in the embodiments are programs that control the CPU, etc. (programs that make a computer function) so as to realize the functions of the above-described embodiments. Information handled by these devices is temporarily stored in a temporary storage device (e.g., RAM) during processing, and then stored in various storage devices such as ROMs (Read Only Memories) and HDDs, and is read, modified, and written by the CPU as needed.

[0184] Here, the recording medium for storing the program may be any of semiconductor media (e.g., ROM, non-volatile memory card, etc.), optical recording media / magneto-optical recording media (e.g., DVD (Digital Versatile Disc), MO (Magneto Optical Disc), MD (Mini Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disc), etc.), magnetic recording media (e.g., magnetic tape, flexible disk, etc.), etc. Furthermore, not only are the functions of the above-described embodiments realized by executing the loaded program, but the functions of the present invention may also be realized by processing in cooperation with an operating system or other application programs, etc., based on instructions from the program.

[0185] Furthermore, when distributing the program on the market, the program can be stored in a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is of course included in the present invention. [Explanation of symbols]

[0186] 1 System 10 Terminal Equipment 100 control section 140 Display section 150 Operation section 160 Storage section 162 Operation App 164 Multifunction device information storage area 190 Communications Department 20 Multifunction device 200 control section 202 Image processing section 220 Image input unit 230 Image forming unit 240 Display section 250 Operation section 260 Storage section 262 Processing Data Storage Area 264 Job detail information storage area 290 Communications Department

Claims

1. The device includes a control unit, a storage unit, and a communication unit, The control unit Establishing a connection with a terminal device via the communication unit using QUIC (Quick UDP Internet Connections); storing, in the storage unit, identification information for identifying the connection and key information used to encrypt transmission / reception information transmitted and received to / from the terminal device; The communication between the terminal device and the transmitted / received information is performed using HTTP (Hypertext Transfer Protocol) / 3; An image forming apparatus characterized in that, when the connection with the terminal device is disconnected, the image forming apparatus establishes a connection with the terminal device using the identification information and resumes communication of the transmitted and received information with the terminal device.

2. 2. The image forming apparatus according to claim 1, wherein the key information is information of a common key used in communication using a TLS (Transport Layer Security) protocol.

3. The control unit storing the identification information, key information, and transmission / reception information received from the terminal device in association with each other; 3. The image forming apparatus according to claim 1, wherein when communication with the terminal device is resumed, communication of the transmission / reception information associated with the identification information is resumed.

4. The control unit setting a time for accepting reconnection from the terminal device when the connection with the terminal device is disconnected; 2. The image forming apparatus according to claim 1, wherein when the terminal device attempts to reconnect after a predetermined time has elapsed, the image forming apparatus refuses to resume communication of transmitted / received information with the terminal device.

5. the communication unit communicates with a plurality of terminal devices including the terminal device; The image forming apparatus according to any one of claims 1 to 4, characterized in that, when a connection with a first terminal device is established and then the connection with the first terminal device is disconnected, the control device further establishes a connection with a second terminal device, communicates transmission and reception information with the second terminal device, re-establishes a connection with the second terminal device, and communicates transmission and reception information with the first terminal device.

6. If a connection with the first terminal device is established and then the connection with the first terminal device is disconnected, the communication of the transmission and reception information is performed after a connection with the second terminal device is established; 6. The image forming apparatus according to claim 5, wherein the control unit notifies the first terminal apparatus of the user's usage status of the image forming apparatus when the first terminal apparatus requests establishment of a reconnection.

7. The device includes a control unit, a storage unit, and a communication unit, the control unit establishes a connection with the image forming apparatus via a communicator using QUIC (Quick UDP Internet Connections); the storage unit stores identification information for identifying the connection and key information used to encrypt information transmitted and received between the image forming apparatus and the storage unit; Communication of information sent and received with the image forming device is performed using HTTP (Hypertext Transfer Protocol) / 3, A terminal device characterized in that, when the connection with the image forming apparatus is cut off, the terminal device reconnects to the image forming apparatus using identification information and resumes communication of transmitted and received information with the image forming apparatus.

8. 8. The terminal device according to claim 7, wherein the control unit controls the image forming device to display a message according to the state of the image forming device when communication of transmitted / received information with the image forming device is resumed.

9. A control method for an image forming apparatus, comprising: Establishing a connection with a terminal device using QUIC (Quick UDP Internet Connections); a step of storing identification information for identifying a connection and key information used to encrypt transmission / reception information transmitted and received between the terminal device; a step of communicating transmission and reception information with a terminal device using HTTP (Hypertext Transfer Protocol) / 3; When the connection with the terminal device is disconnected, reconnecting to the terminal device using the identification information and resuming communication of transmitted and received information with the terminal device; 10. A method for controlling an image forming apparatus, comprising:

10. A method for controlling a terminal device, comprising: establishing a connection with the image forming apparatus using QUIC (Quick UDP Internet Connections); a step of storing identification information for identifying a connection and key information used to encrypt information transmitted and received between the image forming apparatus and the image forming device; A step of communicating information to and from the image forming apparatus using HTTP (Hypertext Transfer Protocol) / 3; When the connection with the image forming apparatus is cut off, reconnecting to the image forming apparatus using the identification information and resuming communication of transmitted and received information with the image forming apparatus; A method for controlling a terminal device, comprising:

Images