Image forming apparatus, terminal device, control method of image forming apparatus, and control method of terminal device

Abstract

To provide an image formation apparatus and the like which can securely resume communication with a terminal apparatus.SOLUTION: An image formation apparatus comprises: a control unit; a storage unit; and a communication unit. The control unit establishes connection with a terminal apparatus via the communication unit with QUIC, stores identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted to / received from the terminal apparatus in the storage unit, performs communication of the transmission / reception information with the terminal apparatus with HTTP / 3, establishes connection with the terminal apparatus by using the identification information and the key information when the connection with the terminal apparatus is cut, and resumes the communication of the transmission / reception information with the terminal apparatus.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image forming apparatus and the like.

Background Art

[0002] Conventionally, a remote operation technique for operating a so-called multi-functional device equipped with a copy function, a scan function, a document printing function, etc. from a terminal device such as a mobile terminal has been used.

[0003] In addition, techniques for improving usability during remote operation have also been proposed. For example, when the wireless communication with the mobile terminal is interrupted during the reception of the processing target data transferred from the mobile terminal, the interruption job information including the unique information (phone number) of the mobile terminal is stored, and if the phone number of the mobile terminal connected after the interruption is the same as the phone number included in the interruption job information, an image processing apparatus that resumes the wireless communication with that mobile terminal has been proposed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to resume the communication between the multi-functional device and the terminal device using the technique described in Patent Document 1, it is necessary to transmit the mobile terminal specific information such as the phone number and the printing information (the storage location of the print data and the print settings) from the terminal device to the multi-functional device and hold them on the multi-functional device side. Therefore, there is a problem with respect to the risk that personal information such as phone numbers and printing information may be leaked due to malicious attacks (for example, eavesdropping on communication). In addition, the mobile terminal specific information such as the phone number may be known to a third party, and there is also a concern about the leakage of print data due to impersonation.

[0006] In view of the above-described problems, an object of the present disclosure is to provide an image forming apparatus or the like that can securely resume communication with a terminal device.

Means for Solving the Problems

[0007] In order to solve the above-described problems, an image forming apparatus of the present disclosure includes a control unit, a storage unit, and a communication unit. The control unit establishes a connection with a terminal device via the communication unit using QUIC, stores in the storage unit identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the terminal device, performs communication of the transmission / reception information with the terminal device using HTTP / 3, and when the connection with the terminal device is disconnected, establishes a connection with the terminal device using the identification information and the key information and resumes communication of the transmission / reception information with the terminal device.

[0008] A terminal device of the present disclosure includes a control unit, a storage unit, and a communication unit. The control unit establishes a connection with an image forming apparatus via the communication unit using QUIC, stores in the storage unit identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the image forming apparatus, performs communication of the transmission / reception information with the image forming apparatus using HTTP / 3, and when the connection with the image forming apparatus is disconnected, reconnects to the image forming apparatus using the identification information and the key information and resumes communication of the transmission / reception information with the image forming apparatus.

[0009] A control method for an image forming apparatus according to the present disclosure includes steps of establishing a connection with a terminal device using QUIC, storing identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the terminal device, performing communication of the transmission / reception information with the terminal device using HTTP / 3, and when the connection with the terminal device is disconnected, establishing a connection with the terminal device using the identification information and the key information and resuming communication of the transmission / reception information with the terminal device.

[0010] The control method of the terminal device according to the present disclosure includes a step of establishing a connection with an image forming apparatus by QUIC, a step of storing identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the image forming apparatus, a step of performing communication of the transmission / reception information with the image forming apparatus by HTTP / 3, and a step of, when the connection with the image forming apparatus is disconnected, performing reconnection to the image forming apparatus using the identification information and the key information and resuming communication of the transmission / reception information with the image forming apparatus.

Effect of the Invention

[0011] According to the present disclosure, it is possible to provide an image forming apparatus or the like that can securely resume communication with a terminal device.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0013] Hereinafter, with reference to the drawings, one embodiment for implementing the present disclosure will be described. Note that the following embodiments are examples for explaining the present disclosure, and the technical scope of the invention described in the claims is not limited to the following description.

[0014] [1. First Embodiment] [1.1 Overall Configuration] Referring to FIG. 1, the overall configuration of the system 1 in this embodiment will be described. As shown in FIG. 1, the system 1 includes a terminal device 10 and a multifunction device 20. The terminal device 10 and the multifunction device 20 can be connected via, for example, a wireless LAN (Local Area Network). The terminal device 10 and the multifunction device 20 may be directly connected or may be connected via a wireless access point.

[0015] The terminal device 10 is an information processing device carried by a user. The terminal device 10 is, for example, a device such as a smartphone, a tablet, or a PC (Personal Computer).

[0016] The multifunction device 20 is an information processing device having functions such as a copying function, a scanning function, and a document printing function, and is also called an MFP (Multi-Function Printer / Peripheral). The multifunction device 20 may be an image processing device that processes images, or may be an image forming device provided with means for forming an image on a recording sheet using means such as an electrophotographic method.

[0017] In this embodiment, when an operation or a job (print job or transmission job) from the terminal device 10 to the multifunction device 20 is executed, a connection using QUIC (Quick UDP Internet Connections) is made as the communication protocol (transport protocol) used for the connection between the terminal device 10 and the multifunction device 20. QUIC is a protocol that uses UDP (User Datagram Protocol) to establish a bidirectional connection between two devices. Two devices that have established a connection by QUIC transmit a UDP packet including a QUIC packet in the UDP payload. The QUIC packet is composed of a QUIC packet header and a payload including predetermined information.

[0018] Although UDP is a connectionless protocol, since the QUIC packet header includes identification information (e.g., Connection ID) used for connection identification, two devices that have established a connection using QUIC can identify the connection (connection) based on the identification information. By identifying the connection based on the identification information, even if the IP (Internet Protocol) address or port number of the device establishing the connection changes, the two devices establishing the connection can maintain the connection between the devices.

[0019] Also, the payload within the QUIC packet is encrypted. As key information used to encrypt or decrypt the encrypted payload, a common key shared by the devices that have established a connection using QUIC is used. The common key is generated in the two devices establishing the connection when the process of establishing a connection using QUIC is executed.

[0020] Also, the terminal device 10 and the multifunction device 20 transfer (send and receive) information (transmission and reception information) related to remote operations and job execution. For the transfer of the transmission and reception information, for example, HTTP / 3, which is a type of the HTTP (Hypertext Transfer Protocol), is used.

[0021] The transmission and reception information includes, for example, operation information and job information transmitted from the terminal device 10 to the multifunction device 20, and information (UI data) of the operation screen transmitted from the multifunction device 20 to the terminal device 10. The operation information is information indicating the content of the operation input by the user. Also, the job information is information indicating the content of the job. The job information may include data (processing data) that is subjected to predetermined processing such as printing or transmission by the multifunction device 20 during job execution. Note that the terminal device 10 may transmit only the job information to the multifunction device 20, or may transmit the operation information and the processing data. Also, the information of the operation screen is information used to display a screen for receiving an operation by the user. Note that the transmission and reception information may be any information as long as it is information for causing the multifunction device 20 to execute a predetermined job.

[0022] The terminal device 10 operates the multifunction device 20 by transmitting transmission and reception information such as operation information and job information to the multifunction device 20. Note that the operation in which the terminal device 10 communicates with the multifunction device 20 is also referred to as a remote operation.

[0023] [1.2 Functional Configuration] [1.2.1 Terminal Device] The functional configuration of the terminal device 10 will be described with reference to FIG. 2. As shown in FIG. 2, the terminal device 10 includes a control unit 100, a display unit 140, an operation unit 150, a storage unit 160, and a communication unit 190.

[0024] The control unit 100 controls the entire terminal device 10. The control unit 100 realizes various functions by reading and executing various programs stored in the storage unit 160, and is composed of one or more arithmetic units (for example, a CPU (Central Processing Unit)). Further, the control unit 100 may be configured as a SoC (System on a Chip) having a plurality of functions among the functions described below.

[0025] The display unit 140 displays various information. The display unit 140 is composed of a display device such as, for example, an LCD (Liquid crystal display), an organic EL (electro-luminescence) display, a micro LED (Light Emitting Diode) display, or the like.

[0026] The operation unit 150 receives operations of a user who uses the terminal device 10. The operation unit 150 is composed of an input device such as a touch sensor. The method of detecting an input in the touch sensor may be a general detection method such as, for example, a resistive film method, an infrared method, an electromagnetic induction method, or a capacitance method. Note that the terminal device 10 may be equipped with a touch panel in which the display unit 140 and the operation unit 150 are integrally formed.

[0027] The storage unit 160 stores various programs and various data necessary for the operation of the terminal device 10. The storage unit 160 is constituted by, for example, a storage device such as an SSD (Solid State Drive) which is a semiconductor memory, an HDD (Hard Disk Drive), or the like.

[0028] The storage unit 160 stores the operation application 162 and secures a multifunction device information storage area 164 as a storage area.

[0029] The operation application 162 is an application for causing the control unit 100 to realize a function for remotely operating the multifunction device 20. For example, the operation application 162 causes the control unit 100 to realize a function of displaying a screen used for remote operation on the display unit 140, or realizes a function related to connection and information communication with the multifunction device 20.

[0030] The multifunction device information storage area 164 stores information (multifunction device information) regarding the multifunction device 20. The multifunction device information includes, for example, as shown in FIG. 3, the name of the multifunction device 20 (for example, "xxxx"), the address of the multifunction device 20 (for example, "192.168.0.11"), a terminal device Connection ID (for example, "9dc9001122334455") and a multifunction device Connection ID (for example, "8ba9001122334455") for identifying the connection with the multifunction device 20, and key information (for example, "******") used for encrypting information transmitted and received to and from the multifunction device 20.

[0031] The address is information used for identifying the multifunction device 20 connected to the network to which the terminal device 10 is connected, and is used for connecting to the multifunction device 20. The address is, for example, an IP address.

[0032] The Connection ID is information (identification information) used to identify the connection between the terminal device 10 and the multifunction device 20. In the present embodiment, a terminal device Connection ID and a multifunction device Connection ID are stored as the Connection ID. The terminal device Connection ID is used when the multifunction device 20 communicates with the terminal device 10. Also, the multifunction device Connection ID is used when the terminal device 10 communicates with the multifunction device 20.

[0033] Also, the key information is, for example, information on a common key (TLS key information) used when communicating using the TLS (Transport Layer Security) protocol.

[0034] Note that the multifunction device information may include information such as the port number of the multifunction device 20 in addition to the information described above.

[0035] The communication unit 190 communicates with external devices such as the multifunction device 20. The communication unit 190 is configured by, for example, a communication device having a function of connecting to a wireless LAN by a method compliant with the IEEE802.11b / g / n standard. For example, the communication unit 190 is configured by a NIC (Network Interface Card) or a communication module capable of connecting to an LTE (Long Term Evolution) / LTE-A (LTE-Advanced) / LAA (License-Assisted Access using LTE) / 5G line.

[0036] [1.2.2 Multifunction Device] The functional configuration of the multifunction device 20 will be described with reference to FIG. 4. As shown in FIG. 4, the multifunction device 20 includes a control unit 200, an image input unit 220, an image forming unit 230, a display unit 240, an operation unit 250, a storage unit 260, and a communication unit 290.

[0037] The control unit 200 controls the entire multifunction device 20. The control unit 200 realizes various functions by reading and executing various programs stored in the storage unit 260, and is composed of one or more arithmetic units (for example, CPU). Further, the control unit 200 may be configured as a SoC having a plurality of functions among the functions described below.

[0038] By executing the program stored in the storage unit 260, the control unit 200 functions as an image processing unit 202. The image processing unit 202 performs various image processes on the image data input by the image input unit 220. For example, the image processing unit 202 performs sharpness processing or color conversion processing on the image data.

[0039] The image input unit 220 reads a document and inputs the data of the read image. For example, the image input unit 220 is composed of a scanning device or the like equipped with a device that converts optical information such as a CIS (Contact Image Sensor) or a CCD (Charge Coupled Device) into an electrical signal, and reads a document placed on the mounting table of the multifunction device 20.

[0040] The image forming unit 230 forms (prints) an image on a recording medium such as recording paper. The image forming unit 230 is composed of, for example, a laser printer or the like using an electrophotographic method.

[0041] The display unit 240 displays various information. The display unit 240 is composed of a display device such as an LCD, an organic EL display, or a micro LED display.

[0042] The operation unit 250 receives the operations of the user who uses the multifunction device 20. The operation unit 250 is composed of input devices such as hardware keys (physical keys, numeric keys) and touch sensors. Note that the multifunction device 20 may be equipped with a touch panel in which the display unit 240 and the operation unit 250 are integrally formed.

[0043] The storage unit 260 stores various programs and various data necessary for the operation of the multifunction device 20. The storage unit 260 is constituted by, for example, a storage device such as an SSD which is a semiconductor memory or an HDD.

[0044] The storage unit 260 secures a processing data storage area 262 and a job detailed information storage area 264 as storage areas.

[0045] The processing data storage area 262 stores data processed by the multifunction device 20. The processing data is, for example, data representing an image formed by the image forming unit 230 or data transmitted to another device via the communication unit 290.

[0046] The job detailed information storage area 264 stores detailed information (job detailed information) regarding the jobs executed by the multifunction device 20. The job detailed information includes, for example, as shown in FIG. 5, a job ID (for example, "25") for identifying a job, an address of the terminal device 10 (for example, "192.168.0.101"), a port number for communicating with the terminal device 10 (for example, "19999"), a terminal device Connection ID (for example, "9dc9001122334455") and a multifunction device Connection ID (for example, "8ba9001122334455") used for communication with the terminal device 10, key information (for example, "******") used for encrypting information transmitted to the terminal device 10, operation information and job information which are transmission / reception information received from the terminal device 10 (for example, "simple copy, paper: A4, color mode: color, processing data: 20210801170000"), and a job status (for example, "not executed").

[0047] The processing data may be stored in the job detailed information or may be stored in the processing data storage area 262. When the processing data is stored in the processing data storage area 262, information (for example, data name) for specifying the processing data stored in the processing data storage area 262 may be stored in the job detailed information.

[0048] The address is, for example, an IP (Internet Protocol) address. The job status stores information indicating the progress of a job, such as "not executed", "executing", "execution completed", etc. Further, the job detailed information may include information other than the above-described information.

[0049] The communication unit 290 communicates with an external device such as the terminal device 10. The communication unit 290 is composed of a communication device (communication module) similar to the communication unit 190.

[0050] [1.3 Process Flow] [1.3.1 Outline of the Process] With reference to FIG. 6, the outline of the processes executed by the terminal device 10 and the multifunction device 20 will be described. First, the terminal device 10 receives a request to start remote operation of the multifunction device 20 (S1000). Subsequently, a handshake (QUIC Handshake) using the QUIC protocol is executed between the terminal device 10 and the multifunction device 20 (S1002). Thereby, a session between the terminal device 10 - multifunction device 20 is started. In S1002, the terminal device 10 and the multifunction device 20 execute the following processes. (1) Generation and Transmission of Connection ID The terminal device 10 generates a terminal device Connection ID and transmits it to the multifunction device 20. The multifunction device 20 generates a multifunction device Connection ID and transmits it to the terminal device 10. Thereby, the Connection ID is shared between the terminal device 10 and the multifunction device 20. (2) Determination of the TLS Common Key The terminal device 10 and the multifunction device 20 generate a common key respectively using the information transmitted and received during the handshake. When the terminal device 10 and the multifunction device 20 confirm that the same common key has been generated, the common key is determined as the TLS key information. For example, the terminal device 10 and the multifunction device 20 can determine that the same common key has been generated when they transmit data encrypted using the common key and can decrypt the data received from the other device using the common key.

[0051] When a handshake is performed, the connection between the terminal device 10 and the multifunction device 20 is established, and a session is started (established). At the start of the session, the multifunction device 20 stores the Connection ID and TLS key information used in the QUIC protocol (S1004). Similarly, the terminal device 10 stores the Connection ID and TLS key information used in the QUIC protocol at the start of the session (S1006).

[0052] Subsequently, the terminal device 10 and the multifunction device 20 communicate the transmission and reception information (Application Data) using the Connection ID and TLS key information (S1008).

[0053] For example, the multifunction device 20 encrypts the information on the operation screen (UI data) using the TLS key information. Further, the multifunction device 20 transmits to the terminal device 10 the encrypted information and a Connection ID (for example, the terminal device Connection ID received from the terminal device 10) for identifying the connection with the terminal device 10. The terminal device 10 decrypts the information received from the multifunction device 20 using the TLS key information corresponding to the multifunction device 20, displays an operation screen based on the received information on the display unit 140, and accepts an operation input from the user.

[0054] Also, the terminal device 10 encrypts operation information and job information using the TLS key information and transmits them to the multifunction device 20. The multifunction device 20 decrypts the operation information and job information using the TLS key information corresponding to the terminal device 10, and stores the decrypted operation information and job information in association with the Connection ID and TLS key information related to the communication with the terminal device 10.

[0055] Subsequently, a case where the connection between the terminal device 10 and the multifunction device 20 is disconnected, the session is interrupted, and the communication is interrupted will be described. The disconnection of the connection occurs, for example, due to a poor wireless communication network on the terminal device 10 side, or a change in the IP address of the terminal device 10 accompanying a change in the base station to which the terminal device 10 is connected.

[0056] When the communication between the terminal device 10 and the multifunction device 20 is interrupted, the multifunction device 20 sets a communication timeout time (S1010). The communication timeout time is the time to accept reconnection from the terminal device 10. The multifunction device 20 maintains a state (communication-acceptable state) in which it can accept reconnection from the terminal device 10 until the communication timeout time elapses. On the other hand, after the communication timeout time has elapsed, the multifunction device 20 rejects reconnection from the terminal device 10 (does not reconnect to the terminal device 10) to suppress consumption of resources (e.g., memory) of the multifunction device 20 and avoid a situation where another user cannot operate the multifunction device 20 for a long time.

[0057] Also, when the change of the IP address is completed or when the network is restored, the terminal device 10 reconnects to the multifunction device 20 (S1012). For example, the terminal device 10 makes a reconnection request by transmitting the multifunction device Connection ID and information obtained by encrypting predetermined information using the TLS key information to the multifunction device 20 (S1014).

[0058] The multifunction device 20 identifies a job in the reconnection waiting state from the Connection ID received from the terminal device 10 and the TLS key information of the terminal device 10 (S1016). Also, the multifunction device 20 executes a reconnection process with the terminal device 10 (S1018). For example, as the reconnection process, the multifunction device 20 confirms that it can communicate normally with the terminal device 10, transmits information indicating operation information and job information that needs to be transmitted to the terminal device 10 next, or transmits information on an operation screen for inputting the next operation to the terminal device 10. By reconnecting the terminal device 10 and the multifunction device 20 through the reconnection process and restarting the session, the communication between the terminal device 10 and the multifunction device 20 is restarted.

[0059] Subsequently, the terminal device 10 and the multifunction device 20 communicate (resume) the transmission and reception information (Application Data) in the same manner as in S1008 using the Connection ID and the TLS key information (S1020). As a result, remote operations become possible again on the terminal device 10 (S1022).

[0060] In this way, after the connection such as wireless on the terminal device 10 side is once disconnected, when the network connection on the terminal device 10 side is restored, the interrupted communication can be performed again. As a result, the terminal device 10 can resume the communication for transmitting the operation information and job information that could not be transmitted to the multifunction device 20.

[0061] When resuming communication with the multifunction device 20, the terminal device 10 resumes communication according to the QUIC protocol using the Connection ID and the TLS key information used in the QUIC protocol. Since the terminal device 10 does not need to perform a process of re - establishing a session with the multifunction device 20, such as a handshake, it can efficiently reconnect to the multifunction device 20.

[0062] Also, on the side of the multifunction device 20, although the communication from the terminal device 10 is interrupted during communication, it stores the Connection ID and the TLS key information used in the QUIC protocol and maintains a state where it can receive communication. Therefore, the multifunction device 20 can resume communication with the terminal device 10 using the Connection ID and the TLS key information. Also, by using the QUIC protocol, the multifunction device 20 can connect to the terminal device 10 at the timing when the communication from the terminal device 10 is resumed, and can resume the session with the terminal device 10 without specially performing a session resumption process. As a result, the multifunction device 20 can efficiently communicate with the terminal device 10 starting from where the interrupted communication left off.

[0063] Thus, when resuming the session between the terminal device 10 and the multifunction device 20, there is no need to perform the process for resuming the session (session process) again between the terminal device 10 and the multifunction device 20, and information transmission and reception (data process) can be performed from where the interrupted communication left off.

[0064] [1.3.2 Processing of Terminal Device] Next, the processing executed by the terminal device 10 will be described with reference to FIG. 7. The processing shown in FIG. 7 is executed by the control unit 100 that has read the operation application 162. It is assumed that the multifunction device information storage area 164 stores in advance multifunction device information including the multifunction device name and address information of the multifunction device 20.

[0065] First, the control unit 100 receives a remote operation start request (step S100). For example, the control unit 100 displays on the display unit 140 a screen (multifunction device selection screen) that lists the multifunction device names included in the multifunction device information stored in the multifunction device information storage area 164. When the user selects one multifunction device 20 and performs an operation to start the connection, the control unit 100 receives a request to start the remote operation of the selected multifunction device 20.

[0066] Next, the control unit 100 executes a handshake with the multifunction device 20 via the communication unit 190 using the QUIC protocol (step S102). For example, the control unit 100 reads from the multifunction device information storage area 164 the multifunction device information corresponding to the multifunction device 20 selected in step S100. Further, the control unit 100 transmits a packet for performing the handshake to the address included in the read multifunction device information.

[0067] At this time, the control unit 100 generates a terminal device Connection ID and transmits it to the multi-function device 20, and receives a multi-function device Connection ID from the multi-function device 20. Further, the control unit 100 generates a common key using the information transmitted and received when performing the handshake. Also, if the common key generated by the control unit 100 is the same as the common key generated by the multi-function device 20, the control unit 100 determines the common key as the TLS key information. For example, when the control unit 100 receives encrypted information from the multi-function device 20 and can decrypt the information with the generated common key, it can determine that the generated common key is the same as the common key generated by the multi-function device 20.

[0068] Subsequently, the control unit 100 stores the terminal device Connection ID, the multi-function device Connection ID, and the TLS key information in the multi-function device information corresponding to the multi-function device 20 selected in step S100 (step S104).

[0069] Subsequently, the control unit 100 starts a remote operation (step S106). For example, the control unit 100 displays a screen (operation screen) for performing a remote operation on the display unit 140.

[0070] Subsequently, the control unit 100 transmits operation information and job information via the communication unit 190 (step S108). For example, the control unit 100 encrypts the operation information indicating the operation input by the user using the TLS key information corresponding to the connected multi-function device 20. Further, the control unit 100 transmits a packet including the multi-function device Connection ID corresponding to the connected multi-function device 20 and the encrypted operation information to the multi-function device 20.

[0071] Note that the control unit 100 may encrypt the job information based on the user's operation using the TLS key information, and transmit a packet including the multi-function device Connection ID corresponding to the connected multi-function device 20 and the encrypted operation information to the multi-function device 20. Also, the control unit 100 may read the file selected by the user and generate data (processing data) indicating the content of the file as the job information.

[0072] Also, when the control unit 100 receives information from the multifunction device 20, it decrypts the received information using the TLS key information determined in step S102. When the control unit 100 receives the information of the operation screen from the multifunction device 20, it may display the operation screen based on the information on the display unit 140.

[0073] Subsequently, the control unit 100 determines whether the connection with the multifunction device 20 has been disconnected (step S110). For example, the control unit 100 monitors the communication unit 190 and detects whether the connection with the multifunction device 20 has been disconnected. Note that when the connection between the control unit 100 and the multifunction device 20 is disconnected, the control unit 100 may display a message indicating that the connection with the multifunction device 20 has been disconnected on the display unit 140.

[0074] When the connection between the control unit 100 and the multifunction device 20 is disconnected, the control unit 100 periodically determines whether the network has been restored (step S110; Yes → step S112). For example, when the connection to the base station becomes possible or the IP address is determined, the control unit 100 determines that the network has been restored. Note that when the network is not restored, the control unit 100 repeatedly executes the process of step S112 (step S112; No).

[0075] When the network is restored, the control unit 100 reconnects to the multifunction device 20 via the communication unit 190 using the multifunction device Connection ID and TLS key information stored in step S104 (step S112; Yes → step S114). Thereby, the control unit 100 makes a reconnection request to the multifunction device 20. Note that the control unit 100 may display a message indicating that the remote connection to the multifunction device 20 is being resumed on the display unit 140.

[0076] Furthermore, the control unit 100 executes reconnection processing (step S116). For example, the control unit 100 receives information indicating operation information and job information that needs to be transmitted next from the multifunction device 20, or receives information on an operation screen for inputting the next operation, and displays an operation screen based on the information on the display unit 140. Also, when the reconnection processing is completed, the control unit 100 may display a message on the display unit 140 indicating that the communication with the multifunction device 20 has resumed or that the remote operation has resumed.

[0077] Note that when the control unit 100 receives information indicating rejection of reconnection from the multifunction device 20 in step S116, the control unit 100 may display a message on the display unit 140 indicating that reconnection with the multifunction device 20 could not be established, or may return to step S100. Also, when the control unit 100 determines in step S110 that the connection with the multifunction device 20 has not been disconnected, the control unit 100 omits the processing from step S112 to step S116 (step S110; No).

[0078] Subsequently, the control unit 100 determines whether a job has been executed (step S118). For example, when the control unit 100 receives information from the multifunction device 20 indicating that the execution of a job has been completed, the control unit 100 determines that the job has been executed.

[0079] When the control unit 100 determines that a job has been executed, the control unit 100 ends the remote operation (step S118; Yes → step S120). At this time, the control unit 100 may display a message indicating that the job has been executed and the multifunction device selection screen on the display unit 140. Note that when the control unit 100 determines that the job has not been executed, the control unit 100 returns to step S108 (step S118; No → step S108).

[0080] [1.3.3 Processing of Multifunction Device] Subsequently, the processing executed by the multifunction device 20 will be described with reference to FIGS. 8 and 9. The processing shown in FIGS. 8 and 9 is executed by the control unit 200 that reads out a program stored in the storage unit 260.

[0081] First, the process shown in FIG. 8 will be described. When the control unit 200 receives information (packet) for performing a handshake from the terminal device 10 via the communication unit 290, the control unit 200 executes a handshake with the terminal device 10 according to the QUIC protocol (step S150). For example, the control unit 200 receives a terminal device Connection ID from the terminal device 10, or generates a multifunction device Connection ID and transmits it to the terminal device 10. Further, the control unit 200 generates a common key using the information transmitted and received during the handshake, and if the generated common key is the same as the common key generated by the terminal device 10, the control unit 200 determines the common key as TLS key information.

[0082] Note that when the control unit 200 executes the process in step S150 and is displaying a screen for receiving an operation via the operation unit 250 on the display unit 240, the control unit 200 may also display a message indicating that it is executing a process for starting a remote operation.

[0083] Subsequently, the control unit 200 stores the terminal device Connection ID, the multifunction device Connection ID, and the determined TLS key information (step S152). For example, the control unit 200 issues a job ID and stores job detail information including the job ID, the terminal device Connection ID, the multifunction device Connection ID, and the TLS key information in the job detail information storage area 264. Further, the control unit 200 stores the address and port number of the terminal device 10 that has performed the handshake in the job detail information, and sets the job status of the job detail information to "not executed".

[0084] Subsequently, the control unit 200 starts accepting a remote operation with the terminal device 10 (step S154). At this time, the control unit 200 may stop accepting an input of an operation via the operation unit 250, or may display a screen (lock screen) indicating that a remote operation is being performed on the display unit 240.

[0085] Next, the control unit 200 receives and stores operation information and job information from the terminal device 10 (step S156). For example, when the control unit 200 receives information from the terminal device 10, the received information is decrypted using the TLS key information corresponding to the terminal device 10. At this time, when the control unit 200 receives operation information or job information from the terminal device 10, the operation information and job information received from the terminal device 10 are stored in the job detail information corresponding to the terminal device 10.

[0086] Also, when the control unit 200 receives processing data from the terminal device 10, the control unit 200 may store the processing data in the processing data storage area 262 and store the data name of the processing data in the job detail information.

[0087] In addition, the control unit 200 may transmit information (UI data) of an operation screen for inputting the next operation to the terminal device 10. For example, the control unit 200 encrypts the information of the operation screen using the TLS key information corresponding to the connected terminal device 10. Further, the control unit 200 transmits a packet including the terminal device Connection ID corresponding to the connected terminal device 10 and the encrypted operation screen information to the terminal device 10.

[0088] Subsequently, the control unit 200 determines whether the connection with the terminal device 10 has been disconnected (step S158). For example, the control unit 200 monitors the communication unit 290 and detects that the connection with the multifunction device 20 has been disconnected.

[0089] When the connection between the control unit 200 and the terminal device 10 is disconnected, the control unit 200 interrupts the reception of remote operations from the terminal device 10 (step S158; Yes → step S160).

[0090] Further, the control unit 200 sets a communication timeout period for the terminal device 10 whose connection has been disconnected (step S162). For example, the control unit 200 may store the communication timeout period in the storage unit 260 in association with the job information detailed information in which the terminal device Connection ID of the connected terminal device 10 is stored. Also, the communication timeout period may be set in advance, or may be settable by an administrator or the like of the multifunction machine 20.

[0091] At this time, the control unit 200 may display on the display unit 240 a screen for receiving an operation via the operation unit 250, and may also display a message indicating that the remote operation has been interrupted.

[0092] Subsequently, the control unit 200 executes a process corresponding to reconnection from the terminal device 10 (reconnection corresponding process) (step S164). The reconnection corresponding process will be described later. Also, the control unit 200 resumes the connection with the terminal device 10 by executing the reconnection corresponding process.

[0093] Subsequently, the control unit 200 determines whether a job based on the operation information and job information stored in step S156 is executable (step S166). If the control unit 200 determines that the job is not executable, it returns to step S156 (step S166; No → step S156).

[0094] On the other hand, if the control unit 200 determines that the job is executable, it executes the job based on the job information stored in step S156 (step S166; Yes → step S168). At this time, the control unit 200 may update the job status of the job detailed information corresponding to the job to be executed to information such as "executing" or "execution completed" according to the progress of the job. Also, when the execution of the job is completed, the control unit 200 may transmit information indicating that the execution of the job has been completed to the terminal device 10.

[0095] Next, the control unit 200 ends the reception of the remote operation (step S170). For example, the control unit 200 disconnects the connection with the terminal device 10 or displays on the display unit 240 a screen for receiving an operation from the operation unit 250.

[0096] Next, with reference to FIG. 9, the reconnection support process will be described. The control unit 200 determines whether reconnection has been made from the terminal device 10 (step S180). If the control unit 200 determines that reconnection has not been made from the terminal device 10, the process of step S180 is repeatedly executed (step S180; No).

[0097] If the control unit 200 determines that reconnection has been made from the terminal device 10, the control unit 200 identifies the job in the reconnection waiting state from the Connection ID received from the terminal device 10 and the TLS key information of the terminal device 10 (step S180; Yes → step S182). For example, the control unit 200 reads out job detail information including the same multifunction device Connection ID as the multifunction device Connection ID received from the terminal device 10 from the job detail information storage area 264. Further, the control unit 200 decrypts the information received from the terminal device 10 using the TLS key information included in the read job detail information. If the decryption is successful, the job based on the operation information and job information included in the read job detail information is identified as the job in the reconnection waiting state.

[0098] Next, the control unit 200 determines whether or not the communication timeout time set for the terminal device 10 that has reconnected has elapsed (step S184). If the communication timeout time has elapsed, the control unit 200 rejects the reconnection from the terminal device 10 and ends the process shown in FIG. 8 (step S184; No → step S190). In this case, the control unit 200 does not perform reconnection with the terminal device 10 as the connection destination and ends the remote operation with the terminal device 10. Note that the control unit 200 may transmit information indicating rejection of reconnection to the terminal device 10. When the communication timeout time has elapsed, the terminal device 10 can establish a new connection by newly performing a handshake with the multifunction device 20 and execute a new remote operation.

[0099] On the other hand, if the communication timeout time has not elapsed, the control unit 200 executes a reconnection process (step S184; Yes → step S186). For example, the control unit 200 checks whether it is possible to communicate with the terminal device 10. Also, the control unit 200 stores (updates) the new IP address and port number received from the terminal device 10 in step S180 in the job detail information read in step S182. Thereby, even if the IP address and port number of the terminal device 10 are changed, the control unit 200 can use the changed IP address and port number.

[0100] Further, based on the content (job information) of the job specified in step S182, the control unit 200 may transmit operation information and job information that need to be transmitted next to the terminal device 10, or may transmit information on an operation screen for inputting the next operation.

[0101] Also, when executing the reconnection process, the control unit 200 may display, on the display unit 240, a message indicating that a reconnection request is being received from the terminal device 10. Note that when the reconnection process is completed and the remote operation from the terminal device 10 is restarted, the control unit 200 may display, on the display unit 240, a screen (lock screen) indicating that a remote operation is being performed.

[0102] [1.4 Operation Example] Next, with reference to the drawings, the screens displayed on the display unit 140 of the terminal device 10 and the display unit 240 of the multifunction device 20 when the processes in this embodiment are executed will be described.

[0103] FIG. 10 is a diagram showing a screen displayed when communication for remote operation (remote communication) is started. FIG. 10(a) is a diagram showing a multifunction device selection screen W100 displayed on the display unit 140. The multifunction device selection screen W100 includes an area E100 in which the names of the multifunction devices included in the multifunction device information stored in the terminal device 10 are displayed so as to be selectable. The user can start connecting the terminal device 10 to the selected multifunction device 20 by selecting one multifunction device 20 from the area E100 and then selecting the connection button B100.

[0104] FIG. 10(b) is a diagram showing a display screen W110 displayed on the display unit 240. The display screen W110 is displayed, for example, when a handshake is being executed. A message M110 indicating that communication for remote operation has been started is displayed on the display screen W110.

[0105] FIG. 11 is a diagram showing a screen displayed when a remote operation is being performed. FIG. 11(a) is an example of a screen of an operation screen W120 displayed on the display unit 140. Buttons and the like for setting jobs to be executed by the multifunction device 20 are displayed on the operation screen W120. For example, as shown in FIG. 11(a), buttons for selecting jobs to be executed by the multifunction device 20 are displayed. Note that buttons and lists for setting jobs (for example, the number of printed copies, color mode, paper size, destination to which processing data is to be sent, etc.) and for selecting files (processing data) to be transmitted to the multifunction device 20 may also be displayed on the operation screen W120. Operation information corresponding to the operations input by the user via the operation screen W120 and job information based on the user's operations are transmitted to the multifunction device 20.

[0106] FIG. 11(b) is an example screen of the lock screen W130 displayed on the display unit 240. By displaying the lock screen, the user can recognize that operations via the operation unit 250 are not accepted. Note that the lock screen W130 may include an interrupt button B130 for performing an operation via the operation unit 250 when a remote operation is being performed. When the interrupt button B130 is selected, the multifunction device 20 may interrupt the remote operation and enter a state in which an operation via the operation unit 250 can be input.

[0107] FIG. 12 is a diagram showing a screen displayed when the connection between the terminal device 10 and the multifunction device 20 is disconnected and communication (remote communication) is interrupted. FIG. 12(a) is an example display of the display screen W140 displayed on the display unit 140. The display screen W140 includes a message M140 indicating that the connection to the multifunction device 20 has been disconnected. By checking the message M140, the user can recognize that the connection to the multifunction device 20 has been disconnected. Also, by selecting the resume button B140, the user can instruct the terminal device 10 to resume communication with the multifunction device 20.

[0108] FIG. 12(b) is an example display of the display screen W150 displayed on the display unit 240. The display screen W150 includes a message M150 indicating that the execution of a job based on a remote operation has been interrupted due to the interruption of remote communication. By checking the message M150, the user can recognize that the remote operation has been interrupted and that there is a possibility that the remote operation will be resumed.

[0109] FIG. 13 is a diagram showing a screen displayed when remote communication is being resumed. FIG. 13(a) is an example display of the display screen W160 displayed on the display unit 140. The display screen W160 displays a message M160 indicating that the connection between the terminal device 10 and the multifunction device 20 is being resumed.

[0110] FIG. 13(b) is an example display of the display screen W170 displayed on the display unit 240. A message M170 indicating that the connection between the terminal device 10 and the multifunction device 20 is being resumed is displayed on the display screen W170. Note that the message M170 may include a cancel button B170 for canceling the resumption of the connection between the terminal device 10 and the multifunction device 20. When the cancel button B170 is selected, the resumption of the connection between the terminal device 10 and the multifunction device 20 may be canceled, and the operation via the operation unit 250 may be made inputtable.

[0111] FIG. 14 is a diagram showing the screen displayed when the remote communication is resumed. FIG. 14(a) is an example display of the display screen W180 displayed on the display unit 140. A message M180 indicating that the communication between the terminal device 10 and the multifunction device 20 has resumed is displayed on the display screen W180. Also, FIG. 14(b) is an example display of the lock screen W190 displayed on the display unit 240. The lock screen W190 is the same screen as the lock screen W130 in FIG. 11(b).

[0112] As described above, the terminal device and the multifunction device of the present embodiment can resume the session and resume the communication by using the Connection ID and the TLS key information used in the QUIC protocol. In particular, the multifunction device of the present embodiment can resume the communication with the terminal device without using mobile-specific information such as a phone number, so there is no need to transmit, receive, or store mobile-specific information. Therefore, even if the communication is intercepted or the storage unit of the multifunction device is illegally read, the mobile-specific information will not leak. Also, since the multifunction device of the present embodiment does not associate mobile-specific information with job information, it is possible to avoid the leakage of job information due to impersonation.

[0113] Also, the terminal device and the multifunction device of this embodiment encrypt and transmit / receive information. Therefore, it is possible to reduce the risk of leakage of processing data such as print data and information being transmitted and received (operation information and job information). In this way, in this embodiment, without using device-specific information and by using a highly secure communication standard, it is possible to enhance the security regarding the resumption of communication and reduce the risk of information leakage. As a result, the terminal device and the multifunction device of this embodiment can securely resume communication.

[0114] Also, the terminal device and the multifunction device of this embodiment communicate using a communication protocol that can flexibly handle network disconnections and IP address changes, and hold communication information such as Connection ID and TLS key information. Thereby, even when a network problem or an IP address change occurs on the terminal device side, the terminal device and the multifunction device of this embodiment enable continuous operation by the user. In particular, when the connection between the multifunction device of this embodiment and the terminal device is disconnected, the multifunction device reconnects to the terminal device using the Connection ID for identifying the connection and the key information corresponding to the terminal device. The terminal device and the multifunction device of this embodiment do not need to perform special processing for resuming the session in the reconnection after the connection is disconnected, so the reconnection and the resumption of communication can be efficiently executed.

[0115] [2. Second Embodiment] Next, the second embodiment will be described. The second embodiment is an embodiment in which, in addition to the processing described in the first embodiment, when the communication between the terminal device and the multifunction device resumes, the operation of the terminal device is switched according to the state of the multifunction device. This embodiment is obtained by replacing FIG. 7 and FIG. 9 in the first embodiment with FIG. 15 and FIG. 16, respectively. Note that the same reference numerals are given to the same processing, and the description thereof is omitted.

[0116] When a failure occurs in the network on the terminal device 10 side, it is assumed that the communication interruption time between the terminal device 10 and the multifunction device 20 will be long due to the time required for network recovery. When the communication interruption time becomes long, the state of the multifunction device 20 may change during the communication interruption. Therefore, when the communication with the multifunction device 20 resumes, the terminal device 10 according to the present embodiment performs an operation according to the state of the multifunction device 20.

[0117] [2.1 Outline of the operation of the terminal device] In the present embodiment, when the communication between the terminal device 10 and the multifunction device 20 is interrupted, if the state of the multifunction device 20 becomes any one of the following (1) to (6), the terminal device 10 switches the operation according to the state of the multifunction device 20.

[0118] (1) When the operation state of the multifunction device 20 has shifted to the power-saving state The power-saving state is one of the operation states of the multifunction device 20, and is an operation state in which the power supply is restricted compared to the normal operation state. The multifunction device 20 can individually set the time (power-saving setting time) for shifting the operation state to the power-saving state as a setting for remote operation. The multifunction device 20 shifts to the power-saving state when the communication with the terminal device 10 is disconnected during remote operation and the power-saving setting time has elapsed without resuming communication with the terminal device 10. Note that the multifunction device 20 may be able to select whether to hold job detailed information, operation information, and job information according to the setting.

[0119] In addition, when the communication from the terminal device 10 resumes, the multifunction device 20 recovers from the power-saving state to the normal state (power-saving return). Note that the normal state is an operation state in which power is supplied to each functional unit of the multifunction device 20.

[0120] When the terminal device 10 receives a notification indicating that the multi-function device 20 is performing an energy-saving return, the terminal device 10 may display a message on the display unit 140 indicating that the multi-function device 20 is performing an energy-saving return. Further, the terminal device 10 may display a message on the display unit 140 indicating that it takes time for the energy-saving return and thus it takes time to update to an operation screen for inputting the next operation from the terminal device 10. Note that by storing UI data for operating the multi-function device 20 in advance in the storage unit 160, the terminal device 10 may display an operation screen based on the UI data and accept the next operation.

[0121] (2) When the power of the multi-function device 20 is turned off When the multi-function device 20 is in a state where communication with the terminal device 10 is disconnected, the power may be directly turned off by the user. In this case, since the terminal device 10 cannot resume communication with the multi-function device 20, the terminal device 10 displays a message prompting the user to check the state of the multi-function device 20. Further, the terminal device 10 may prepare in advance by storing a resume screen (for example, an operation screen for inputting the next operation) for resuming the operation when the power of the multi-function device 20 is turned on in the storage unit 160 or the like, and display the resume screen.

[0122] Further, when it is impossible to continue a print / send job after the power of the multi-function device 20 is turned on, the terminal device 10 may display a message on the display unit 140 indicating that it is impossible to continue the job.

[0123] (3) When a communication timeout occurs from the destination When the multi-function device 20 transmits image data or the like to a destination specified by the user, such as in the Scan To Email function, the multi-function device 20 does not receive all job information from the terminal device 10 at once, but receives the job information from the terminal device 10 page by page and can execute sequentially.

[0124] During the execution of a job based on page-by-page job information by the multifunction device 20, when communication with the terminal device 10 is disconnected, a timeout may occur in communication with the transmission destination. In this case, the multifunction device 20 stores the occurrence of the timeout in association with the job detail information of the terminal device 10 where the communication was interrupted. Also, when a timeout occurs in communication with the transmission destination, the terminal device 10 displays a message indicating that a timeout has occurred at the transmission destination of the job information on the display unit 140.

[0125] (4) When the job is canceled on the multifunction device 20 side When the multifunction device 20 is in a state where communication with the terminal device 10 is disconnected, if it is possible to cancel an interrupted operation or job, the job related to the remote operation with the terminal device 10 may be canceled on the multifunction device 20 side. In this case, the terminal device 10 displays a message indicating that the job has been canceled on the display unit 140.

[0126] (5) When a trouble occurs in the multifunction device 20 When the multifunction device 20 is in a state where communication with the terminal device 10 is disconnected, a trouble may occur in the multifunction device 20. In this case, the terminal device 10 displays a message indicating that a trouble has occurred in the multifunction device 20 on the display unit 140.

[0127] (6) When the settings of the multifunction device 20 are changed When the multifunction device 20 is in a state where communication with the terminal device 10 is disconnected, the settings and status of the multifunction device 20 may be changed. For example, in the multifunction device 20, the size, orientation of the paper in the paper tray, or the content of the address book (information on the transmission destination of image data) may be changed.

[0128] When communication with the terminal device 10 resumes, the multifunction device 20 transmits information indicating that the settings of the multifunction device 20 have been changed to the terminal device 10. In this case, the terminal device 10 displays a message indicating that the settings of the multifunction device 20 have been changed on the display unit 140. Further, the terminal device 10 may confirm with the user whether to update the changed part (changed information) among the job information from the past information.

[0129] [2.2 Processing Flow] [2.2.1 Terminal Device] Referring to FIG. 15, the processing executed by the terminal device 10 of the present embodiment will be described. In the present embodiment, after the control unit 100 reconnects to the multifunction device 20 (step S114), it determines whether a response has been received from the multifunction device 20 within a predetermined time (step S200). The predetermined time may be determined in advance or may be settable by the user.

[0130] If the control unit 100 receives a response from the multifunction device 20 within the predetermined time, it acquires the state of the multifunction device 20 (step S200; Yes → step S202). For example, the control unit 100 may acquire the state of the multifunction device 20 by inquiring about the state to the multifunction device 20. Further, by configuring the multifunction device 20 to send a response including information indicating the state of the multifunction device 20 when reconnected from the terminal device 10, the control unit 100 may acquire the state of the multifunction device 20 based on the response from the multifunction device 20.

[0131] As the state of the multifunction device 20, as described in the outline of the processing, the following states are acquired. · Performing power saving recovery · Communication timeout occurred from the destination · Job canceled · Trouble occurred on the multifunction device 20 side · Settings changed on the multifunction device 20 side

[0132] The control unit 100 displays a message indicating the state of the multifunction machine 20 according to the state of the multifunction machine 20 (step S204). For example, when the state of the multifunction machine 20 is a state in which energy saving recovery is being performed, the control unit 100 displays a message indicating that the multifunction machine 20 is performing energy saving recovery on the display unit 140. At this time, the control unit 100 may display an operation screen for inputting the next operation when the state of the multifunction machine 20 is a state in which energy saving recovery is being performed. Further, when the state of the multifunction machine 20 is a state in which the settings have been changed on the multifunction machine 20 side, the control unit 100 may inquire of the user whether to update the changed part from the past information.

[0133] After displaying the message indicating the state of the multifunction machine, the control unit 100 executes reconnection processing (step S116). Note that if there is no change in the state of the multifunction machine 20, the control unit 100 may omit the processing of step S204 and execute the reconnection processing.

[0134] On the other hand, when there is no response from the multifunction machine 20 within a predetermined time, the control unit 100 displays a message prompting confirmation of the multifunction machine on the display unit 140 (step S200; No → step S206). At this time, the control unit 100 may display a restart screen on the display unit 140.

[0135] Further, the control unit 100 may reconnect to the multifunction machine 20 periodically so that the connection is restarted when the power of the multifunction machine 20 is turned on (at power-on). Here, when the connection to the multifunction machine 20 is restarted, the control unit 100 receives information indicating whether the job can be continued (step S208) and determines whether the job cannot be continued (step S210).

[0136] If the job cannot be continued, the control unit 100 displays a message indicating that the job cannot be continued on the display unit 140 (step S210; Yes → step S212) and ends the processing shown in FIG. 15. On the other hand, if the job can be continued, the control unit 100 executes the processing of step S116 (step S210; No → step S116).

[0137] [2.2.2 Multifunction Device] (Reconnection Handling) Referring to FIG. 16, the reconnection handling executed by the multifunction device 20 of this embodiment will be described. The control unit 200 determines whether there is a change in the state of the multifunction device 20 (step S280). The state changes that occur in the multifunction device 20 include changes such as shifting to the power-saving mode, a timeout occurring in the transmission destination of job information, a job being canceled, a trouble occurring, and a setting being changed.

[0138] When there is a change in the state of the multifunction device 20, the control unit 200 stores the state of the multifunction device 20 after the change in correspondence with the information of the terminal device 10 during which the communication is interrupted (step S280; Yes → step S282). For example, the control unit 200 may store the information indicating the state of the multifunction device 20 in the job detail information whose status is "not executed" among the job detail information, or may store the information indicating the state of the multifunction device 20 separately from the job detail information in association with the job detail information.

[0139] Note that when there is no change in the state of the multifunction device 20, the control unit 200 omits the process in step S282 (step S280; No).

[0140] Also, before executing the reconnection process, the control unit 200 transmits the information indicating the state of the multifunction device 20 to the terminal device 10 that has transmitted the information regarding the job in the reconnection waiting state (step S284). The control unit 200 may transmit the information indicating the state of the multifunction device 20 in response to an inquiry from the terminal device 10, or may transmit it included in the response to the reconnection from the terminal device 10.

[0141] (Power-On Processing) When the power is turned on, the control unit 200 executes the power-on process shown in FIG. 17. First, the control unit 200 identifies jobs in the reconnection waiting state. For example, the control unit 200 identifies, as a job in the reconnection waiting state, a job corresponding to the operation information / job information included in the job detail information whose status is "not executed".

[0142] Subsequently, the control unit 200 determines whether the job identified in step S290 is resumable, and transmits information indicating whether the job is resumable to the terminal device 10 that transmitted the job information corresponding to the job (step S292).

[0143] By executing the above-described processing by the terminal device 10 and the multifunction device 20, the terminal device 10 can acquire the state of the multifunction device 20 after the communication with the multifunction device 20 is resumed, and can switch the processing according to the state of the multifunction device 20.

[0144] In this way, when the terminal device of the present embodiment reconnects to the multifunction device, it is possible to display an appropriate message according to the state of the multifunction device or to prompt the user to input the next operation. As a result, the terminal device of the present embodiment can realize a highly convenient remote operation for the user.

[0145] [3. Third Embodiment] Subsequently, the third embodiment will be described. The third embodiment is an embodiment that deals with the case where a remote operation by another user is requested when the remote operation by a certain user is interrupted, in addition to the processing described in the first embodiment. This embodiment is obtained by replacing FIG. 6 with FIGS. 18 to 20, FIG. 7 with FIG. 21, FIG. 8 with FIG. 22, and FIG. 9 with FIG. 23, respectively. Note that the same reference numerals are given to the same processing, and the description thereof is omitted.

[0146] When the communication between the multifunction device 20 and the terminal device 10 is interrupted, the multifunction device 20 may execute operations or jobs by another user (another terminal device 10) through remote operations. In this case, if the interrupted communication resumes and another user is performing a remote operation, the terminal device 10 that resumes the interrupted communication will display a message indicating that another user is operating.

[0147] In addition, when the interrupted communication resumes and another user is performing a remote operation, the multifunction device 20 may reserve a remote operation from the terminal device 10 that resumes the communication. Also, the terminal device 10 may store UI data for operating the multifunction device 20 in advance, and when the multifunction device 20 is being operated by another user, display an operation screen based on the UI data and accept the next operation. Further, when the remote operation by another user ends, the multifunction device 20 may notify the terminal device 10 that resumed the communication that remote operations are now possible. When the terminal device 10 receives a notification indicating that remote operations are possible from the multifunction device 20, it resumes the remote operation.

[0148] In addition, when the communication between the multifunction device 20 and the terminal device 10 is interrupted, the multifunction device 20 may be able to set whether to permit or prohibit remote operations and job executions by another user. When prohibiting another user from performing another operation or job execution, the multifunction device 20 may display a message indicating that the execution of another operation or job is prohibited to the said other user on the display unit 240.

[0149] When the communication with the terminal device 10 is interrupted, the setting of whether to permit or prohibit remote operations and job executions by another user is set, for example, by the administrator of the multifunction device 20 and stored in the storage unit 260.

[0150] [3.1 Outline of the Process] Next, an overview of the processing of this embodiment will be described with reference to FIGS. 18 to 20. Note that the MFP 20 will be described as being connectable to two terminal devices 10 (the first terminal device 10 and the second terminal device 10).

[0151] First, the first terminal device 10 and the MFP 20 execute the processing from S1000 to S1010. As a result, the MFP 20 maintains a state (a state in which communication can be received) in which it can receive a reconnection request from the first terminal device 10.

[0152] On the other hand, the second terminal device 10 receives a remote operation start request (S3000), and performs a handshake (QUIC Handshake) using the QUIC protocol with the MFP 20 to establish a connection (S3002).

[0153] The MFP 20 stores the Connection ID and the TLS key information related to the communication with the second terminal device 10 (S3004). Similarly, the second terminal device 10 stores the Connection ID and the TLS key information (S3006). The processing from S3000 to S3006 is the same as the processing from S1000 to S1006.

[0154] Next, the MFP 20 determines whether or not it is set to permit a remote operation or job execution by another user while the communication with the terminal device 10 is interrupted (S3008). If remote operation or job execution by another user is not permitted while the communication with the terminal device 10 is interrupted, the MFP 20 issues a busy notification to the second terminal device 10 (S3008; No → S3010). The busy notification is a notification indicating that a remote operation cannot be performed because another user is using the MFP 20. The second terminal device 10 that has received the busy notification displays a message indicating that another user is using the MFP 20 on the display unit 140 (S3012).

[0155] When allowing remote operations or job executions by another user while the communication with the terminal device 10 is interrupted, the multifunction device 20 starts communication with the second terminal device 10 and performs communication of transmission / reception information (Application Data) using the Connection ID and the TLS key information (S3008; Yes → S3014). As a result, the second terminal device 10 can perform remote operations (S3016). Note that the second terminal device 10 ends the remote operation by performing an end process on the multifunction device 20. The end process is, for example, a process of transmitting information (Application Data) indicating the end of the remote operation (S3018). At this time, the second terminal device 10 may disconnect the connection with the multifunction device 20. As a result, the use of the multifunction device 20 by the user using the second terminal device 10 ends.

[0156] On the other hand, when the change of the IP address of the first terminal device 10 is completed or the network of the first terminal device 10 is restored while the remote operation by the second terminal device 10 is being performed, the first terminal device 10 makes a reconnection request to the multifunction device 20 (S1014). The multifunction device 20 identifies the job in the reconnection waiting state from the multifunction device Connection ID received from the first terminal device 10 and the TLS key information corresponding to the multifunction device Connection ID (S1016). As a result, the multifunction device 20 can identify that a remote operation is being performed again from the first terminal device 10 whose communication has been interrupted.

[0157] When another user is using the multifunction device 20, the multifunction device 20 sends a busy notification to the first terminal device 10 (S3018; Yes → S3020). The first terminal device 10 that has received the busy notification displays a message indicating that another user is using the multifunction device 20 on the display unit 140 (S3022). Note that at this time, the multifunction device 20 sets the user of the first terminal device 10 that has made a reconnection when another user is using the multifunction device 20 as a waiting user. As a result, the multifunction device 20 sets the first terminal device 10 in the remote operation waiting state.

[0158] On the other hand, when no other user is using the multifunction device 20, the multifunction device 20 executes a reconnection process with the terminal device 10 without sending a busy notification and resumes communication of transmission / reception information (S3018; No → S1018 → S1020).

[0159] Also, when the remote operation ends, the multifunction device 20 determines whether there is a waiting user (S3024). If there is a waiting user, the multifunction device 20 executes a reconnection process for the terminal device 10 (the first terminal device 10 in the remote waiting state) used by the waiting user (S3024; Yes → S1018). At this time, the multifunction device 20 may notify the terminal device 10 that the operation by another user has ended (operation end notification). When the operation end notification is received from the multifunction device 20, the terminal device 10 and the multifunction device 20 may execute a reconnection process.

[0160] Thereby, the first terminal device 10 and the multifunction device 20 resume communication of predetermined information (S1020). Also, the first terminal device 10 becomes capable of remote operation (S1022).

[0161] [3.2 Processing of Terminal Device] The flow of the processing of the terminal device 10 in the present embodiment will be described with reference to FIG. 21.

[0162] In the present embodiment, after the control unit 100 executes the processing in step S104, the control unit 100 determines whether a busy notification has been received from the multifunction device 20 (step S300). If a busy notification has been received, the control unit 100 displays a message indicating that another user is using the multifunction device 20 on the display unit 140 (step S300; Yes → step S302) and ends the processing shown in FIG. 21. In this case, the control unit 100 may execute the processing again from step S100. Note that when no busy notification is received, the control unit 100 starts a remote operation (step S300; No → step S106).

[0163] After the control unit 100 executes the process in step S114, it determines whether a busy notification has been received from the multifunction device 20 (step S304). If a busy notification has been received, the control unit 100 displays a message on the display unit 140 indicating that another user is using the multifunction device 20 (step S304; Yes → step S306). Note that if no busy notification is received, the control unit 100 executes a reconnection process (step S304; No → step S116).

[0164] Subsequently, the control unit 100 determines whether it has been notified from the multifunction device 20 that the operation of the multifunction device 20 by another user has ended (step S308). When the control unit 100 receives an operation end notification from the multifunction device 20 indicating that the operation of the multifunction device 20 by another user has ended, it executes a reconnection process (step S308; Yes → step S116). Note that if no operation end notification is received, the control unit 100 repeats the process in step S308 until an operation end notification is received (step S308; No).

[0165] Through the above-described process, when a busy notification is received when the control unit 100 reconnects to the multifunction device 20, the control unit 100 can wait until an operation end notification is received and execute a reconnection process when the operation end notification is received.

[0166] [3.3 Processing of Multifunction Device] The processing flow of the multifunction device 20 in the present embodiment will be described with reference to FIGS. 22 and 23. First, the processing described in FIG. 22 will be described.

[0167] After the control unit 200 executes the process in step S152, it determines whether another user is using the multifunction device 20 (step S350). If another user is using the multifunction device 20, a busy notification is sent to the terminal device 10 that has performed a handshake (step S350; Yes → step S356).

[0168] Further, when no other user is using it, the control unit 200 determines whether there is a user whose communication based on remote operation is interrupted (step S350; No → step S352). For example, when the remote operation is interrupted, the control unit 200 stores information indicating that the remote operation has been interrupted in the storage unit 260. When the information indicating that the remote operation has been interrupted is stored in the storage unit 260, the control unit 200 determines that there is an interrupted user.

[0169] When there is an interrupted user, the control unit 200 determines whether remote operation or job execution by another user is permitted (step S352; Yes → step S354). When remote operation or job execution by another user is not permitted, the control unit 200 notifies the terminal device 10 that has performed the handshake of being busy (step S354; No → step S356).

[0170] Note that when there is no interrupted user, the control unit 200 starts accepting remote operations from the terminal device 10 that has performed the handshake (step S352; No → step S154). Also, even when there is an interrupted user, when remote operation or job execution by another user is permitted during the interruption, the control unit 200 starts accepting remote operations from the terminal device 10 that has performed the handshake (step S354; Yes → step S154).

[0171] Next, with reference to FIG. 23, the reconnection support process executed by the multifunction machine 20 of the present embodiment will be described. In the description of FIG. 23, the terminal device 10 whose connection has been interrupted is described as the first terminal device 10, and another terminal device 10 different from the first terminal device 10 is described as the second terminal device 10.

[0172] First, the control unit 200 determines whether a handshake is requested from the second terminal device 10 (step S380). When a handshake is requested, the multifunction machine 20 performs a handshake with the second terminal device (step S380; Yes → step S382).

[0173] Subsequently, when there is a user in the interrupted state, the control unit 200 determines whether remote operations or job executions by another user are permitted (step S384). When remote operations or job executions by another user are permitted, the control unit 200 accepts remote operations from the user (another user) using the second terminal device (step S384; Yes → step S386). On the other hand, when remote operations or job executions by another user are not permitted, the control unit 200 sends a busy notification to the second terminal device (step S384; No → step S388).

[0174] Note that when the control unit 200 determines in step S380 that no handshake is requested from the second terminal device 10, the processing from step S382 to step S388 is omitted (step S380; No).

[0175] Subsequently, the control unit 200 determines whether it has been reconnected from the first terminal device 10 (step S390). When it has not been reconnected from the first terminal device 10, the control unit 200 returns to step S380 (step S390; No → step S380). On the other hand, when it has been reconnected from the first terminal device 10, the control unit 200 identifies the job in the reconnection waiting state from the Connection ID received from the first terminal device 10 and the TLS key information of the first terminal device 10 (step S390; Yes → step S182).

[0176] Subsequently, the control unit 200 determines whether the communication timeout time set for the first terminal device 10 that has been reconnected has elapsed (step S184). When the timeout time has elapsed, the control unit 200 rejects the reconnection from the first terminal device 10 (step S184; No → step S190). On the other hand, when the timeout time has not elapsed, the control unit 200 determines whether another user is using the multifunction device 20 (step S184; Yes → step S392).

[0177] If another user is using the multifunction device 20, the control unit 200 notifies the first terminal device 10 of the busy status (step S392; Yes → step S394). Further, the control unit 200 determines whether or not the use of the multifunction device 20 by another user has ended (step S396). If the use of the multifunction device 20 by another user has not ended, the control unit 200 repeatedly executes the process in step S396 (step S396; No). On the other hand, when the use of the multifunction device 20 by another user has ended, the control unit 200 notifies the first terminal device 10 of the end of the operation (step S396; Yes → step S398).

[0178] Note that if the control unit 200 determines in step S392 that another user is not using the multifunction device 20, the control unit 200 omits the processes from step S394 to step S398 described above and executes the reconnection process in step S186 (step S392; No → step S186). In this case, the control unit 200 does not notify the first terminal device 10 of the busy status. As a result, when the reconnection from the first terminal device 10 to the multifunction device 20 is performed, the reconnection process is executed as it is between the first terminal device 10 and the multifunction device 20.

[0179] By the above-described process, when the multifunction device 20 is remotely operated via the first terminal device 10 and the second terminal device 10, the multifunction device 20 can start the remote operation or notify of the busy status based on the communication state and settings with the terminal device 10.

[0180] As described above, when the communication is interrupted and a remote operation is started by another user, the terminal device and the multifunction device in the present embodiment can appropriately perform the remote operation.

[0181] [4. Modification Example] The present invention is not limited to the above-described embodiments, and various modifications are possible. That is, embodiments obtained by combining technical means appropriately changed without departing from the gist of the present invention are also included in the technical scope of the present invention.

[0182] In addition, although the above-described embodiments have parts that are separately described for convenience of explanation, it goes without saying that they may be combined and executed within a technically possible range. For example, the second embodiment and the third embodiment may be combined. In this case, when the communication with the multifunction device is resumed, the terminal device can execute processing according to the state of the multifunction device. Further, the terminal device and the multifunction device can appropriately respond when remotely operated by a plurality of users.

[0183] Also, the programs operating in each device in the embodiment are programs (programs that function a computer) that control a CPU or the like so as to realize the functions of the above-described embodiments. And the information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) during its processing, and then stored in a storage device such as various ROMs (Read Only Memory) and HDDs, and read out by the CPU as necessary for correction and writing.

[0184] Here, the recording medium for storing the program may be any of a semiconductor medium (for example, ROM, non-volatile memory card, etc.), an optical recording medium or a magneto-optical recording medium (for example, DVD (Digital Versatile Disc), MO (Magneto Optical Disc), MD (Mini Disc), CD (Compact Disc), BD (Blu-ray (registered trademark) Disk), etc.), a magnetic recording medium (for example, magnetic tape, flexible disk, etc.). Also, by executing the loaded program, not only the functions of the above-described embodiments are realized, but also the functions of the present invention may be realized by processing in cooperation with an operating system or other application programs or the like based on the instructions of the program.

[0185] When distributing it on the market, the program can be stored in a portable recording medium for distribution, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server computer is of course also included in the present invention.

Explanation of Signs

[0186] 1 System 10 Terminal Device 100 Control Unit 140 Display Unit 150 Operation Unit 160 Storage Unit 162 Operation Application 164 Multifunction Machine Information Storage Area 190 Communication Unit 20 Multifunction Machine 200 Control Unit 202 Image Processing Unit 220 Image Input Unit 230 Image Forming Unit 240 Display Unit 250 Operation Unit 260 Storage Unit 262 Processed Data Storage Area 264 Job Detail Information Storage Area 290 Communication Unit

Claims

1. A control unit, a storage unit, and a communication unit capable of communicating with a plurality of terminal devices, wherein the control unit, establishes a connection with a terminal device via the communication unit by QUIC, stores in the storage unit identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the terminal device, performs communication of the transmission / reception information with the terminal device by HTTP / 3, when the connection with the terminal device is disconnected, establishes a connection with the terminal device using the identification information and the key information, and resumes communication of the transmission / reception information with the terminal device, when the connection with the first terminal device is disconnected after establishing a connection with the first terminal device, establishes a connection with the second terminal device, performs communication of transmission / reception information with the second terminal device, and after the connection with the second terminal device is disconnected, establishes a connection with the first terminal device and performs communication of transmission / reception information with the first terminal device An image forming apparatus characterized by the above.

2. The image forming apparatus according to claim 1, wherein the key information is information on a common key used for communication using the TLS (Transport Layer Security) protocol.

3. The control unit, associates and stores the identification information, the key information, and the transmission / reception information received from the terminal device, when communication with the terminal device is resumed, resumes communication of the transmission / reception information associated with the identification information and the key information The image forming apparatus according to claim 1 or 2, characterized by the above.

4. The image forming apparatus according to any one of claims 1 to 3, wherein the control unit sets a time for accepting reconnection from the terminal device when the connection with the terminal device is disconnected, and rejects resumption of communication of the transmission / reception information with the terminal device when reconnection is made from the terminal device after the time has elapsed.

5. The image forming apparatus according to claim 1, wherein the control unit, when the connection with the first terminal device is disconnected after establishing the connection with the first terminal device and communication of transmission / reception information is being performed with the second terminal device after establishing a connection with the second terminal device, notifies the first terminal device that another user is using it when a connection establishment is requested from the first terminal device.

6. A control method for an image forming apparatus capable of communicating with a plurality of terminal devices, A step of establishing a connection with a terminal device by QUIC; A step of storing identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the terminal device; A step of performing communication of the transmission / reception information with the terminal device by HTTP / 3; When the connection with the terminal device is disconnected, a step of establishing a connection with the terminal device using the identification information and the key information and resuming communication of the transmission / reception information with the terminal device; including; A control method for an image forming apparatus, which, when the connection with the first terminal device is established and then the connection with the first terminal device is disconnected, establishes a connection with a second terminal device, performs communication of transmission / reception information with the second terminal device, and after the connection with the second terminal device is disconnected, establishes a connection with the first terminal device and performs communication of transmission / reception information with the first terminal device.

7. An image forming apparatus comprising a control unit, a storage unit, a display unit, and a communication unit, wherein the control unit establishes a connection with a terminal device by QUIC via the communication unit; stores in the storage unit identification information for identifying the connection and key information used for encrypting transmission / reception information transmitted / received with the terminal device; performs communication of the transmission / reception information with the terminal device by HTTP / 3; when the connection with the terminal device is disconnected, establishes a connection with the terminal device using the identification information and the key information and resumes communication of the transmission / reception information with the terminal device; while resuming, displays a message during resumption on the display unit and displays a cancel button for receiving an instruction to cancel the resumption on the display unit characterized by the above.

Images