Remote operation system, image processing device, and remote operation program

Abstract

To make it possible to perform remote operation of a printer from a terminal device without performing complicated processing.SOLUTION: A remote operation system 1 includes a printer 100, a server 200, and a terminal device 300, where user operation at the terminal device 300 enables remote operation of the printer 100 through the server 200. The printer 100 executes an S30 of making a screen based on a state at the point of time be displayed on a touch panel 140, an S35 of generating image information corresponding it, and an S40 of transmitting the image information to the server 200. The server 200 executes an S45 of receiving the image information and an S50 of transmitting the image information to the terminal device 300. The terminal device 300 executes an S55 of receiving the image information and an S60 of making a display unit 340 display a screen on the basis of the image information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a remote control system for remotely controlling an image processing device from a terminal, an image processing device, and a remote control program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, as described in Patent Document 1, for example, a configuration is known in which a local terminal is directly connected to an image processing device equipped with a touch screen, and the touch screen of the image processing device can be remotely operated by a user operation on the local terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-122424 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, for example, when an image processing device is on an office's internal network, it is necessary to perform an activation process to connect a local terminal to the internal network, or to install a special connection tool on the image processing device, making the process complicated.

[0005] An object of the present invention is to provide a remote operation system, a printing device, and a remote operation program that enable an image processing device to be remotely operated from a terminal without performing complicated processing. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides a remote operation system including an image processing device having an image processing control unit and an image processing display unit, a server having a server control unit, and at least one NW terminal having a NW terminal control unit and a NW terminal display unit, and enabling remote operation of the image processing device via the server by a user operation on the NW terminal, wherein the image processing control unit performs a first display process for displaying a first display screen on the image processing display unit, and a first screen data generation process for generating first screen data showing the first display screen displayed in the first display process, the first screen data being capable of displaying the first display screen on the NW terminal display unit. the server control unit executes a first reception process to receive the first screen data transmitted from the image processing control unit in the first transmission process and a second transmission process to transmit the first screen data received in the first reception process to the NW terminal; and the NW terminal control unit executes a second reception process to receive the first screen data transmitted from the server control unit in the second transmission process and a second display process to display the first display screen on the NW terminal display unit based on the first screen data received in the second reception process.

[0007] In the present invention, the NW terminal is connected to the image processing device via the server. The first display screen displayed on the image processing display unit of the image processing device in the first display process is displayed on the NW terminal display unit in the second display process by transmitting the first screen data generated by the image processing control unit to the NW terminal via the server control unit.

[0008] According to the present invention, the image processing device accesses the server, and the NW terminal also accesses the server, so that the screen display on the image processing device and the screen display on the NW terminal can be linked via the server and remotely controlled. As a result, there is no need to perform complicated processing, unlike when remotely controlling the image processing device from a local terminal directly connected to the image processing device, and convenience can be improved.

[0009] In order to achieve the above object, the present invention provides an image processing device that has an image processing control unit and an image processing display unit and is remotely operated via a server by a user operation on a NW terminal, wherein the image processing control unit performs a first display process for causing a first display screen to be displayed on the image processing display unit, a first screen data generation process for generating first screen data that indicates the first display screen displayed in the first display process and that can cause the NW terminal to display the first display screen, a first transmission process for transmitting the first screen data generated in the first screen data generation process to the server and causing the NW terminal to display the first display screen based on the first screen data received from the server, and a second transmission process for transmitting the first screen data instead of the first display screen. The display device is configured to execute a third display process for displaying a second display screen obtained by changing the display screen on the image processing display unit, a second screen data generation process for generating second screen data indicating the second display screen when the second display screen is displayed on the image processing display unit in the third display process, the second screen data being capable of displaying the second display screen on the NW terminal, and not generating the second screen data when the second display screen is not displayed on the image processing display unit but the first display screen is displayed, and a third transmission process for transmitting the second screen data generated in the second screen data generation process to the server and displaying the first display screen on the NW terminal using the first screen data received from the server.

[0010] In addition, in order to achieve the above object, the present invention is configured to execute the following steps for the calculation unit of a terminal capable of remotely operating an image processing device via a server by a user operation: a second reception process for receiving from the server first screen data indicating a first display screen displayed on an image processing display unit of the image processing device, the first screen data being capable of causing the first display screen to be displayed on the terminal display unit based on the first screen data received in the second reception process; a fourth reception process for receiving from the server second screen data indicating a second display screen to be displayed in place of the first display screen on the image processing display unit of the image processing device, the second display screen being a second display screen obtained by changing the first display screen, the second display screen being capable of causing the second display screen to be displayed on the terminal display unit; and a fourth display process for switching the first display screen displayed on the terminal display unit to the second display screen based on the second screen data received in the fourth reception process. Effect of the Invention

[0011] According to the present invention, it is possible to remotely control an image processing device from a terminal without performing complicated processing. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram illustrating an example of a network connection configuration of a printing system and a remote control system according to the embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a configuration of a printer. [Diagram 3] FIG. 2 is a block diagram illustrating an example of a configuration of a server. [Figure 4] FIG. 2 is a block diagram illustrating an example of the configuration of a NW terminal. [Diagram 5] FIG. 1 is a block diagram showing three connection configurations during remote control. [Figure 6] FIG. 2 is a diagram illustrating an example of the appearance of a touch panel and operation keys of a printer. [Figure 7]11 is a diagram illustrating an example of a screen displayed on a web browser screen of a terminal device, which imitates the layout of a touch panel and operation keys of a printer. [Figure 8] 13 is a diagram illustrating a screen display in which a home screen of a touch panel is switched to an ink setting screen. FIG. [Figure 9] 13A and 13B are diagrams illustrating an example of a ripple animation displayed when an operation is performed on a web browser screen of a user's terminal. [Figure 10] 13A and 13B are diagrams illustrating an example of a ripple animation displayed when an operation is performed on a web browser screen of another terminal. [Figure 11] 11 is a sequence chart illustrating an example of a control procedure from when a NW terminal in a push-type connection configuration transmits a connection request until remote operation is performed. [Figure 12] 11 is a sequence chart illustrating an example of a control procedure when a NW terminal in a push-type connection configuration transmits a disconnection request to end a remote operation. [Figure 13] 11 is a sequence chart illustrating an example of a control procedure from when a network terminal in a pull-type connection topology transmits a connection request until remote operation is performed. [Figure 14] 11 is a sequence chart illustrating an example of a control procedure when a NW terminal in a pull-type connection topology transmits a disconnection request to end a remote operation. [Figure 15] 10 is a sequence chart illustrating an example of a control procedure from when an LC terminal in a local type connection configuration transmits a connection request to when it executes a remote operation and ends the remote operation. [Figure 16] 11 is a sequence chart illustrating an example of a control procedure when two NW terminals in the same pull-type connection configuration remotely operate the same printer at the same time. [Figure 17] 11 is a sequence chart illustrating an example of a control procedure for exclusive execution when an LC terminal in a local connection topology requests a connection during remote operation by a NW terminal in a pull connection topology. [Figure 18] 13 is a sequence chart illustrating an example of a control procedure in which a NW terminal in a push-type connection configuration performs a process up to calculating a time interval in response to a long press operation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings.

[0014] <Outline of network connection configuration for remote control system> As shown in Fig. 1, the remote operation system includes a printer 100 and an LC terminal 300L connected to a local area network LAN such as a LAN (Local Area Network) provided in a specific site such as an office or a factory, and a server 200E, a server 200I, and NW terminals 300E and 300I connected to a wide area network GAN such as the Internet that covers a wide area. The local area network LAN can communicate with the wide area network GAN. Specifically, the printer 100 to be operated and the LC terminal 300L are connected to the local area network LAN, and the servers 200E and 200I, the NW terminal 300E, and the NW terminal 300I, which are managed as part of the remote operation system 1, are connected to the wide area network GAN.

[0015] In order to ensure information security within the local area network LAN, the local area network LAN and the wide area network GAN are connected to each other via a firewall server (not shown) so that they can transmit and receive information to each other. In the remote operation system 1, a user can not only directly operate the printer 100 itself, but also perform task processing and maintenance work by remote operation via the local area network LAN and the wide area network GAN. The printer 100 may be an image processing office machine having other printing functions such as a copy machine, a FAX machine, and a scanner, a communication function, and an optical reading function, and a multifunction machine thereof. The printer 100 is an example of an image processing device, the server 200I and the server 200E are examples of servers, and the NW terminal 300I and the NW terminal 300E are examples of NW terminals.

[0016] <Printer> 2, the printer 100 has a processor 110, a storage device 115, a touch panel 140, operation keys 150, a transport mechanism 160, a communication interface 170, and a printing unit 190. The processor 110, the storage device 115, the touch panel 140, the operation keys 150, the transport mechanism 160, the communication interface 170, and the printing unit 190 are connected to each other via a bus 105 so as to be able to transmit and receive data therebetween.

[0017] The storage device 115 includes a volatile storage device 120 and a non-volatile storage device 130. The volatile storage device 120 includes an image data storage area 122 that stores image data to be printed. The volatile storage device 120 is, for example, a DRAM. The non-volatile storage device 130 includes a program storage area 131 and a data storage area 132. The non-volatile storage device 130 is, for example, an NVRAM, a flash memory, or the like. The program storage area 131 stores various programs. The various programs include a remote operation program 410 described later based on sequence charts in Figs. 11 to 18 described later, firmware such as a print processing program, and the like. The data storage area 132 stores various data generated by the above processing programs.

[0018] The processor 110 is a device that performs data processing. The processor 110 is, for example, a CPU. The processor 110 executes various programs stored in the program storage area 131.

[0019] The touch panel 140 can display various information and accepts user operations at coordinates on the display screen. The touch panel 140 is, for example, a device that integrally combines a liquid crystal display and a transparent touch pad. The operation keys 150 are devices that mechanically accept user press operations. The communication interface 170 is a network interface for connecting to an in-house network and transmitting or receiving information. The communication interface 170 has a local connection unit 401, a pull type connection unit 402, and a push type connection unit 403. The local connection unit 401, the pull type connection unit 402, and the push type connection unit 403 are modules for transmitting or receiving information to or from the server 200 when a remote operation described later is executed.

[0020] The conveying mechanism 160 conveys a sheet supplied from a tray capable of holding a plurality of sheets. The printing unit 190 is provided on a sheet conveying path by the conveying mechanism 160, and forms an image corresponding to a print job on the sheet conveyed by the conveying mechanism 160 in a predetermined manner. The predetermined manner is not particularly limited, and may be, for example, an inkjet method or a laser method. The printing unit 190 prints on the sheet in accordance with a print job transmitted from the NW terminals 300I, 300E or the LC terminal 300L via the local area network LAN or the wide area network GAN, or a print job generated by the printing unit 190 itself based on operations on the touch panel 140 and the operation keys 150.

[0021] In addition, the storage device 115 is not limited to being configured with the above-mentioned device elements, but may be configured with, for example, a RAM, a ROM, an EEPROM, a HDD, a portable recording medium such as a USB memory that is detachably attached to the printer 100, a buffer provided in the processor 110, or a combination of these.

[0022] Furthermore, the storage device 115 may be a computer-readable storage medium. A computer-readable storage medium is a non-transitory medium. In addition to the above examples, non-transitory media also include recording media such as CD-ROMs and DVD-ROMs. A non-transitory medium is also a tangible medium. The same applies to the storage device 215 of the server 200 described later.

[0023] The processor 110 is an example of an image processing control unit, and the touch panel 140 is an example of an image processing display unit and a touch screen.

[0024] <Server> Server 200I and server 200E have almost the same configuration as server 200 equipped with a communication function for transmitting and receiving data and an information processing function for processing data, and the common configuration is shown in Fig. 3. As shown in Fig. 3, server 200 has a processor 210, a storage device 215, and a communication interface 290. The processor 210, the storage device 215, and the communication interface 290 are connected to each other via a bus 205 so as to be able to transmit and receive data to and from each other.

[0025] The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230. The volatile storage device 220 is, for example, a DRAM, and stores various programs and data to be processed. The non-volatile storage device 230 is, for example, a hard disk drive or a solid state drive, and has a program storage area 231 and a data storage area 232. Various programs are stored in the program storage area 231. The various programs include processing programs based on sequence charts in Figs. 11 to 18, which will be described later. The data storage area 232 stores various data generated by the processing programs.

[0026] The processor 210 is a device that performs data processing, such as a CPU, and executes various programs stored in the program storage area 231. The processor 210 performs various processes including data communication with the printer 100 connected to each network LAN and GAN, and the NW terminal 300 and the LC terminal 300. The processor 210 is an example of a server control unit, and the storage device 215 is an example of a server storage unit.

[0027] The communication interface 290 is a network interface for connecting to a local area network LAN or a wide area network GAN to send and receive information.

[0028] <Terminal device> Each of the terminal devices 300, that is, the NW terminal 300E connected to the server 200E, the NW terminal 300I connected to the above-mentioned server 200I, and the LC terminal 300, is configured, for example, by a general-purpose personal computer having a display function for image information, an operation input function, a communication function for sending and receiving data, and an information processing function for processing data, and their common configuration is shown in FIG. 4.

[0029] As shown in FIG. 4, the terminal device 300 has a CPU 310, a memory 320, a communication interface 330 for connecting to each network, a display unit 340, an operation unit 350, and a large-capacity storage device 370.

[0030] The mass storage device 370 includes a program storage area 371 and a data storage area 372. The mass storage device 370 is, for example, a hard disk or a flash memory. Various programs are stored in the program storage area 371 as, for example, firmware. The various programs include processing programs based on sequence charts in Figs. 11 to 18 described below. The CPU 310 executes the various programs stored in the program storage area 371. The CPU 310 executes various processes including data communication with the server 200 and the printer 100 connected to each network LAN and GAN via the communication interface 330.

[0031] The terminal device 300 displays various information on a display unit 340 configured, for example, by a liquid crystal display or the like, and accepts various operations from the user via an operation unit 350 configured by a pointing device such as a mouse and a keyboard.

[0032] The terminal device 300 may be, instead of the general-purpose personal computer, another information terminal such as a tablet computer or a smartphone. The CPU 310 is an example of a NW terminal control unit, an LC terminal control unit, and a calculation unit, the memory 320 is an example of a terminal storage unit, and the display unit 340 is an example of a NW terminal display unit and an LC terminal display unit.

[0033] The processor 210 of the printer 100 can execute a predetermined process when a user standing near the printer 100 operates the touch panel 140 or operation keys 150 of the printer 100. The processor 210 of the printer 100 can execute a process remotely by accepting operations on the NW terminals 300I, 300E and LC terminal 300L of other users who are physically distant from the printer 100 via the network. In other words, the program that executes the process based on the output of various information and the input operation contents in the printer 100 includes a remote operation program 410 that executes remote operation from outside the printer 100.

[0034] <Remote control processing mode> The printer 10 transmits and receives information using a communication protocol according to the connection form with the terminal 300. Specifically, the processor 200 of the printer 10 communicates individually via a local connection unit 401, a pull type connection unit 402, and a push type connection unit 403 based on a remote operation program 410. The following describes the connection form between the terminal and the printer 100 in the remote operation process executed by the remote processing system of the embodiment.

[0035] 5 shows a pull-type connection topology in which the NW terminal 300I is indirectly connected to the printer 100 via the server 200I and the remote processing system 1 executes remote operation processing. In the pull-type topology, the printer 10 is connected to the server 200I via a pull-type connection unit 402. The printer 10 polls the server 200I via the pull-type connection unit 402 based on the HTTPS protocol to send and receive information.

[0036] 5 shows a push-type connection topology in which the NW terminal 300E is indirectly connected to the printer 100 via the server 200E and the remote processing system 1 executes remote operation processing. In the push-type connection topology, the printer 10 is connected to the server 200E via the push-type connection unit 402. The server 200E transmits commands directly to the printer 10 via the push-type connection unit 403 based on the MQTT protocol or XMPP. In the push-type connection topology, the printer 10 in the on-premise NW is constantly connected to the server 200E in the off-premise NW using the above protocol, which makes it possible to issue instructions in real time from the off-premise NW.

[0037] For reference, the left side of FIG. 5 shows a local type connection configuration in which the LC terminal 300L is connected to the printer 100 via a local type connection unit 401 and the remote processing system 1 executes remote operation processing.

[0038] Such a remote control system 1 has the advantage that a specific task or operation can be performed even at a location far away from the printer 100.

[0039] In particular, in the remote operation system 1 based on the push-type connection configuration, the NW terminal 300E present in the wide area network GAN is indirectly connected to the printer 100 in the local area network LAN via the server 200E also present in the wide area network GAN. In this case, the printer 100 does not need to connect using special settings or tools because it only needs to access the server 200E present outside the local area network LAN on the normal Internet using a URL. The NW terminal 300E present in the wide area network GAN can also normally access the server 200E using a URL in the same way.

[0040] Furthermore, according to the remote control system 1, using the above-mentioned connection form, the same screen display content as that of the touch panel 140, which is the display unit of the printer 100 itself, can be displayed on the display unit 340 of the NW terminal, and it is also possible to input operations on the operation unit 350 of each terminal device 300 in the same manner as the touch panel 140 and operation keys 150 of the printer 100.

[0041] In the pull-type connection configuration, the printer 100 polls the server 200I via the pull-type connection unit 402 based on the HTTPS protocol to send and receive information. This allows the processor 210 of the printer 100 to send changed image information to each terminal device 300 only when the screen display has changed significantly in response to various operational inputs. This allows a screen display to the extent necessary for remote operation while significantly reducing communication traffic, allowing smooth remote operation by the user and improving convenience.

[0042] Here, the polling of the printer 100 in the pull-type connection configuration will be described. In this case, the operation information corresponding to the operation on the NW terminal 300I side is stored in the storage device 215 of the server 200I. The printer 100 inquires the server 200I about the presence or absence of the operation information by a software interrupt at a period of a predetermined time interval such as its own machine cycle or an interval of 5 seconds. At the time of the inquiry, the server 200I transmits the stored operation information to the printer 100, and the printer 100 acquires the information. According to this, the printer 100 can acquire the operation information from the server 200 by a so-called polling method, so that the communication load between the printer 100 and the server 200 can be reduced to the minimum necessary. Note that the process of storing the operation information in the storage device 215 in the server 200I is an example of a first event storage process, the machine cycle of the printer 100 is an example of a first predetermined period, and the process of the printer 100 actively inquiring the server 200I about the presence or absence of the operation information at the machine cycle of the printer 100 is an example of a first event inquiry process.

[0043] In this embodiment, when remote operation is started, for example, a WEB program written in XML or the like is transmitted from the printer 100, and the terminal device 300 that receives it executes it on a WEB browser in the form of a so-called WEB application. Alternatively, it may be in the form of a so-called desktop application that is independently started on the OS of the terminal device 300.

[0044] <Specific screen display contents during remote operation> In the following, the screen display contents on the touch panel 140 of the printer 100 and the display unit 340 of each NW terminal during the above remote operation will be illustrated and explained.

[0045] 6 shows an example of the appearance of the touch panel 140 and operation keys 150 provided in the console section of the actual printer 100. In the example shown, a plurality of operation keys 150 are arranged around the rectangular touch panel 140, with a connection key 151 on the lower side and a return key 152, a home key 153, a cancel key 154, and a numeric keypad 155 on the immediate right side. A user standing near the actual printer 100 can refer to various information such as messages and parameters displayed on the touch panel 140 and press various icons and specific operation keys 150 displayed on the same touch panel 140 to cause the printer 100 to execute a desired process.

[0046] As shown in Fig. 7, the display unit 340 of each terminal device 300 performing remote operation displays a screen configuration that mimics the arrangement of the touch panel 140 and operation keys 150 of the printer 100 shown in Fig. 6 above, on a WEB browser screen 341 of the terminal device 300, allowing cursor operation with a so-called GUI (Graphical User Interface). A user of the terminal device 300 can refer to various information such as messages and parameters displayed on the touch panel 140, and can operate various icons displayed on the virtual touch panel 140 and icons of the virtual operation keys 150 by operating a cursor C on the WEB browser screen 341 using a pointing device such as a mouse (not shown). This allows the user of each terminal device 300 to remotely operate the printer 100 with the same operational feeling as if they were directly operating the printer 100.

[0047] In addition, in a state where the printer 100 and the terminal device 300 are connected via the server 200, the home screen 140a is displayed on the touch panel 140 of the printer 100 as shown in FIG. 8(a). When the user presses the INK icon at the bottom right of the touch panel 140a, a menu process is executed to switch the display to the ink setting screen 140b shown in FIG. 8(b). In response to this, when the INK icon on the home screen 140a is pressed on the WEB browser screen 341 of the terminal device 300, instruction information including the operation event content and the pressed coordinate position is sent to the printer 100, and only the image information of the ink setting screen 140b is immediately received as a reply. This allows the user to quickly switch the screen display in response to the operation input on the terminal device 300, and allows the user to remotely operate the printer 100 with the same operational feeling as if the user were directly operating the printer 100, with the display content synchronized with the touch panel 140 of the printer 100 in almost real time.

[0048] Here, the display contents on the WEB browser screen 341 of the terminal device 300 are a composite of the display contents on the liquid crystal screen of the touch panel 140 and a screen on which icons of the multiple operation keys 150 are arranged. The display contents on the liquid crystal screen change significantly depending on the operation input and the processing contents, while the display contents on the icon screen of the multiple operation keys 150 can be generated by combining images prepared in advance. Therefore, when at least the printer 100 generates image information on the liquid crystal screen, transmits it via the server 200, and displays it on the terminal device 300, any one of the printer 100, the server 200, and the terminal device 300 may store image information on the icon screen of the operation keys 150 and combine it with image information on the liquid crystal screen. In the example of this embodiment, the terminal device 300 itself stores image information on the icon screen of the operation keys 150, and combines the image information on the icon screen of the operation keys 150 with the image information on the received liquid crystal screen to display it on the WEB browser screen 341. This makes it possible to reduce the data volume of image information circulated over the wide area network GAN between the printer 100, the server 200, and the terminal device 300, thereby reducing traffic delays.

[0049] It is also desirable to be able to visually indicate which icon or which position in the screen coordinates the user pressed on the WEB browser screen 341 of the terminal device 300 or on the touch panel 140 of the printer 100. In response to this, in this embodiment, when a pressing operation is performed with the cursor C on the WEB browser screen 341, an animation 342 of ripples of an action that spreads and disappears in a short time from the pressing operation position as the center, as shown in Fig. 9. Although not particularly shown, the printer 100 may also display a similar ripple animation on the touch panel 140 when the user presses on the touch panel 140 of the printer 100.

[0050] Furthermore, when multiple users who operate the printer 100 and the terminal device 300 are simultaneously accessing the printer 100, it is desirable to be able to clearly indicate which user performed which operation while sharing the display contents of the same touch panel 140 in real time. For this reason, in this embodiment, when a user of a terminal other than the own terminal performs a pressing operation, as shown in Fig. 10, a ripple animation 343 centered on the pressing operation position (a position different from the own cursor C) is displayed in a color corresponding to the user. For example, by displaying the operation of the own terminal in white and displaying the operations of the user of the printer 100 and the user of the other terminal in a different color other than white, it becomes easy to visually distinguish who performed which operation.

[0051] In this case, when the terminal device 300 of the own terminal receives image information reflecting an operation by the user of the printer 100 or the user of the terminal device 300 of the other terminal, coordinate position information of the pressed operation on the liquid crystal screen of the touch panel 140 or pressed key information indicating which operation key 150 was pressed is also transmitted and received together with the image information. This allows the terminal device 300 of the own terminal to display a ripple animation 343 based on the operation of the user of the printer 100 or the user of the other terminal at an appropriate position centered on the coordinate position on the liquid crystal screen or the pressed key position. Note that this coordinate position information and pressed key information are examples of touch event information.

[0052] The change conditions for printer 100 to send image information to terminal device 300 to change the screen display do not include auxiliary changes in the screen display, such as a change in the clock display on the screen or a change over time in the amount of remaining ink. Printer 100 only needs to send new image information to change the screen display when a substantial change in the operation screen display occurs due to an operation performed by any user.

[0053] <Control procedure for push-type connection> In order to realize the above remote operation processing in the push-type connection configuration of this embodiment, an example of the control procedure executed by the processor 110 of the printer 100, the processor 210 of the server 200E, and the CPU 310 of the NW terminal 300E will be described with reference to the sequence charts of Fig. 11 and Fig. 12. Fig. 11 shows the control procedure during the period from when the NW terminal 300E transmits a connection request to when the remote operation is being performed, for example, and Fig. 12 shows the control procedure when the NW terminal 300E transmits a no-operation timeout or a predetermined disconnection request to end the remote operation. Note that before this flow starts, a state in which the printer 100 and the server 200E can access each other via a general Internet connection using a URL has already been established.

[0054] First, in step S5, the user presses the connection key 151 on the WEB browser screen 341 of the NW terminal 300E in this example, causing the NW terminal 300E to transmit a connection request to the server 200E to start remote operation of the printer 100. After receiving this connection request in step S10, the server 200E transmits a remote operation start request to the printer 100 in step S15. After receiving this remote operation start request in step S20, the printer 100 executes various processes in step S25 to become available for accepting remote operation from the NW terminal 300E.

[0055] Next, in step S30, printer 100 displays a screen based on the current state on touch panel 140, generates image information corresponding to that screen in step S35, and then transmits the image information to server 200E in step S40. Note that the image information transmitted by printer 100 is preferably image data in so-called BMP format, which is obtained by converting bitmap information of a bitmap display into data without encoding it. However, if reducing communication traffic is a priority, image data in so-called JPG or PNG format, which has been encoded to reduce the size of the image data, may be transmitted.

[0056] After receiving the image information in step S45, the server 200E transmits the image information to the NW terminal 300E in step S50. After receiving the image information in step S55, the NW terminal 300E displays a screen based on the image information on the display unit 340 in step S60. This starts remote operation of the printer 100 from the NW terminal 300E.

[0057] After that, the NW terminal 300E judges whether or not a no-operation timeout has occurred in step S65. Specifically, the NW terminal 300E judges whether or not a predetermined time has passed without any operation input since the operation input most recently received from the user of the NW terminal 300E. If a no-operation timeout has occurred, the judgment is satisfied (S65: YES), and the process proceeds to step S140 shown in FIG. 12, which will be described later. On the other hand, if any operation input is received from the user before the no-operation timeout elapses, the judgment is not satisfied (S65: NO), and the NW terminal 300E accepts the operation input in step S70. In the next step S75, the NW terminal 300E judges whether or not the accepted operation input is an operation for terminating remote operation. If the accepted operation input is an end operation, the judgment is satisfied (S75: YES), and the process proceeds to step S140 in FIG. 12. On the other hand, if the accepted operation input is not an end operation, the condition is not satisfied (S75: NO), and in step S80, the NW terminal 300E transmits operation information to the server 200E.

[0058] Thereafter, if the operation input in step S81 is a pressing operation, the NW terminal 300E displays animation 342 in which a white ripple is enlarged to a predetermined size and then disappears, with the center at the pressed position of the cursor C (see FIG. 9 above). After that, the NW terminal 300E does not accept any operation input until changed image information is received in step S130 described later. This makes it possible to ignore unnecessary repeated tapping operations from the user and stabilize operability. Note that the NW terminal 300E may accept operation input by the user while displaying animation 342 in step S81 and until changed image information is received in step S130.

[0059] After receiving the operation information in step S85, the server 200E transmits the operation information to the printer 100 in step S90. After receiving the operation information in step S95, the printer 100 executes processing corresponding to the operation information in step S100. Note that the printer 100 also accepts operation input from the user via the touch panel 140 and the operation keys 150 in steps S92 and S93 before receiving the operation information from the server 200E in step S95. If the printer 100 determines that some operation input has been made to the touch panel 140 and the operation keys 150, it executes an operation response process corresponding to the operation input in step S100. Note that this operation response process includes, in addition to the process corresponding to the operation input, a process for displaying a corresponding new change screen and generation of touch event information.

[0060] Thereafter, in step S105, printer 100 determines whether or not a substantial change has occurred in the screen display as a result of the immediately preceding operation response processing. If a substantial screen change has occurred as a result of the operation input, the determination is satisfied (S105: YES), printer 100 generates image information of a changed screen that reflects the change in step S110, and transmits the changed image information to server 200E in step S115. On the other hand, if the change in the screen is merely a supplementary change such as a clock display or remaining ink amount, and there has been no substantial change in the operation screen as a result of the operation input, the determination is not satisfied (S105: NO), and steps S110 and S115 are omitted.

[0061] In addition, if the screen change is not caused by an operation input from the NW terminal 300E to which the changed image information is returned, the printer 100 generates changed image information in step S110 above, including the ID of the device that caused the screen change and the coordinate information of the pressing operation, and transmits the changed image information to the server 200E.

[0062] After receiving the changed image information in step S120, server 200E transmits the changed image information to NW terminal 300E in step S125. After receiving the changed image information in step S130, NW terminal 300E displays a changed screen reflecting the changed image information on display unit 340 in step S135. As a result, the screen whose contents have changed in response to the operation input on NW terminal 300E is also reflected and displayed on display unit 340 of NW terminal 300E. Then, NW terminal 300E returns to step S65 and repeats the same procedure.

[0063] In addition, in the above step S135, if the changed image information includes coordinate information of the pressing operation and the ID of the device that performed the operation, the NW terminal 300E displays on the display unit 340 an animation 343 of expanding and disappearing ripples centered on the coordinate position in a color corresponding to the ID.

[0064] On the other hand, if the NW terminal 300E determines in the above steps S65 and S75 that the remote operation is to be terminated, the NW terminal 300E transmits a disconnection request to the server 200E in step S140 of FIG. 12, and then displays an end screen indicating that the remote operation has been terminated on the display unit 340 in step S145.

[0065] After receiving the disconnection request in step S150, server 200E transmits a remote control end request in step S155 to printer 100. After receiving the remote control end request in step S160, printer 100 executes various processes in step S165 to end the remote control.

[0066] <Control procedure for pull-type connection> An example of the control procedure executed by the processor 110 of the printer 100, the processor 210 of the server 200I, and the CPU 310 of the NW terminal 300I to realize remote operation processing in a pull-type connection configuration will be described with reference to the sequence charts of Fig. 13 and Fig. 14. Fig. 13 shows the control procedure during the period from when the NW terminal 300I transmits a connection request to when the remote operation is being performed, for example, and Fig. 14 shows the control procedure when the NW terminal 300I transmits a no-operation timeout or a predetermined disconnection request to end the remote operation, for example.

[0067] First, in step S205, the user presses the connection key 151 on the WEB browser screen 341 of the NW terminal 300I in this example, causing the NW terminal 300I to transmit a connection request to the server 200I to start remote operation of the printer 100. After receiving this connection request in step S210, the server 200I transmits a remote operation start request to the printer 100 in step S215. More specifically, the printer 100 queries the server 200I at predetermined time intervals by polling, and upon receiving the query from the printer 100, the server 200I transmits a remote operation start request to the printer 100. After receiving this remote operation start request in step S220, the printer 100 executes various processes in step S225 to become available for accepting remote operation from the NW terminal 300I.

[0068] Next, the printer 100 displays a screen based on the state at that time on the touch panel 140 in step S230, generates image information corresponding to the screen in step S235, and transmits the image information to the server 200I in step S240. After receiving the image information in step S245, the server 200I transmits the image information to the NW terminal 300I in step S250, and the NW terminal 300I receives the image information in step S255. In this pull-type connection configuration, the NW terminal 300I obtains the image information from the server 200I by polling. Specifically, although not particularly shown in the figure, the server 200I stores the received image information in the storage device 215 in the above step S245, and the NW terminal 300I actively inquires of the server 200I about the presence or absence of image information at a predetermined time interval, for example, every 5 seconds. If the server 200I stores image information in response to the inquiry, the server 200I transmits the image information in step S250, and the NW terminal 300I receives the image information from the server 200I in step S255. After receiving the image information in step S255, the NW terminal 300I displays a screen based on the image information on the display unit 340 in step S260. This starts remote operation of the printer 100 from the NW terminal 300I.

[0069] After that, the NW terminal 300I judges whether or not a no-operation timeout has occurred in step S265. Specifically, the NW terminal 300I judges whether or not a predetermined time has passed without any operation input since the operation input most recently received from the user of the NW terminal 300I. If a no-operation timeout has occurred, the judgment is satisfied (S265: YES), and the process proceeds to step S340 shown in FIG. 14, which will be described later. On the other hand, if any operation input is received from the user before the no-operation timeout elapses, the judgment is not satisfied (S265: NO), and the NW terminal 300I accepts the operation input in step S270. In the next step S275, the NW terminal 300I judges whether or not the accepted operation input is an operation for terminating the remote operation. If the accepted operation input is an end operation, the judgment is satisfied (S275: YES), and the process proceeds to step S340 in FIG. 14. On the other hand, if the accepted operation input is not the end operation, the determination is not satisfied (S275: NO), and the NW terminal 300I transmits the operation information to the server 200I in step S280.

[0070] After that, when the display unit 340 is pressed in step S281, the NW terminal 300I displays animation 342 in which a white ripple is enlarged to a predetermined size and then disappears, with the center being the pressed position of the cursor C (see FIG. 9 above). After that, no operation input is accepted until changed image information is received in step S330 described later.

[0071] After receiving the operation information in step S285, the server 200I transmits the operation information to the printer 100 in step S290. More specifically, the printer 100 queries the server 200I at a predetermined time interval by polling, and when the server 200I receives the query from the printer 100, it transmits the operation information to the printer 100. After receiving the operation information in step S295, the printer 100 executes processing corresponding to the operation information in step S300. As described above, in this pull-type connection form, the printer 100 obtains the operation information from the server 200I by polling. For this reason, although not shown in the figure, the server 200I stores the operation information received in step S285 in the storage device 215, and the printer 100 actively queries the server 200I about the presence or absence of the operation information at a predetermined time interval, for example, every 5 seconds. Then, when the server 200I stores the operation information in response to the query, the transmission and reception of the operation information is performed in steps S290 and S295. In this case, the process in which server 200I stores the operation information in storage device 215 in step S285 is an example of a first event storage process. The 5-second interval in the above example is an example of a first predetermined period, and the process in which printer 100 actively inquires of server 200 about the presence or absence of operation information at the 5-second interval in the above example before step S290 is an example of a first event inquiry process.

[0072] Note that even in steps S292 and S293 before receiving operation information from server 200I in step S295, if printer 100 receives operation input from the user of printer 100 via touch panel 140 and operation keys 150 and determines that some operation input has been received, printer 100 executes an operation response process corresponding to the operation input in step S300. Note that this operation response process includes not only the process corresponding to the operation input, but also the process of displaying a corresponding new screen and the generation of touch event information.

[0073] Thereafter, in step S305, printer 100 determines whether or not a substantial change has occurred in the screen display as a result of the immediately preceding operation response processing. If a substantial screen change has occurred as a result of the operation input, the determination is satisfied (S305: YES), and printer 100 generates changed image information, which is image information of a changed screen that reflects the change, in step S310, and transmits the changed image information to server 200I in step S315. On the other hand, if the screen change is a supplementary change such as a clock display or remaining ink amount, and there has been no substantial change as a result of the operation input, the determination is not satisfied (S305: NO), and steps S310 and S315 are omitted.

[0074] In addition, if the screen change is not caused by an operation input from the NW terminal 300I to which the changed image information is returned, the printer 100 generates changed image information in step S310 above, including the ID of the device that caused the screen change and the coordinate information of the pressing operation, and transmits the changed image information to the server 200E.

[0075] After receiving the changed image information in step S320, the server 200I transmits the changed image information to the NW terminal 300I in step S325. After receiving the changed image information in step S330, the NW terminal 300I displays a changed screen reflecting the changed image information on the display unit 340 in step S335. Note that even when the changed image information is transmitted or received in steps S325 and S330, the NW terminal 300I obtains the changed image information from the server 200I by polling. As a result, the screen whose contents have changed in response to the operation inputted on the NW terminal 300I is reflected and displayed on the display unit 340 of the NW terminal 300I. The NW terminal 300I then returns to step S265 and repeats the same procedure.

[0076] In addition, in step S335 above, if the changed image information includes coordinate information of the pressing operation and the ID of the device that performed the operation, the NW terminal 300I also displays on the display unit 340 an animation 343 of expanding and disappearing ripples centered on that coordinate position in a color corresponding to that ID.

[0077] On the other hand, if it is determined in the above steps S265 and S275 that the remote operation is to be terminated, the NW terminal 300I transmits a disconnection request to the server 200I in step S340 of FIG. 14, and then displays an end screen indicating that the remote operation has been terminated on the display unit 340 in step S345.

[0078] After receiving the disconnection request in step S350, the server 200I transmits in step S355 a remote operation end request to the printer 100. After receiving the remote operation end request in step S360, the printer 100 executes various processes in step S365 to end the remote operation.

[0079] <Control procedure for local connection> Next, an example of the control procedure executed by the processor 110 of the printer 100 and the CPU 310 of the LC terminal 300L to realize remote operation processing in the local connection configuration will be described with reference to the sequence chart of Fig. 15. Fig. 15 shows the control procedure from when the LC terminal 300L transmits a connection request until a no-operation timeout or a predetermined disconnection request is transmitted to end the remote operation.

[0080] First, in step S405, the user presses the connection key 151 on the WEB browser screen 341 in this example, and the LC terminal 300L transmits to the printer 100 a connection request for starting remote operation of the printer 100. After receiving this remote operation start request in step S410, the printer 100 executes various processes in step S415 to become ready to accept remote operation from the LC terminal 300L.

[0081] Next, printer 100 displays a screen based on the current state on touch panel 140 in step S420, generates image information corresponding to that screen in step S425, and then transmits the image information to LC terminal 300L in step S430. LC terminal 300L receives the image information in step S435, and then displays a screen based on that image information on display unit 340 in step S440. This starts remote operation of printer 100 from LC terminal 300L.

[0082] Thereafter, the LC terminal 300L determines in step S445 whether a no-operation timeout has occurred. Specifically, the LC terminal 300L determines whether a predetermined time has passed without any operation input since the operation input most recently received from the user of the LC terminal 300L. If a no-operation timeout has occurred, the determination is satisfied (S445: YES), and the process proceeds to step S500. On the other hand, if any operation input is received from the user before the no-operation timeout elapses, the determination is not satisfied (S445: NO), and the operation input is accepted in step S450. In the next step S455, the LC terminal 300L determines whether the accepted operation input is an operation for terminating remote operation. If the accepted operation input is an end operation, the determination is satisfied (S455: YES), and the process proceeds to step S500. On the other hand, if the accepted operation input is not an end operation, the determination is negative (S455: NO), and the LC terminal 300L transmits the operation information to the printer 100 in step S460.

[0083] Thereafter, when the display unit 340 is pressed in step S461, the LC terminal 300L displays animation 342 in which a white ripple is enlarged to a predetermined size and then disappears, with the center being the pressed position of the cursor C (see FIG. 9 above). After that, the LC terminal 300L does not accept any operation input until it receives changed image information in step S490 described later.

[0084] After receiving the operation information in step S465, printer 100 executes processing corresponding to the operation information in step S470. Note that even in steps S462 and S463 before receiving the operation information from LC terminal 300L in step S465, if printer 100 receives an operation input from the user of printer 100 via touch panel 140 and operation keys 150 and determines that some operation input has been received, printer 100 proceeds to step S470 and executes processing corresponding to the operation input. Note that this operation response processing also includes processing for displaying a corresponding new screen and generating touch event information.

[0085] Thereafter, in step S475, printer 100 determines whether or not a substantial change has occurred in the screen display as a result of the immediately preceding operation response processing. If a substantial screen change has occurred as a result of the operation input, the determination is satisfied (S475: YES), printer 100 generates image information of a changed screen that reflects the change in step S480, and transmits the changed image information to LC terminal 300L in step S485. On the other hand, if the screen change has only been auxiliary, such as a clock display or remaining ink level, and there has been no substantial screen change as a result of the operation input, the determination is not satisfied (S475: NO), and steps S480 and S485 are omitted.

[0086] Furthermore, if the cause of the screen change is not due to an operational input from the LC terminal 300L to which the changed image information is returned, the printer 100 generates changed image information in step S485 above that includes the ID of the device that caused the screen change and the coordinate information of the press operation, and transmits the changed image information to the LC terminal 300L.

[0087] After receiving the changed image information in step S490, the LC terminal 300L displays a changed screen reflecting the changed image information in step S495 on the display unit 340. As a result, the screen with the changed content corresponding to the operation input on the LC terminal 300L is also reflected and displayed on the display unit 340 of the LC terminal 300L. The LC terminal 300L then returns to step S445 and repeats the same procedure.

[0088] In addition, in step S495 above, if the changed image information includes coordinate information of the pressing operation and the ID of the device that performed the operation, the LC terminal 300L displays on the screen in a color corresponding to that ID, including an animation 343 of expanding and disappearing ripples centered on that coordinate position.

[0089] On the other hand, if the LC terminal 300L determines in steps S445 and S455 above that the remote operation is to be ended, it sends a disconnection request to the printer 100 in step S500, and then displays an end screen indicating that the remote operation has been ended on the display unit 340 in step S505. After receiving the remote operation end request in step S510, the printer 100 executes various processes in step S515 to end the remote operation.

[0090] Note that the screens displayed in steps S30 and S230 above are an example of a first display screen, steps S30 and S230 are an example of a first display process, the image information generated in steps S35 and S235 is an example of a first screen data generation process, steps S40 and S240 are an example of a first transmission process, steps S45 and S245 are an example of a first reception process, steps S50 and S250 are an example of a second transmission process, steps S55 and S255 are an example of a second reception process, and steps S60 and S260 are an example of a second display process.

[0091] Moreover, the above-mentioned changed screen is an example of the second display screen, step S100 is an example of the third display process, the changed image information is an example of the second screen data, steps S110 and S310 are an example of the second screen data generation process, steps S115 and S315 are an example of the third transmission process, steps S120 and S320 are an example of the third reception process, steps S125 and S325 are an example of the fourth transmission process, steps S130 and S330 are an example of the fourth reception process, and steps S135 and S335 are an example of the fourth display process.

[0092] Furthermore, the above steps S70 and S270 are an example of a first operation reception process, the operation information is an example of touch event information, steps S80 and S280 are an example of a first event transmission process, steps S85 and S285 are an example of a first event reception process, steps S90 and S290 are an example of a second event transmission process, and steps S95 and S295 are an example of a second event reception process.

[0093] Moreover, the above steps S105 and S305 are examples of the change identification process.

[0094] Moreover, the process of disabling operational input between steps S81 and S130 is an example of an operational disabling process.

[0095] Moreover, the above steps S92, S292, and S462 are examples of an instruction acceptance process and a device-specific operation acceptance process, steps S100, S300, and S470 are examples of a device-specific event generation process, the screen of FIG. 10 when printer 100 is operated is an example of a third display screen, the process-corresponding screen display in steps S100 and S300 is an example of a fifth display process, the data of the third display screen is an example of third screen data, and steps S110 and S310 are examples of a third screen data generation process. where steps S115 and S315 are an example of a fifth transmission process, steps S120 and S320 are an example of a fifth reception process, steps S125 and S325 are an example of a sixth transmission process, steps S130 and S330 are an example of a sixth reception process, a mode in which the ripple animation is displayed in white is an example of the third mode, a mode in which the ripple animation is displayed in a color other than white that corresponds to printer 100 is an example of the fourth mode, and steps S135 and S335 are an example of a sixth display process.

[0096] In addition, the image information generated in step S425 above is an example of the fourth screen data, step S425 is an example of a fourth screen data generation process, step S430 is an example of a seventh transmission process, step S435 is an example of a seventh reception process, step S440 is an example of a seventh display process, the changed image information is an example of the fifth screen data, step S480 is an example of a fifth screen data generation process, step S485 is an example of an eighth transmission process, step S490 is an example of an eighth reception process, and step S495 is an example of an eighth display process.

[0097] <Effects of the embodiment> As described above, in the remote operation system 1 of this embodiment, the NW terminals 300I, 300E are connected to the printer 100 via the servers 200I, 200E. The screens displayed on the touch panel 140 of the printer 100 in steps S30 and S230 are displayed on the display unit 340 in steps S60 and S260 by the processor 110 of the printer 100 transmitting image information generated in steps S35 and S235 to the NW terminals 300I, 300E via the server 200.

[0098] According to this embodiment, the screen display of the printer 100 and the screen display of the NW terminals 300I and 300E can be linked and remotely controlled via the server 200. The remote control can be performed, particularly for the wide area network GAN, by accessing the URL of the server 200E from the printer 100 and also accessing the URL of the server 200E from the NW terminal 300E. As a result, there is no need to perform complicated processing, as in the case of remote control from a local terminal directly connected to the printer 100, and convenience can be improved. As a result, the user can remotely control the printer 100 from the NW terminals 300I and 300E without performing complicated processing.

[0099] Furthermore, particularly in this embodiment, when the display of the touch panel 140 changes to a changed screen in step S100, changed image information generated by the processor 110 of the printer 100 is transmitted to the NW terminals 300I, 300E via the server 200, and is displayed on the display unit 340 in steps S135 and S335. According to this embodiment, the changed image information is transmitted to the NW terminals 300I, 300E via the server 200 only when the display of the touch panel 140 changes, so that the communication load from the printer 100 to the server 200 and the processing load on the server 200 can be reduced.

[0100] Particularly in this embodiment, by performing an input operation at a predetermined position on the screen displayed on the display unit 340 of the NW terminals 300I, 300E, corresponding operation information is transmitted to the printer 100 via the server 200, causing a corresponding change on the screen, and switching to a changed screen is possible. According to this embodiment, operations on the NW terminals 300I, 300E side can be smoothly transmitted to the printer 100 side, enabling reliable remote operation.

[0101] Particularly in this embodiment, when the screen is switched to a changed screen, it is identified in steps S105 and S305 whether the change is based on operation information corresponding to the operation of the NW terminals 300I and 300E, or is unrelated to the above operation, for example, a change in the time display or a change over time in the consumption status of a consumable. Then, changed image information is generated and transmitted to the server 200 only when it is based on the operation of the NW terminals 300I and 300E, and the changed screen is not transmitted to the server 200 when it is not based on the operation of the NW terminals 300I and 300E. According to this embodiment, the communication load from the printer 100 to the server 200 and the processing load in the server 200 are reduced, and the remote operation process can be executed smoothly.

[0102] Particularly in this embodiment, after an input operation is performed on the screen of the NW terminals 300I and 300E, until a corresponding changed screen is displayed, a new input operation on the display unit 340 is disabled. According to this embodiment, continuous communication from the NW terminals 300I and 300E to the server 200 is prevented, thereby reducing the server load.

[0103] Similarly, from when an input operation is performed on the screen at the NW terminals 300I, 300E until a corresponding changed screen is displayed, operation information corresponding to a new input operation on the display unit 340 is not transmitted to the printer 100. According to this embodiment, continuous communication from the server 200 to the printer 100 is prevented, thereby reducing the communication load. In this case, the process of disabling acquisition of operation information from step S95 until the display in step S135 is an example of an acquisition disabling process.

[0104] Although not shown in the figure, a plurality of NW terminals 300E may simultaneously remotely operate the same printer 100 via the server 200E in the wide area network GAN. For example, assume that both the first NW terminal 300Ea and the second NW terminal 300Eb are configured to be able to remotely operate the printer 100 via the server 200E. When the screen is switched to a changed screen by the first NW terminal 300Ea of the two NW terminals 300Ea remotely operating the printer 100, a white ripple animation 342 corresponding to the first terminal is displayed on the changed screen on the display unit 340 of the first NW terminal 300Ea. At that time, in conjunction with the operation on the first NW terminal 300Ea side, the display unit 340 of the second NW terminal 300Eb also switches from the screen to the changed screen, but the display unit 340 of the second NW terminal 300Eb may display a ripple animation 343 in a color other than the white color (e.g., yellow) corresponding to the other terminal on the changed screen.

[0105] This allows the user to visually easily see whether the image display of each of the multiple NW terminals 300E has been updated by remote operation of the terminal itself or by remote operation of another terminal. The above may also be applied to the case where multiple LC terminals 300L simultaneously remotely operate the same printer 100 in the local area network LAN. The above-mentioned mode in which the ripple animation is displayed in white corresponding to the terminal itself is an example of a first mode, and the mode in which the ripple animation is displayed in a color other than white corresponding to another terminal is an example of a second mode.

[0106] The display of the white ripple animation 342 corresponding to the operation of the own terminal is a limited display for a predetermined time period until the display expands and disappears. It is preferable that the NW terminal 300E of the own terminal does not accept a new input operation while displaying the white ripple animation 342. This can stabilize operability by ignoring unnecessary repeated tapping operations from the user.

[0107] In particular, in this embodiment, the touch panel 140 of the printer 100 is configured to be directly operable by touching or approaching the user's finger. When a screen is displayed by remote operation on the NW terminal 300I, 300E and the screen is switched to a changed screen by direct operation on the touch panel 140, the changed screen is displayed on the display unit 340 of the NW terminal 300I, 300E in conjunction with the screen. At that time, a ripple animation 343 is displayed in a color corresponding to the printer 100 (e.g., green) different from the white color of the own terminal or the color corresponding to the other terminal. According to this embodiment, the user can visually easily understand that the screen display on the NW terminal 300I, 300E has been updated by another user's direct operation on the printer 100, not by remote operation of the own terminal.

[0108] In this embodiment, particularly in the pull-type connection configuration, the printer 100 acquires operation information from the server 200I by a polling technique. Specifically, the server 200I stores the operation information received in step S285 in the storage device 215, and the printer 100 actively queries the server 200I about the presence or absence of operation information at a predetermined time interval, for example, every 5 seconds. If the server 200I has stored operation information in response to the query, the operation information is sent and received in the above steps S290 and S295. This allows communication between the printer 100 and the server 200 to be performed with a minimum necessary communication load.

[0109] Furthermore, in the local type connection configuration, since it is possible to send and receive large volumes of information with little delay, there is no problem in sending and receiving image information by polling at a period of the machine cycle of the printer 100 or a predetermined time interval, for example, every 5 seconds. In this case, operation information corresponding to an operation on the LC terminal 300L side is stored in the memory 320 of the LC terminal 300L. The printer 100 side inquires of the server 200I whether or not the operation information is available at the machine cycle or at the period of the predetermined time interval, and at the time of the inquiry, the operation information from the LC terminal 300L is acquired by the printer 100. According to this, the printer 100 can acquire the operation information from the LC terminal 300L by a so-called polling method, so that the communication between the printer 100 and the LC terminal 300L can be executed with the minimum necessary communication load. In this case, step S450 is an example of a second operation reception process, the process of storing operation information in memory 320 in step S450 is an example of a second event storage process, the machine cycle or the specified time interval of printer 100 is a second specified period, and the process of printer 100 actively inquiring of LC terminal 300L about the presence or absence of operation information at the machine cycle or the specified time interval before step S460 is an example of a second event inquiry process, and the process of transmitting operation information to printer 100 in response to the inquiry process is an example of a third event transmission process.

[0110] In addition, when remotely operated from the LC terminal 300L, image display update processing based on polling processing can be performed, while when remotely operated from the NW terminals 300I and 300E, image display update processing can be performed without being based on polling processing.

[0111] <Modification> The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit and technical concept of the present invention. Such modifications will be described below in order.

[0112] (1) Exclusive execution of remote operations between different connection types In the above embodiment, only the case where one terminal device 300 remotely controls the printer 100 has been described, but multiple terminal devices 300 in the same connection configuration may remotely control the same printer 100 at the same time, for example as shown in FIG. 16.

[0113] In the illustrated example, there are NW terminals 300A and 300B, both of which are in a pull-type connection topology via the same server 200I. The NW terminal 300A first transmits a connection request in step S605, and the server 200I receives it in step S610, and then transmits a remote operation start request to the printer 100 in step S615. The printer 100 receives the remote operation start request in step S620, and starts the remote operation normally in step S625.

[0114] If another NW terminal 300B transmits a connection request to the same server 200I in step S630 while the NW terminal 300A is remotely controlled, the server 200I receives the request in step S635 and accepts the connection request in step S640. That is, in step S640, a connection to the printer 100 is established by the same processes as those in steps S615 to S625, and remote control becomes possible (not shown).

[0115] Thereafter, the NW terminals 300A and 300B, which are in a pull-type connection configuration via the same server 200I, can simultaneously and in parallel remotely operate the printer 100. In the illustrated example, the NW terminal 300A first transmits a disconnection request in step S645, which the server 200I receives in step S650 and terminates the remote operation of the NW terminal 300A. After that, the NW terminal 300B transmits a disconnection request in step S655, which the server 200I receives in step S660 and then transmits a remote operation end request to the printer 100 in step S665. The printer 100 receives the remote operation end request in step S670 and terminates the remote operation normally in step S675.

[0116] As described above, multiple terminal devices 300 in the same connection configuration can simultaneously remotely operate the same printer 100. Although not shown in the figure, multiple terminal devices 300 in a push-type connection configuration via the same server 200E, or multiple terminal devices 300 in the same local connection configuration may simultaneously remotely operate the same printer 100.

[0117] However, the users of the multiple terminal devices 300 connected to the same printer 100 in different connection forms are likely to be in a relationship where there is little need to share information with each other, for example, they belong to different departments or organizations. Therefore, from the viewpoint of preventing information leakage, it is desirable to execute an exclusive connection process so that the multiple terminal devices 300 with different connection forms cannot remotely operate the same printer 100 at the same time. In other words, remote operation is permitted when the multiple terminal devices 300 with different connection forms connect to the printer 100 at different time periods that do not overlap, but even if one terminal device 300 with a different connection form issues a connection request while the other terminal device 300 is already connected, remote operation is not permitted, and the other terminal device 300 cannot connect to the printer 100.

[0118] 17, the NW terminal 300 in the pull-type connection topology first transmits a connection request in step S705, and the server 200I receives the connection request in step S710, and then transmits a remote operation start request to the printer 100 in step S715. If the printer 100 receives the request in step S720, the printer 100 starts the remote operation normally in step S725.

[0119] Even if the LC terminal 300L in the local connection mode sends a connection request to the printer 100 in step S730 while this remote operation is being performed, the printer 100 receives this in step S735 and replies to reject the connection request in step S740. The LC terminal 300L that receives this connection request rejection in step S745 cannot connect to the printer 100.

[0120] The above exclusive execution process is also rejected when the NW terminal 300I in the pull type connection topology transmits a connection request while the LC terminal 300L in the local connection topology is first connected in the reverse order. The same applies when the pull type connection topology is replaced with the push type connection topology (not shown).

[0121] In other words, the processor 110 of the printer 100 exclusively executes one of a first mode-related process including generation, transmission, and display processing of image information and generation, transmission, and display processing of modified image information according to a first mode associated with the NW terminals 300I, 300E, and a second mode-related process including generation, transmission, and display processing of image information and generation, transmission, and display processing of modified image information according to a second mode associated with the LC terminal 300L and executed exclusively from the first mode.

[0122] As described above, in this modification, the first mode-related process when the printer 100 is remotely operated from the NW terminals 300I and 300E via the server 200 and the second mode-related process when the printer 100 is remotely operated from the LC terminal 300L are executed exclusively. This prevents the printer 100 from being remotely operated from the LC terminal 300L when the user is remotely operating the printer 100 from the NW terminals 300I and 300E, and prevents the printer 100 from being remotely operated from the NW terminals 300I and 300E when the user is remotely operating the printer 100 from the LC terminal 300L. According to this modification, by not mixing remote operation via the server 200 and remote operation not via the server 200, it is possible to ensure information security and smoothness of the control process.

[0123] Although not shown in the figure, if a process for obtaining consent between users via a separate application program or the like is possible, multiple users with different connection forms may be allowed to simultaneously remotely operate the same printer 100. Also, for example, if a user who directly operates the printer 100 and a call center operator who is a user of a NW terminal 300E present in the wide area network GAN can simultaneously operate the same printer 100, they can check each other's operation of the printer 100, improving the efficiency and quality of customer service.

[0124] (2) When detecting a long press For example, when a user performs an important operation such as a system reset, a long press operation may be required to continue for a predetermined time or longer, unlike other normal press operations. In order to reliably determine whether such a long press operation has been performed, it is necessary to strictly detect the timing of each of the press input operation and the lift input operation performed consecutively on the same operation key. However, traffic delays are likely to occur in the NW terminal 300E present in the wide area network GAN, and a discrepancy is likely to occur in the timing of detecting each of the press input operation and the lift input operation in the NW terminal 300E from the server 200E side, making it difficult to accurately determine whether the NW terminal 300E has been pressed or not.

[0125] Furthermore, the long press determination process calculates the time interval between the detection timings of the press input operation and the lift input operation, and determines whether the time interval is equal to or greater than a predetermined threshold. However, the threshold itself is often set and stored on the printer 100 side. From the above, at least the NW terminal 300E needs to detect the detection timings of the press input operation and the lift input operation, but the final comparison of the time interval with the threshold may be performed by the printer 100, and the calculation process of the time interval up to that point may be performed by any one of the NW terminal 300E, the server 200E, or the printer 100.

[0126] For example, a sequence chart in the case where the NW terminal 300E performs the calculation process of the time interval is shown in FIG. 18. As shown in the figure, when the NW terminal 300E detects a trigger of a press input operation of a specific operation key from the user in step S805, it immediately starts time measurement using a timer in the CPU 310 in step S810. When it subsequently detects a trigger of a lift input operation in step S815, it immediately ends the time measurement of the timer in step S820. As a result, the time measured by the timer corresponds to the time interval between the press input operation and the lift input operation, which is converted and calculated in normal time units in step S825 and then transmitted to the server 200E in step S830. The server 200E receives the information on the time interval in step S835 and then transmits it to the printer 100 in step S840. The printer 100 receives the information on the time interval in step S845 and then compares the time interval with a threshold value in step S850 to determine whether or not a long press operation has been performed.

[0127] In addition, the detection timing of a press input operation is an example of press input operation time information, the detection timing of a lift input operation is an example of lift input operation time information, special processing such as reset is an example of the first determination processing and the third specified processing, and processing in the case of a normal short press operation is an example of the second specified processing and the fourth specified processing.

[0128] As described above, in this modified example, the time interval during which the NW terminal 300E present in the wide area network GAN is continuously pressed can be accurately measured, and the printer 100 can perform accurate long press determination processing by comparing the measured time interval with a set and stored threshold value.

[0129] Although not particularly shown, the server 200 may determine whether or not a specific operation key is pressed and held down by contact with or proximity to a user's finger on the touch panel 140 of the printer 100. In this case, it is desirable for the server 200 to store a threshold value. In this case, the user's finger is an example of an indicator, and the process of the server 200 comparing the time interval with the threshold value to make a determination is an example of a first long press determination process. This makes it possible to execute a determination process in the case where the touch panel 140 of the printer 100 is directly pressed and held down.

[0130] Furthermore, the sequence charts shown in Figures 11 to 18 do not limit the present invention to the procedures shown in the above sequences, and procedures may be added or deleted, or the order may be changed, etc., within the scope that does not deviate from the spirit and technical concept of the invention.

[0131] In addition to the above, the methods according to the above embodiments and their modifications may be used in appropriate combination.

[0132] Although not specifically illustrated, the present invention can be implemented with various modifications without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0133] 1 Remote Control System 100 Printer (an example of an image processing device) 110 Processor (an example of an image processing control unit) 140 Touch panel (image processing display unit, an example of a touch screen) 200 Servers 200I Server 200E Server 210 Processor (an example of a server control unit) 215 Storage device (an example of a server storage unit) 300 Terminal Equipment 300I NW terminal with pull-type connection 300E Push-type connection network terminal 300L Local connection LC terminal 310 CPU (example of NW terminal control unit, LC terminal control unit, calculation unit) 320 Memory (an example of a terminal memory unit) 340 Display unit (NW terminal display unit, LC terminal display unit) 341 WEB browser screen 342 Ripple animation of your device's operation 343 Ripple animation of other device operation 401 Local Cooperation Department 402 Pull-type Collaboration Department 403 Push-type Collaboration Department LAN (Local Area Network) GAN Wide Area Network

Claims

1. A remote operation system including an image processing device having an image processing control unit and an image processing display unit, a server having a server control unit, and at least one NW terminal having a NW terminal control unit and a NW terminal display unit, the system enabling remote operation of the image processing device via the server by a user operation on the NW terminal, The image processing control unit a first display process for displaying a first display screen on the image processing display unit; a first screen data generation process for generating first screen data indicating the first display screen displayed in the first display process, the first screen data being capable of displaying the first display screen on the NW terminal display unit; a first transmission process of transmitting the first screen data generated in the first screen data generation process to the server; Run The server control unit a first reception process for receiving the first screen data transmitted from the image processing control unit in the first transmission process; a second transmission process of transmitting the first screen data received in the first reception process to the NW terminal; Run The network terminal control unit a second receiving process for receiving the first screen data transmitted from the server control unit in the second transmitting process; a second display process for displaying the first display screen on the NW terminal display unit based on the first screen data received in the second reception process; A remote control system that performs the above.

2. The image processing control unit a third display process for displaying a second display screen obtained by changing the first display screen on the image processing display unit in place of the first display screen; a second screen data generation process for generating, when the second display screen is displayed on the image processing display unit in the third display process, second screen data showing the second display screen, the second screen data being capable of displaying the second display screen on the NW terminal display unit, and not generating, when the second display screen is not displayed on the image processing display unit but the first display screen is displayed; a third transmission process of transmitting the second screen data generated in the second screen data generation process to the server; Run The server control unit a third reception process for receiving the second screen data transmitted from the image processing control unit in the third transmission process; a fourth transmission process of transmitting the second screen data received in the third reception process to the NW terminal; Run The network terminal control unit a fourth reception process for receiving the second screen data transmitted from the server control unit in the fourth transmission process; a fourth display process for switching the first display screen displayed on the NW terminal display unit to the second display screen based on the second screen data received in the fourth reception process; The remote control system according to claim 1 , further comprising:

3. The NW terminal control unit further a first operation receiving process for receiving an input operation to any position on the first display screen displayed on the NW terminal display unit in the second display process; a first event transmission process of transmitting touch event information corresponding to the input operation accepted in the first operation acceptance process to the server; Run The server control unit further a first event receiving process of receiving the touch event information transmitted from the NW terminal control unit in the first event transmitting process; a second event transmission process of transmitting the touch event information received in the first event reception process to the image processing device; Run The image processing control unit further executing a second event receiving process to receive the touch event information transmitted from the server control unit in the second event transmitting process; In the third display process, displaying, on the image processing display unit, the second display screen obtained by changing the first display screen in response to the touch event information received in the second event receiving process; The remote control system according to claim 2.

4. The image processing control unit further execute a change identification process to identify whether the first display screen is the second display screen changed in response to the touch event information or the second display screen is the first display screen changed in response to information other than the touch event information; when it is identified in the change identification process that the first display screen is the second display screen which has changed in response to the touch event information, the second screen data is generated in the second screen data generation process, and the second screen data is transmitted to the server in the third transmission process; when it is identified in the change identification process that the first display screen is the second display screen that has changed in response to information other than the touch event information, the second screen data generation process and the second transmission process are not executed. The remote control system according to claim 3.

5. the image processing display unit of the image processing device is a touch screen, The image processing device further includes a plurality of operation keys that can be pressed, the first display screen and the second display screen include a liquid crystal display of the touch screen and icons of the plurality of operation keys; The touch event information includes at least one of coordinate position information of a pressed operation on the liquid crystal screen and pressed key information indicating which icon of the plurality of operation keys has been pressed. The remote control system according to claim 4.

6. The image processing control unit further a device operation reception process for receiving input operations of the touch screen and the plurality of operation keys; a device-specific event generating process for generating the touch event information corresponding to the input operation received in the device-specific operation receiving process; Run In the third display process, displaying, on the image processing display unit, the second display screen obtained by changing the first display screen in accordance with the touch event information generated in the own device event generation process; The remote control system according to claim 5.

7. The server further includes a server storage unit, The server control unit further executing a first event storage process of storing the touch event information received in the first event reception process in the server storage unit; The image processing control unit A first event inquiry process is executed to send an inquiry about the presence or absence of the touch event information stored in the server storage unit at a first predetermined period. The server control unit when the inquiry from the image processing control unit by the first event inquiry process is received, the touch event information stored in the server storage unit by the first event storage process is transmitted by the second event transmission process. The remote control system according to claim 3.

8. the image processing display unit of the image processing device is a touch screen, The input operation accepted in the first operation acceptance process is a pressing input operation corresponding to contact or proximity of a pointer to the touch screen; a lift input operation corresponding to the pointer being moved away from the touch screen in contact with or in proximity to the touch screen; Including, The touch event information is press event information corresponding to the press input operation, the press event information including press input operation time information; take-off event information corresponding to the take-off input operation, the take-off event information including take-off input operation time information; Including, The server control unit further execute a first long press determination process to determine whether or not an interval between the press input operation time information of the press event information and the lift-off input operation time information of the lift-off event information is equal to or greater than a predetermined threshold, based on the press event information and the lift-off event information included in the touch event information received in the first event reception process; 4. The remote control system of claim 3, wherein, depending on whether the first long press determination process determines that the interval is equal to or greater than the threshold, the second event transmission process transmits to the image processing device touch event information that causes the image processing device to execute a first predetermined process, or touch event information that causes the image processing device to execute a second predetermined process different from the first predetermined process.

9. the image processing display unit of the image processing device is a touch screen, The input operation accepted in the first operation acceptance process is a pressing input operation corresponding to contact or proximity of a pointer to the touch screen; a lift input operation corresponding to the pointer being moved away from the touch screen in contact with or in proximity to the touch screen; Including, The touch event information is press event information corresponding to the press input operation, the press event information including press input operation time information; take-off event information corresponding to the take-off input operation, the take-off event information including take-off input operation time information; Including, The image processing control unit further execute a second long press determination process to determine whether or not an interval between the press input operation time information of the press event information and the lift-off input operation time information of the lift-off event information is equal to or greater than a predetermined threshold, based on the press event information and the lift-off event information included in the touch event information received in the second event reception process; execute a third predetermined process or a fourth predetermined process different from the third predetermined process depending on whether or not it is determined in the second long press determination process that the interval is equal to or greater than a threshold value. The remote control system according to claim 3.

10. The network terminal control unit 4. The remote operation system according to claim 3, further comprising: a process for disabling an operation of the network terminal display unit after the input operation is accepted and before the second display screen is displayed on the network terminal display unit; and a process for disabling an operation of the network terminal display unit after the input operation is accepted and before the second display screen is displayed on the network terminal display unit.

11. 4. The remote control system of claim 3, wherein when the second display screen is displayed on the image processing display unit instead of the first display screen in the third display process in response to the touch event information received in the second event reception process corresponding to the input operation on the NW terminal display unit, the image processing control unit executes an acquisition disable process that makes it impossible to acquire new touch event information corresponding to the new input operation on the NW terminal display unit until the second display screen is displayed on the image processing display unit after the touch event information is received.

12. The remote control system includes: The image processing device is configured to be remotely controlled via the server by a user operation on each of the first NW terminal and the second NW terminal, and the image processing device is configured to be remotely controlled via the server by a user operation on each of the first NW terminal and the second NW terminal, The image processing control unit In the first screen data generation process, the first screen data capable of displaying the first display screen on the NW terminal display unit of each of the first NW terminal and the second NW terminal is generated; In the first transmission process, the first screen data generated in the first screen data generation process is transmitted to the server; The server control unit receiving the first screen data transmitted from the image processing control unit in the first transmission process; In the second transmission process, the first screen data is transmitted to the first NW terminal and the second NW terminal, The NW terminal control unit of each of the first NW terminal and the second NW terminal In the second reception process, the first screen data transmitted from the server control unit in the second transmission process is received; In a second display process, the first display screen is displayed on the NW terminal display unit of the first NW terminal and the second NW terminal; The network terminal control unit of the first network terminal, In the first operation reception process, an input operation is received at any position of the first display screen displayed on the NW terminal display unit of the first NW terminal in the second display process; In the first event transmission process, touch event information corresponding to the input operation accepted in the first operation acceptance process is transmitted to the server; The server control unit further In the first event reception process, the touch event information transmitted from the NW terminal control unit of the first NW terminal in the first event transmission process is received; In the second event transmission process, the touch event information received in the first event reception process is transmitted to the image processing device; The image processing control unit receiving the touch event information transmitted from the server control unit in the second event transmission process; In the third display process, the second display screen obtained by changing the first display screen in response to the touch event information received in the second event receiving process is displayed on the image processing display unit; In the second screen data generation process, the second screen data capable of displaying the second display screen on the NW terminal display unit of each of the first NW terminal and the second NW terminal is generated; In the third transmission process, the second screen data generated in the second screen data generation process is transmitted to the server; The server control unit receiving the second screen data transmitted from the image processing control unit in the third transmission process; In the fourth transmission process, the second screen data received in the third reception process is transmitted to the first NW terminal and the second NW terminal, respectively; The network terminal control unit of the first network terminal, In the fourth reception process, the second screen data transmitted from the server control unit in the fourth transmission process is received; In the fourth display process, based on the second screen data received in the fourth reception process, the first display screen displayed on the NW terminal display unit of the first NW terminal is switched to the second display screen, and the second display screen is displayed in a first mode; The network terminal control unit of the second network terminal, In the fourth reception process, the second screen data transmitted from the server control unit in the fourth transmission process is received; In the fourth display process, based on the second screen data received in the fourth reception process, the first display screen displayed on the NW terminal display unit of the first NW terminal is switched to the second display screen, and the second display screen is displayed in a second aspect different from the first aspect. The remote control system according to claim 3.

13. The network terminal control unit of the first network terminal, The remote control system according to claim 12 , wherein in the display of the second display screen in the fourth display process, the display in the first aspect is performed for a predetermined time only, and the first operation acceptance process is not executed during the predetermined time.

14. the image processing display unit of the image processing device is a touch screen, The image processing control unit further an instruction receiving process for receiving an instruction operation by a pointer touching, approaching, or separating from the touch screen; a fifth display process for displaying, on the image processing display unit, a third display screen obtained by changing the first display screen in response to the instruction operation received in the instruction receiving process, in place of the first display screen in the third display process; a third screen data generation process for generating third screen data indicating the third display screen when the third display screen is displayed on the image processing display unit in the fifth display process, the third screen data being capable of displaying the third display screen on the NW terminal display unit, and not generating the third screen data when the third display screen is not displayed on the image processing display unit but the first display screen is displayed; a fifth transmission process of transmitting the third screen data generated in the third screen data generation process to the server; Run The server control unit further a fifth reception process for receiving the third screen data transmitted from the image processing control unit in the fifth transmission process; a sixth transmission process of transmitting the third screen data received in the fifth reception process to the NW terminal; Run The network terminal control unit a sixth receiving process for receiving the third screen data transmitted from the server control unit in the sixth transmitting process; a sixth display process for switching the first display screen, which has been displayed in a third aspect on the NW terminal display unit in the second display process, to the third display screen based on the third screen data received in the sixth reception process, and displaying the first display screen in a fourth aspect different from the third aspect; The remote control system according to claim 2 , further comprising:

15. The remote control system includes: an LC terminal having an LC terminal control unit and an LC terminal display unit, which is connected to the image processing device without going through the server, and configured to enable remote operation of the image processing device by a user operation of the LC terminal; The image processing control unit a fourth screen data generating process for generating fourth screen data indicating the first display screen displayed in the first display process, the fourth screen data being capable of displaying the first display screen on the LC terminal display unit; a seventh transmission process of transmitting the fourth screen data generated in the fourth screen data generation process to the LC terminal; Run The LC terminal control unit of the LC terminal a seventh reception process for receiving the fourth screen data transmitted from the image processing control unit in the seventh transmission process; a seventh display process for displaying the first display screen on the LC terminal display unit based on the fourth screen data received in the seventh reception process; The image processing control unit further a fifth screen data generation process for generating fifth screen data representing the second display screen and capable of displaying the second display screen on the LC terminal display unit when the second display screen obtained by changing the first display screen is displayed on the image processing display unit in place of the first display screen in the seventh display process, and not generating the fifth screen data when the second display screen is not displayed on the image processing display unit but the first display screen is displayed; an eighth transmission process of transmitting the fifth screen data generated in the fifth screen data generation process to the LC terminal; Run The LC terminal control unit further an eighth reception process for receiving the fifth screen data transmitted from the image processing control unit in the eighth transmission process; an eighth display process for switching the first display screen displayed on the LC terminal display unit of the LC terminal to the second display screen based on the fifth screen data received in the eighth reception process; Run and, The image processing control unit of the image processing device a first mode-related process including the first screen data generation process, the first transmission process, the third display process, the second screen data generation process, and the third transmission process according to a first mode associated with the NW terminal connected to the image processing device via the server; a second mode associated with the LC terminal, the second mode being executed exclusively with the first mode, and including a fourth screen data generation process, a seventh transmission process, a fifth screen data generation process, and an eighth transmission process according to the second mode; 3. The remote control system according to claim 2, wherein one of the above is exclusively executable.

16. The LC terminal further includes a terminal storage unit, The LC terminal control unit further a second operation receiving process for receiving an input operation to any position on the first display screen displayed on the LC terminal display unit in the seventh display process; a second event storage process for storing, in the device storage unit, touch event information corresponding to the input operation accepted in the second operation acceptance process; Run The image processing control unit A second event inquiry process is executed to send an inquiry about the presence or absence of the touch event information stored in the device storage unit at a second predetermined period. The LC terminal control unit further executes a third event transmission process to transmit, when the inquiry from the image processing control unit by the second event inquiry process, the touch event information stored in the device storage unit by the second event storage process to the image processing device. The remote control system according to claim 15.

17. An image processing device having an image processing control unit and an image processing display unit, and being remotely operated via a server by a user operation on a network terminal, The image processing control unit a first display process for displaying a first display screen on the image processing display unit; a first screen data generation process for generating first screen data indicating the first display screen displayed in the first display process, the first screen data being capable of displaying the first display screen on the NW terminal; a first transmission process of transmitting the first screen data generated in the first screen data generation process to the server and causing the NW terminal to display the first display screen based on the first screen data received from the server; a third display process for displaying a second display screen obtained by changing the first display screen on the image processing display unit in place of the first display screen; a second screen data generation process for generating second screen data showing the second display screen when the second display screen is displayed on the image processing display unit in the third display process, the second screen data being capable of displaying the second display screen on the NW terminal, and not generating the second screen data when the second display screen is not displayed on the image processing display unit but the first display screen is displayed; a third transmission process of transmitting the second screen data generated in the second screen data generation process to the server and displaying the first display screen on the NW terminal based on the first screen data received from the server; An image processing device that performs the above.

18. A terminal has a terminal display unit and a computing unit, and the computing unit is capable of remotely operating the image processing device via a server by a user operation, a second receiving process for receiving from the server first screen data indicating a first display screen displayed on an image processing display unit of the image processing device, the first screen data being capable of displaying the first display screen on the terminal display unit; a second display process for displaying the first display screen on the terminal display unit based on the first screen data received in the second reception process; a fourth receiving process for receiving from the server second screen data showing a second display screen to be displayed in place of the first display screen on the image processing display unit of the image processing device, the second display screen being obtained by changing the first display screen, and capable of displaying the second display screen on the terminal display unit; and a fourth display process for switching the first display screen displayed on the terminal display unit to the second display screen based on the second screen data received in the fourth reception process; A remote control program for executing the above.

Images