Image forming system, image forming apparatus, and remote operation program

Abstract

To provide an image forming system, an image forming apparatus, and a remote operation program that allow a user to cause the image forming apparatus to perform a desired operation by using a portable terminal device regardless of the model of the image forming apparatus.SOLUTION: In the operations of an image forming system, a portable terminal device photographs an operation panel of an image forming apparatus to create a photographic image S1, displays the photographic image and detects a user operation for the photographic image S2, specifies the position of the user operation in the photographic image and extracts, from the photographic image, a partial image of a specific range including the position in the photographic image as an operation partial image S3, and transmits operation partial image data indicating the operation partial image to the image forming apparatus S4. The image forming apparatus receives the operation partial image data from the portable terminal device S4, specifies an operation portion in the operation panel corresponding to the operation partial image indicated by the photographed partial image data S5, and executes an operation corresponding to the specified operation portion S6.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an image forming system, an image forming apparatus, and a remote operation program.

Background Art

[0002] In a certain system, a mobile terminal device (a) captures an image of an operation panel of an image forming apparatus and displays the captured image, (b) detects a user operation on the captured image of the operation panel, (c) generates an operation command corresponding to the user operation, and (d) transmits the generated command to the image forming apparatus. When the image forming apparatus receives the operation command, it performs the operation specified by the operation command (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of the above-described system, the mobile terminal device needs to previously hold, as layout data, layout information (information such as the position of the operation screen, the position and size of operation keys, etc.) of the operation panel of the image forming apparatus in order to generate an operation command.

[0005] Generally, since the layout of the operation panel of the image forming apparatus varies depending on the model, the mobile terminal device needs to previously hold different layout data for each model of the image forming apparatus, and installation of the layout data to the mobile terminal device is required for each model of the image forming apparatus, which is inconvenient for the user.

[0006] The present invention has been made in view of the above problems, and an object thereof is to obtain an image forming system capable of causing an image forming apparatus to perform a desired operation using a portable terminal device regardless of the model of the image forming apparatus, as well as an image forming apparatus and a remote operation program that can be used in the image forming system.

Means for Solving the Problems

[0007] The image forming system according to the present invention includes an image forming apparatus having an operation panel and a portable terminal device capable of communicating with the image forming apparatus. The portable terminal device (a) captures an image of the operation panel of the image forming apparatus to generate a captured image, (b) displays the captured image, (c) detects a user operation on the captured image, (d) specifies the position of the user operation in the captured image, (e) extracts, as an operation partial image, a partial image of a specific range including the position in the captured image from the captured image, and (f) transmits operation partial image data indicating the operation partial image to the image forming apparatus. The image forming apparatus (a) receives the operation partial image data from the portable terminal device, (b) specifies an operation part corresponding to the operation partial image indicated by the partial image data on the operation panel, and (c) executes an operation corresponding to the specified operation part. Operation

[0008] The image forming apparatus according to the present invention includes a communication device capable of communicating with a user's portable terminal device, an internal device, and a remote operation processing unit. The remote operation processing unit (a) receives operation partial image data from the portable terminal device using the communication device, (b )Oper in the operation panel, the Operation operation part corresponding to the Su Opera operation partial image indicated by the partial image data is specified, and (c) an operation corresponding to the specified operation part is executed using the internal device. Here, the operation partial image data is image data indicating a partial image of a specific range including the detection position of a user operation in a captured image generated by capturing an image of the operation panel of the image forming apparatus by the portable terminal device. ​

[0009] The remote operation program according to the present invention causes a computer in a portable terminal device including a photographing device, a display device, an input device, and a wireless communication device to (a) photograph an operation panel of an image forming device with the photographing device to generate a photographed image, (b) a photographing processing unit that displays the photographed image on the display device, and (a) detect a user operation on the photographed image with the input device, (b) specify the position of the user operation in the photographed image, (c) extract a partial image of a specific range including the position in the photographed image as an operation partial image from the photographed image, and (f) function as an operation information transmission unit that transmits operation partial image data indicating the operation partial image to the image forming device with the wireless communication device.

Effect of the Invention

[0010] According to the present invention, an image forming system that enables a user to cause a desired operation of an image forming device using a portable terminal device regardless of the model of the image forming device, and an image forming device and a remote operation program that can be used in the image forming system can be obtained.

[0011] The above or other objects, features, and advantages of the present invention will become more apparent from the following detailed description together with the accompanying drawings.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0014] FIG. 1 is a diagram showing the configuration of an image forming system according to an embodiment of the present invention. In the system shown in FIG. 1, a portable terminal device 1 such as a smartphone or a tablet PC is carried by a user who uses the image forming device 2. The portable terminal device 1 and the image forming device 2 can communicate with each other by wireless communication or the like. In this system, a plurality of users each have a plurality of portable terminal devices 1, and a plurality of image forming devices 2 can be used using the portable terminal devices 1.

[0015] FIG. 2 is a block diagram showing the configuration of the portable terminal device 1 in FIG. 1. As shown in FIG. 2, the portable terminal device 1 includes a wireless communication device 11, a display device 12, an input device 13, a storage device 14, a photographing device 15, and an arithmetic processing device 16.

[0016] The wireless communication device 11 is a communication circuit of a predetermined wireless communication standard. The wireless communication device 11 is a wireless LAN interface, a short-range wireless communication device such as Bluetooth (registered trademark), or an interface of a mobile phone network (RF interface, baseband unit, etc.).

[0017] The display device 12 is a device such as a liquid crystal display arranged in the housing of the mobile terminal device 1, and displays various screens for the user. The input device 13 is a device that detects user operations, and includes a touch panel, hard keys, etc. arranged on the display device 12. The storage device 14 is a non-volatile storage device such as a flash memory, and stores programs and data. The imaging device 15 includes an image sensor and an optical system that forms an image of light from a subject on the image sensor, and generates image data of the captured image.

[0018] The arithmetic processing unit 16 is a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., loads a program from the storage device 14 or ROM into the RAM and executes it, and operates as various processing units. In this embodiment, the arithmetic processing unit 16 operates as a shooting processing unit 21 and an operation information transmission unit 22 by executing the remote operation program 14a stored in the storage device 14.

[0019] The shooting processing unit 21 shoots the operation panel 32 of the image forming apparatus 2 with the shooting device 15 to generate a captured image, and (b) displays the captured image on the display device 12.

[0020] Specifically, the shooting processing unit 21 causes the shooting device 15 to start shooting according to a user operation on the input device 13, repeatedly in real time, acquires image data of the captured image (a rectangular image obtained in the field of view of the optical system of the shooting device 15), and displays the captured image based on the image data on the display device 12.

[0021] The operation information transmission unit 22 (a) detects a user operation on the captured image by the input device 13, (b) identifies the position of the user operation in the captured image, (c) extracts, from the captured image, a partial image of a specific range (here, a rectangular area of a predetermined size centered on the detection position of the user operation in the captured image) including the position in the captured image as an operation partial image, and (f) transmits, by the wireless communication device 11, operation partial image data indicating the operation partial image to the image forming apparatus 2.

[0022] Specifically, when the operation information transmission unit 22 detects, at a certain point in time, a predetermined user operation (such as a touch) on the captured image by the input device 13 (for example, a touch panel), it identifies the captured image (still image) at that time and identifies the position of the user operation in the captured image.

[0023] FIG. 3 is a block diagram showing the configuration of the image forming apparatus 2 in FIG. 1. As shown in FIG. 3, the image forming apparatus 2 includes a storage device 31, an operation panel 32, an arithmetic processing device 33, a printing device 34, an image reading device 35, a facsimile device 36, a communication device 37, and the like.

[0024] The storage device 31 is a non-volatile rewritable storage device such as a flash memory. The storage device 31 stores operation site data 31a, which will be described later, in advance.

[0025] The operation panel 32 is disposed on the front side of the upper surface of the housing and includes a display device 32a such as a liquid crystal display, and an input device 32b such as a hard key and a touch panel. The display device 32a displays various screens for the user. Key images displayed on the display device 32a and the touch panel realize soft keys. The input device 32b receives user operations on the hard key or soft key input by the user.

[0026] FIG. 4 is a diagram showing an example of the operation panel 32 of the image forming apparatus 2 in FIGS. 1 and 3. FIG. 5 is a diagram showing an example of the portable terminal device 1 that displays a photographed image of the operation panel 32 shown in FIG. 4. FIG. 6 is a diagram showing an example of an operation partial image extracted from the photographed image of the operation panel 32 shown in FIG. 4. For example, as shown in FIG. 4, the operation panel 32 includes, as an input device 32b, a touch panel 32b1 that causes an icon image to function as a soft key, and hard keys 32b2. Then, according to a predetermined user operation, the portable terminal device 1 uses the photographing device 15 to photograph the operation panel 32 shown in FIG. 4, and displays the photographed image on the display device 12 as shown in FIG. 5, for example. While the photographed image is being displayed, when a certain position in the photographed image is pressed by the user, the portable terminal device 1 detects the pressing with the touch panel, identifies the operation image (still image) that was being displayed at the time the user operation was detected, and also extracts, for example, as shown in FIG. 6, an operation partial image including the position where the user operation (pressing) was detected, and transmits the operation partial image data to the image forming apparatus 2.

[0027] Returning to FIG. 3, the arithmetic processing unit 33 is a computer having a CPU, a ROM, and a RAM, and operates as various processing units by loading a program stored in the ROM or a storage device (not shown) into the RAM and executing it with the CPU. The arithmetic processing unit 33 operates as a controller 41 and a remote operation processing unit 42.

[0028] The printing device 34 is an internal device that prints a document image or the like on printing paper based on image data. Specifically, the printing device 34 outputs a printed product obtained by printing the document image specified by the user's printing request. The image reading device 35 is an internal device that optically reads a document image from a document and generates image data of the document image. The facsimile device 36 is an internal device that generates and transmits a facsimile signal such as a document image based on image data, and receives a facsimile signal from the outside and generates image data from the facsimile signal. The communication device 37 is a wireless or wired network interface, a short-range wireless communication device, etc., and is an internal device that performs data communication with the portable terminal device 1 directly or indirectly via a network.

[0029] The controller 41 controls internal devices such as the printing device 34, the image reading device 35, the facsimile device 36, and the communication device 37 to execute various processes.

[0030] The remote operation processing unit 42 (a) uses the communication device 37 to receive the above-described operation partial image data from the portable terminal device 1, (b) identifies an operation site (a hard key, a soft key, etc. indirectly operated by the user remotely using the portable terminal device 1) corresponding to the operation partial image indicated by the shooting partial image data on the operation panel 32, and (c) uses the above-described internal device to cause the controller 41 to execute an operation corresponding to the identified operation site.

[0031] For example, the remote operation processing unit 42 searches for the image pattern of each operation site indicated by the operation site data 31a in the operation partial image indicated by the shooting partial image data by an existing pattern matching method, and detects the operation site where the image pattern is detected in the operation partial image as the operation site indirectly operated by the user.

[0032] Further, for example, the remote operation processing unit 42 may derive the feature amounts of both the operation partial image indicated by the captured partial image data and the images of the respective operation parts indicated by the operation part data 31a by an existing method, compare the feature amounts of both, and identify the operation part corresponding to the operation partial image indicated by the captured partial image data based on the comparison result. For example, regarding the hard keys and soft keys as operation parts, this feature amount includes the shape and color of the keys, characters (including symbols) and character colors within the keys, etc. Note that regarding characters, they are converted into character codes by image recognition processing and compared.

[0033] Next, the operation of the above image forming system will be described. FIG. 7 is a sequence diagram for explaining the operation of the image forming system shown in FIG. 1.

[0034] In the mobile terminal device 1, the imaging processing unit 21 starts imaging processing using the imaging device 15 according to a user operation on the input device 13 (step S1). As a result, for example, as shown in FIG. 5, the captured image captured by the imaging device 15 is displayed on the display device 12 in real time.

[0035] During the display of the captured image, the operation information transmission unit 22 monitors whether a predetermined user operation (such as a touch) is detected on the input device 13 for the captured image. At a certain point in time, when such a user operation is detected (step S2), the captured image (still image) at that time and the position of the user operation in the captured image are identified. For example, as shown in FIG. 6, a partial image of a specific range including the position is extracted from the captured image as an operation partial image (step S3), and operation partial image data indicating the operation partial image is transmitted to the image forming device 2 using the wireless communication device 11 (step S4). At that time, the operation information transmission unit 22 presents the specific range on the captured image. For example, the operation information transmission unit 22 presents the specific range by superimposing a frame image indicating the specific range on the captured image and displaying it on the display device 12, or by changing the display mode (such as brightness and color) of the specific range in the captured image.

[0036] In the image forming apparatus 2, the remote operation processing unit 42 monitors whether operation partial image data is received from the portable terminal device 1 by the communication device 37. When the operation partial image data is received, based on the received operation partial image data and the operation site data 31a, as described above, the operation site corresponding to the operation partial image is specified (step S5).

[0037] When the remote operation processing unit 42 specifies the operation site where the user operation has been made indirectly in this way, it notifies the controller 41 that the operation site has been operated, and causes the controller 41 to execute the operation corresponding to the operation (such as screen transition, execution of various processes (printing, etc.) using internal devices) (step S6). Note that when the operation site corresponding to the operation partial image data is not detected, the operation partial image data is ignored.

[0038] As described above, according to the above embodiment, the portable terminal device 1 (a) captures the operation panel 32 of the image forming apparatus 2 to generate a captured image, (b) displays the captured image, (c) detects a user operation on the captured image, (d) specifies the position of the user operation in the captured image, (e) extracts a partial image of a specific range including the position in the captured image as an operation partial image from the captured image, and (f) transmits operation partial image data indicating the operation partial image to the image forming apparatus 2. The image forming apparatus 2 (a) receives the operation partial image data from the portable terminal device 1, (b) specifies the operation site corresponding to the operation partial image indicated by the captured partial image data on the operation panel 32, and (c) executes the operation corresponding to the specified operation site.

[0039] As a result, since the portable terminal device 1 does not need to hold layout data, regardless of the model of the image forming apparatus 2, the user can use the portable terminal device 1 to cause the image forming apparatus 2 to perform a desired operation. In addition, since the user can use his or her own portable terminal device 1 to cause the image forming apparatus 2 to perform a desired operation, there is no need to touch the operation panel 32 of the image forming apparatus 2, and it is possible to suppress the contact infection of infectious diseases caused by viruses and the like in recent years.

[0040] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing the intended advantages. That is, it is intended that such changes and modifications be included within the scope of the claims.

[0041] For example, in the above embodiment, the remote operation program 14a may be stored in a portable computer-readable recording medium or a server, read from the recording medium or the server, installed in the portable terminal device 1, and stored in the storage device 14.

Industrial Applicability

[0042] The present invention is applicable to, for example, an image forming apparatus.

Explanation of Signs

[0043] 1 Portable terminal device 2 Image forming apparatus 11 Wireless communication device 12 Display device 13 Input device 14a Remote operation program 15 Photographing device 21 Photographing processing unit 22 Operation information transmission unit 31 Storage device 31a Operation part data 32 Operation panel 34 Printing device (an example of an internal device) 35 Image reading device (an example of an internal device) 36 Facsimile device (an example of an internal device) 37 Communication device (an example of an internal device) 42 Remote operation processing unit

Claims

1. An image forming apparatus including an operation panel, and a mobile terminal device capable of communicating with the image forming apparatus, wherein the mobile terminal device: (a) captures an image of the operation panel of the image forming apparatus to generate a captured image; (b) displays the captured image; (c) detects a user operation on the captured image; (d) specifies the position of the user operation in the captured image; (e) extracts, as an operation partial image, a partial image of a specific range including the position in the captured image from the captured image; (f) transmits operation partial image data indicating the operation partial image to the image forming apparatus, and the image forming apparatus: (a) receives the operation partial image data from the mobile terminal device; (b) specifies an operation part corresponding to the operation partial image indicated by the operation partial image data on the operation panel; and (c) executes an operation corresponding to the specified operation part. An image forming system characterized by the above.

2. A communication device capable of communicating with a user's mobile terminal device, an internal device, and a remote operation processing unit, wherein the remote operation processing unit: (a) receives operation partial image data from the mobile terminal device using the communication device; (b) specifies an operation part corresponding to the operation partial image indicated by the operation partial image data on the operation panel; and (c) executes an operation corresponding to the specified operation part using the internal device, and the operation partial image data is image data indicating a partial image of a specific range including a detection position of a user operation in a captured image generated by capturing the operation panel of the image forming apparatus by the mobile terminal device. An image forming apparatus characterized by the above.

3. Further including a non-volatile storage device, wherein the storage device stores in advance operation part data indicating operation parts on the operation panel of the image forming apparatus. The remote operation processing unit identifies the operation site by referring to the operation site data. The image forming apparatus according to claim 2, characterized in that.

4. A computer in a mobile terminal device including a photographing device, a display device, an input device, and a wireless communication device, (a) A photographing processing unit that photographs an operation panel of an image forming apparatus with the photographing device to generate a photographed image, and (b) displays the photographed image on the display device, and (a) Detect a user operation on the photographed image with the input device, (b) identify the position of the user operation in the photographed image, (c) extract a partial image of a specific range including the position in the photographed image as an operation partial image from the photographed image, and (f) an operation information transmission unit that transmits operation partial image data indicating the operation partial image to the image forming apparatus with the wireless communication device A remote operation program that functions as.

Images