Information processing device and information processing program

Abstract

To enable a user to visually recognize a correspondence relation between a distance from a screen and operation contents different depending on the distance when performing a contactless operation on the screen.SOLUTION: An image processing apparatus 10 comprises a CPU 41. When detecting an object under contactless operation on a screen, the CPU 41 causes a first frame centered at a detected position to be displayed depending on a distance between the screen and the object and causes a plurality of second frames centered at the detected position and different from the first frame in display form to be displayed in association with a plurality of areas which are delimited depending on the distances from the screen and to which different operation contents are preliminarily assigned for the contactless operation.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing program.

Background Art

[0002] For example, Patent Document 1 describes a method for preventing soiling of a display screen by a user's finger. This method has a step of determining a first threshold distance from the display screen, and the user's finger is detected within the first threshold distance. This method further has a step of determining a second threshold distance from the display screen, and the user's finger is detected within the second threshold distance, where the second threshold distance is shorter than the first threshold distance. This method further has a first detection step of detecting that the user's finger is within the first threshold distance, and a first notification step of notifying the user, in response to the detection in the first detection step, that the user's finger has entered a region near a screen where interaction with a graphical user interface is made possible. The graphical user interface enables various display functions to be operated by the movement of the user's finger. This method further has a second detection step of detecting that the user's finger is within the second threshold distance, and a second notification step of notifying the user, in response to the detection in the second detection step, that the user's finger has entered a prohibited region.

[0003] Furthermore, Patent Document 2 describes an input support device that provides feedback to the user when non-contact operation is performed without using a controller. This input support device includes a spatial coordinate acquisition means that continuously acquires spatial coordinates of a predetermined part of the user via a detection unit, and a display control means that performs display control to move a marker to a corresponding position on the display screen of a display unit based on the spatial coordinates updated by the spatial coordinate acquisition means. The initial shape of this marker is a plurality of shapes obtained by dividing an arbitrary shape. The display control means moves the plurality of shapes further apart from each other at the corresponding position on the display screen as the spatial coordinate distance to a predetermined plane in the coordinate system of the spatial coordinates increases, and moves the plurality of shapes closer to each other at the corresponding position on the display screen as the distance decreases. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Special Publication No. 2008-505381 [Patent Document 2] Japanese Patent Publication No. 2014-219938 [Overview of the project] [Problems that the invention aims to solve]

[0005] By the way, there is a technology that allows users to operate the screen without touching it. In this contactless operation, when the user holds an object such as their hand or finger over the screen, the position of the object is detected, and the operation (function) corresponding to the detected position is executed.

[0006] In some cases, the operation is determined by the difference in distance from the screen. However, with contactless operation, the operation takes place in mid-air, making it difficult for the user to visually determine the distance from the screen, and also difficult to determine the operation corresponding to that distance.

[0007] This disclosure aims to provide an information processing device and an information processing program that allow a user to visually grasp the correspondence between the distance from the screen and the different operation content depending on that distance when performing contactless operations on the screen. [Means for solving the problem]

[0008] To achieve the above objective, the information processing device according to the first embodiment includes a processor, which, when it detects an object to be operated on a screen without contact, displays a first frame centered on the detected position according to the distance between the screen and the object, and displays a plurality of second frames, each having a different display format from the first frame, centered on the detected position, corresponding to the plurality of regions which are divided according to the distance from the screen and each region is pre-assigned different operation content for the non-contact operation.

[0009] Furthermore, in the information processing device according to the second embodiment, the first frame and the plurality of second frames differ in at least one of the line type and color.

[0010] Furthermore, in the information processing device according to the third embodiment, the information processing device according to the second embodiment has a plurality of second frames, in which at least one of the line type and color differs depending on the corresponding area.

[0011] Furthermore, the information processing device according to the fourth embodiment is an information processing device according to any one of the first to third embodiments, wherein the size of the first frame displayed when the distance between the screen and the object is a first distance is smaller than the size of the frame displayed when the distance is a second distance which is longer than the first distance, and the size of the second frame is such that the frame corresponding to the area closest to the screen is smallest, and the frame corresponding to the area furthest from the screen is largest.

[0012] Furthermore, the information processing device according to the fifth embodiment is an information processing device according to any one of the first to third embodiments, wherein the size of the first frame displayed when the distance between the screen and the object is a first distance is larger than the size of the frame displayed when the distance is a second distance which is longer than the first distance, and the size of the second frame is such that the frame corresponding to the area closest to the screen is the largest, and the frame corresponding to the area furthest from the screen is the smallest.

[0013] Furthermore, in the information processing device according to the sixth embodiment, the first frame and the plurality of second frames are displayed concentrically with respect to the detected position, as in the information processing device according to the fourth or fifth embodiment.

[0014] Furthermore, in the information processing device according to the seventh embodiment, in the information processing device according to any one of the first to sixth embodiments, the amount of movement per unit time of the first frame corresponding to the detected position is smaller than the amount of movement per unit time of the object.

[0015] Furthermore, in the information processing device according to the eighth embodiment, if the processor in the information processing device according to any one of the first to seventh embodiments detects that the object is located in the area closest to the screen among the plurality of areas, it notifies that the object is located in an area close to the screen.

[0016] Furthermore, in the information processing device according to the ninth embodiment, if the processor detects a specific action by the object within any one of the multiple areas, it fixes the operation content to the area where the specific action was detected, regardless of the distance between the screen and the object.

[0017] Furthermore, in the information processing device according to the tenth embodiment, in the information processing device according to the ninth embodiment, the specific operation is an operation that changes the shape of the object from a first shape to a second shape, and if the processor detects an operation that changes the shape from the first shape to the second shape and then detects an operation that returns the shape from the second shape to the first shape, the fixation is released.

[0018] Furthermore, in the information processing device according to the 11th embodiment, in the information processing device according to any one of the first to tenth embodiments, when the processor displays the plurality of second frames, it displays character information indicating the operation content assigned to each of the plurality of areas corresponding to the plurality of second frames.

[0019] Furthermore, in the information processing device according to the 12th embodiment, in the information processing device according to any one of the first to tenth embodiments, when the processor displays the plurality of second frames, it selectively displays character information indicating the operation content assigned to the area where the object is detected from among the plurality of areas corresponding to the plurality of second frames.

[0020] Furthermore, in order to achieve the above objective, the information processing program according to the 13th embodiment causes the computer to, when it detects an object to be operated on the screen without contact, to display a first frame centered on the detected position according to the distance between the screen and the object, and to display a plurality of second frames, each with a different display format from the first frame, centered on the detected position, corresponding to the plurality of regions which are divided according to the distance from the screen and each region which has different operation content assigned to it in advance for the non-contact operation. [Effects of the Invention]

[0021] According to the first and thirteenth embodiments, when performing contactless operations on the screen, the user can visually grasp the correspondence between the distance from the screen and the different operations performed depending on that distance.

[0022] According to the second aspect, there is an effect that the first frame and the plurality of second frames can be visually recognized.

[0023] According to the third aspect, there is an effect that each of the plurality of second frames can be visually recognized.

[0024] According to the fourth aspect, there is an effect that the correspondence relationship between the first frame and the second frame can be visually recognized.

[0025] According to the fifth aspect, similar to the fourth aspect, there is an effect that the correspondence relationship between the first frame and the second frame can be visually recognized.

[0026] According to the sixth aspect, there is an effect that the first frame and the plurality of second frames can be visually recognized regardless of the position where the object is detected.

[0027] According to the seventh aspect, there is an effect that fine operations that are difficult in non-contact operations can be easily performed as compared with the case where the amount of movement per unit time for each of the first frame and the object is made the same.

[0028] According to the eighth aspect, there is an effect that the object can be prevented from touching the screen.

[0029] According to the ninth aspect, there is an effect that the occurrence of misoperations in non-contact operations can be suppressed as compared with the case where the operation content is changed according to the distance from the screen to the object.

[0030] According to the tenth aspect, there is an effect that the fixed operation content can be released only by restoring the shape of the object.

[0031] According to the eleventh aspect, there is an effect that the operation content assigned to the plurality of regions corresponding to the plurality of second frames can be easily grasped as compared with the case where only the plurality of second frames are displayed.

[0032] According to the twelfth embodiment, compared to the case where only multiple second frames are displayed, the operation content assigned to the area where the object was detected can be easily grasped without reducing visibility. [Brief explanation of the drawing]

[0033] [Figure 1] This figure shows an example configuration of an information processing system that includes an information processing device equipped with a contactless user interface that allows users to operate the device without physical contact. [Figure 2] This is a perspective view showing the main parts of the image processing apparatus according to the embodiment. [Figure 3] (A) is a cross-sectional view of the control panel, and (B) is a plan view of the control panel as seen from a position opposite the display surface of the control panel. [Figure 4] This figure shows an example of the functional configuration of an image processing apparatus according to the embodiment. [Figure 5] This diagram illustrates screen transitions, showing how the screens displayed on the control panel change based on user actions on the control panel. [Figure 6] This diagram shows an example of a user selecting the copy button, which is an example of an icon image, on the home screen. [Figure 7] This is a block diagram showing an example of the main components of the electrical system of an image processing apparatus according to an embodiment. [Figure 8] This is a block diagram showing an example of the functional configuration of an image processing apparatus according to the first embodiment. [Figure 9] This diagram schematically shows an example of a spatial region when the operation panel according to the embodiment is viewed from the side. [Figure 10] (A) is a schematic diagram showing an example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing an example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 11] This figure shows another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 12](A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 13] (A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 14] (A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 15] (A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 16] (A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 17] (A) is a schematic diagram showing yet another example of multiple spatial regions when the control panel is viewed from the side, and (B) is a diagram showing yet another example of the first frame and multiple second frames when the control panel is viewed from the front. [Figure 18] This flowchart shows an example of the processing flow by the information processing program according to the first embodiment. [Figure 19] This is a block diagram showing an example of the functional configuration of an image processing apparatus according to the second embodiment. [Figure 20] This figure shows an example of operation fixed shape setting information according to the second embodiment. [Figure 21] This figure shows the screen transitions in response to the user's hand movements in the comparative example. [Figure 22] This figure shows an example of how the screen transitions in response to the user's hand movements according to the second embodiment. [Figure 23] This flowchart shows an example of the processing flow by the information processing program according to the second embodiment. [Modes for carrying out the invention]

[0034] Hereinafter, an example of an embodiment for carrying out the technology of this disclosure will be described in detail with reference to the drawings. Components and processes that perform the same operation, action, or function are given the same reference numerals throughout the drawings, and redundant explanations may be omitted as appropriate. Each drawing is only a schematic representation to the extent that the technology of this disclosure can be fully understood. Therefore, the technology of this disclosure is not limited to the illustrated examples. Furthermore, in this embodiment, explanations of configurations not directly related to the present invention or well-known configurations may be omitted.

[0035] [First Embodiment] Figure 1 shows an example configuration of an information processing system 1, which includes an information processing device equipped with a contactless user interface that allows users to operate the system without physical contact.

[0036] The information processing device in Information Processing System 1 may be any device applicable to any field, as long as it is equipped with a contactless user interface. Examples of information processing devices include image processing devices, ATMs (Automatic Teller Machines), vending machines, and ticket dispensers. The information processing device may be a device for personal use or a device used by an unspecified number of users.

[0037] Referring to Figures 1 and 2, an image processing device 10 installed in a workplace or similar location will be described as an example of an information processing device.

[0038] Figure 2 is a perspective view showing the main parts of the image processing apparatus 10 according to this embodiment.

[0039] The image processing device 10 is a device that performs image-related functions according to user instructions, as will be explained later. The image processing device 10 is connected, for example, to multiple terminals 4 used by each user via a communication line 2.

[0040] The user sends the image data generated on terminal 4 to the image processing device 10 via communication line 2, thereby causing the image processing device 10 to perform the desired image processing. Alternatively, the user may store the image data on a portable storage medium such as a USB (Universal Serial Bus) memory or memory card, move it to the image processing device 10, and connect the portable storage medium to the image processing device 10, thereby causing the image processing device 10 to perform the desired image processing. Furthermore, the user may move a document 11 containing at least one of text and / or an image to the image processing device 10, and have the image processing device 10 read the document 11, thereby causing the image processing device 10 to perform the desired image processing.

[0041] There are no restrictions on the connection configuration of the communication line 2 connecting the image processing device 10 and the terminal 4; it may be wired, wireless, or a combination of wired and wireless. Furthermore, there are no restrictions on the number of terminals 4 connected to the image processing device 10; for example, it is not necessary for any terminals 4 to be connected to the image processing device 10.

[0042] Terminal 4 refers to an information device used by the user. Terminal 4 can be any type of information device as long as it has data storage and data communication functions. Terminal 4 includes, for example, computers that are not intended to be carried around, as well as mobile terminals that are intended to be carried around, such as smartphones and wearable devices.

[0043] As shown in Figure 2, the image processing device 10, as an example, has a scanning function that reads an image written on a recording medium such as paper as image data, a printing function that forms an image represented by the image data on a recording medium, and a copying function that forms an image identical to the image formed on the recording medium on another recording medium. The copying function, printing function, and scanning function are examples of image processing in the image processing device 10.

[0044] As shown in Figure 2, the image processing apparatus 10 is equipped with, for example, a document reading unit 12 at its upper part, and an image forming unit 14 positioned below the document reading unit 12.

[0045] The document scanning unit 12 comprises an optical scanning device (not shown) and a document transport device 18 located within the document cover 16. The document transport device 18 sequentially pulls in documents 11 placed on the document tray 16A provided in the document cover 16 and transports them onto a document scanning glass (not shown). The document scanning unit 12 reads the contents of the documents 11 transported onto this document scanning glass as image data using the optical scanning device. After that, the document transport device 18 discharges the documents 11, whose contents have been read, onto an output tray 16B provided in the document cover 16.

[0046] Meanwhile, the image forming unit 14 forms an image represented by image data on a recording medium stored in a storage shelf 19, which is classified according to the type and size of the recording medium. There are no restrictions on the color of the image that the image forming unit 14 forms on the recording medium; it may be a color image or a monochrome image.

[0047] An operation display unit 13 is provided on the front of the image processing device 10 to receive instructions from the user for performing various functions such as copying, printing, and scanning.

[0048] Specifically, the operation display unit 13 includes a reader device 17 that acquires information about the user performing the operation, and an operation panel 15 that accepts user input.

[0049] The reader device 17 is a device that, for example, when an employee ID card owned by a user is brought close to it, reads identification information ("User ID") that uniquely identifies the user from the IC chip embedded in the employee ID card in a contactless manner.

[0050] The operation panel 15 is a display with a superimposed touch panel, and the operation panel 15 displays icon images representing items that the user will operate on to perform the desired function. There are no restrictions on the type of icon image as long as it represents something that the user will operate on, and icon images include, for example, buttons, scroll bars, checkboxes, and radio buttons. When the user operates on an icon image, the image processing device 10 executes a pre-associated process for the operation, and the response to the operation is displayed on the operation panel 15.

[0051] Figure 3 shows an example of an operation panel 15 that detects the user's operating position 6 without contact. Figure 3(A) is a cross-sectional view of the operation panel 15, and Figure 3(B) is a plan view of the operation panel 15 as seen from a position opposite the display surface of the operation panel 15.

[0052] The control panel 15 detects the position of the user's finger, i.e., the operating position 6, without contact. "Detecting the operating position 6 without contact" means that the position of the user's finger can be detected simply by holding the finger over a space on the display surface of the control panel 15, away from the display surface, without the user having to press their finger against the display surface of the control panel 15. Hereafter, the space on the display surface of the control panel 15, within the range of the display surface of the control panel 15, will be referred to as "on the control panel 15". Furthermore, "holding a finger over the control panel 15" means that the user points their finger over the control panel 15 without touching the display surface of the control panel 15.

[0053] The control panel 15 is equipped with a so-called capacitive touch panel that detects the operation position 6 from the change in capacitance caused by the user placing their finger over the control panel 15. In a control panel 15 equipped with such a touch panel, the change in capacitance at the location closest to the user's finger will be greater than the change in capacitance at other locations. Therefore, the control panel 15 outputs the location with the largest change in capacitance within the range of the control panel 15 as the user's operation position 6.

[0054] To identify the user's operating position 6 on the control panel 15, the control panel 15 has a defined operating coordinate system for defining a detection area that non-contactually detects the position of the user's finger. The operating coordinate system is represented as a three-dimensional coordinate system with the origin P at any point on the control panel 15. In the example of the control panel 15 shown in Figures 3(A) and 3(B), the origin P is set at one of the vertices on the rectangular contour of the control panel 15. Furthermore, in the example of the control panel 15 shown in Figures 3(A) and 3(B), the X-axis is set along the horizontal direction of the control panel 15 with respect to the origin P, the Y-axis is set along the vertical direction of the control panel 15, and the Z-axis is set perpendicular to the X-axis and Y-axis, respectively. The direction along the Z-axis is called the height direction of the control panel 15.

[0055] The user's operating position 6 on the control panel 15 is represented by a coordinate point (x,y) which is a combination of the X coordinate value x and the Y coordinate value y at the point within the range of the control panel 15 where the change in capacitance is greatest.

[0056] When an icon image is displayed on the control panel 15, the icon image displayed so as to include the user's operation position 6 is recognized as the icon image being operated by the user. In the example of the control panel 15 shown in Figure 3(B), the user's operation position 6 is included within the area of ​​the button 8 located on the screen 30 displayed on the control panel 15, so the user is recognized as operating the button 8. Hereafter, an icon image displayed so as to include the user's operation position 6 may be referred to as the "icon image corresponding to operation position 6". Operation position 6 is also an example of a "detection position where user operation is detected" according to this embodiment.

[0057] As shown in Figure 3(A), the length of the perpendicular line drawn from the user's hand 3, which is held over the control panel 15, to the display surface of the control panel 15, that is, the height distance of the control panel 15 from the user's hand 3 to the control panel 15, is defined as the "operating distance D". The user's hand 3 is an example of an object held by the user. This object may be a part of the user other than their hand or fingers, or it may be a stylus owned by the user. On the control panel 15, as the operating distance D decreases, the change in capacitance at the user's operating position 6 increases. Conversely, on the control panel 15, as the operating distance D increases, the change in capacitance at the user's operating position 6 decreases. Therefore, by pre-establishing a relationship between the operating distance D and the change in capacitance, the operating distance D can be obtained from the change in capacitance on the control panel 15.

[0058] By using the correspondence between the operating distance D and the change in capacitance, the control panel 15 can recognize the user's operating position 6 not only as a two-dimensional operating position 6 along the display surface of the control panel 15, but also as a three-dimensional operating position 6 that takes the operating distance D into consideration. That is, when the user's operating position 6 is represented in three dimensions, the user's operating position 6 is represented by a coordinate point (x,y,z) which is formed by combining the coordinate value z, which represents the operating position 6 in the height direction of the control panel 15, with the coordinate point (x,y). The coordinate value z is the coordinate value of the Z axis at a position located a distance D from the origin P along the Z axis.

[0059] Furthermore, when the coordinate value z=0, it means that the user is operating the control panel 15 by touching its display surface with their finger. Therefore, the image processing device 10 can also recognize differences in the user's operation style, such as whether the user is operating the control panel 15 without contact or by touching the control panel 15 with their finger. In this way, the control panel 15 supports both contact operations, where the user touches the display surface of the control panel 15 with their finger, and non-contact operations, where the user holds their finger over the control panel 15.

[0060] As described above, in the control panel 15, the change in capacitance at the user's operating position 6 becomes smaller as the operating distance D increases, so there is an upper limit to the operating distance D. Even if the user holds their finger over the control panel 15 at a position beyond the upper limit of the operating distance D, the capacitance at the user's operating position 6 will no longer change, and therefore the control panel 15 will not respond to the user's operation.

[0061] The detection area corresponding to the icon image is defined as an area in space approximately 3 cm away from the operation panel 15. In other words, when the user brings their hand 3 within approximately 3 cm of the operation panel 15, the capacitance of the icon image changes, and non-contact input is detected. At this time, the XYZ coordinates of the hand 3 within the detection area are acquired as the operation position 6. Then, when the user brings their hand 3 even closer than 3 cm, the XYZ coordinates at that position are acquired.

[0062] Figure 4 shows an example of the functional configuration of the image processing apparatus 10 according to this embodiment. The image processing apparatus 10 includes the following functional units: a control unit 20, a reception unit 21, a display unit 22, a document reading unit 23, and an image forming unit 24.

[0063] The reception unit 21 receives the user ID of the user operating the image processing device 10 from the reader device 17 of the operation display unit 13, and also receives the user's operating position 6 on the operation panel 15 from the operation panel 15 of the operation display unit 13. The reception unit 21 also receives image data from the terminal 4 or a portable storage medium connected to the image processing device 10. The reception unit 21 notifies the control unit 20 of the received user ID, the user's operating position 6, and the image data.

[0064] When the control unit 20 receives a user ID from the reception unit 21, it performs an authentication process to determine whether the user represented by the user ID is a user authorized to use the image processing device 10 (referred to as a "registered user"). When the control unit 20 receives notification from the reception unit 21 of the user's operating position 6 on the operation panel 15, it determines whether the icon image displayed at the user's operating position 6 on the screen 30 displayed on the operation panel 15 has been selected, and executes a process pre-associated with the selected icon image. For example, if the icon image is a button 8 to start the print function, the control unit 20 starts the print function to form an image represented by the image data received by the reception unit 21 onto the recording medium.

[0065] Since the image processing device 10 has copy, print, and scan functions, the control unit 20 includes a scan control unit 20A for controlling the scan function, a print control unit 20B for controlling the print function, and a copy control unit 20C for controlling the copy function. Depending on the content of the process associated with the icon image operated by the user, control is performed by one of the scan control unit 20A, print control unit 20B, or copy control unit 20C. Although not shown in the figures, the image processing device 10 may also have a facsimile function, in which case the control unit 20 includes a facsimile control unit for controlling the facsimile function.

[0066] If the user's operation via the icon image relates to the scanning function, the scan control unit 20A executes the scanning function by controlling the document reading unit 23. If the user's operation via the icon image relates to the printing function, the print control unit 20B executes the printing function by controlling the image forming unit 24. If the user's operation via the icon image relates to the copying function, the copy control unit 20C generates image data of the document 11 by controlling the document reading unit 23. Subsequently, the copy control unit 20C forms an image represented by the generated image data on the recording medium by controlling the image forming unit 24.

[0067] The document reading unit 23 drives the document reading unit 12 according to the control of the scan control unit 20A and the copy control unit 20C, and performs, for example, transporting the document 11 placed on the document tray 16A and generating image data of the transported document 11.

[0068] The image forming unit 24 drives the image forming unit 14 according to the control of the print control unit 20B and the copy control unit 20C, for example, transporting the recording media stored in the storage shelf 19 and forming an image represented by the image data on the transported recording media.

[0069] The display unit 22, in accordance with instructions from the control unit 20, displays, for example, the results of the authentication process for the user and the results of the processing performed by the control unit 20 in response to user operations via icon images, on the operation panel 15 in the operation display unit 13.

[0070] Figure 5 is a diagram illustrating an example of screen transitions in which the screens 30 displayed on the control panel 15 change in response to user operations on the control panel 15.

[0071] Although the display unit 22 displays the screen 30 on the operation panel 15, the display unit 22 displays the screen 30 according to the instructions of the control unit 20, so it can also be said that the control unit 20 displays the screen 30 on the operation panel 15. Furthermore, the space along the Z-axis with the display range of the screen 30 displayed on the operation panel 15 as the base is referred to as "above the screen 30," and the space along the Z-axis with the display range of the icon image displayed within the screen 30 as the base is referred to as "above the icon image." The notations "above the screen 30" and "above the icon image," like the notation "above the operation panel 15," do not mean the upper side based on up, down, left, and right in real space, but rather refer to the space in the direction opposite to the screen 30 and the space in the direction opposite to the icon image, respectively.

[0072] For the sake of clarity, when describing the different types of screen 30, an alphabetical code corresponding to each type of screen 30 will be added after "screen 30". If it is not necessary to describe the different types of screen 30, all types of screen 30 will be collectively referred to as "screen 30". Similarly, when describing the different types of button 8, which are an example of an icon image, an alphabetical code corresponding to each type of button 8 will be added after "button 8". If it is not necessary to describe the different types of button 8, all types of button 8 will be collectively referred to as "button 8".

[0073] If the control unit 20 determines through authentication that the user performing the operation is a registered user, it displays the start screen 30A on the operation panel 15. The start screen 30A displays instructions to the user, such as "Hold your hand over the screen to start Touchless!"

[0074] When a user places their finger over the start screen 30A, a cursor appears at the user's operating position 6 on the start screen 30A. In the example of the start screen 30A shown in Figure 5, a hand-shaped cursor is displayed, but the cursor shape is just one example; for example, a circular cursor may be displayed. When the user places their finger over the start screen 30A, the home screen 30B is displayed. The instructions given to the user on the start screen 30A also serve to teach the user how to operate the control panel 15.

[0075] The home screen 30B displays, for example, buttons 8 for selecting various functions of the image processing device 10, and a navigation bar 9 that displays useful information for the user during operation. Since the image processing device 10 has copy, print, and scan functions, the home screen 30B displays a copy button 8A for selecting the copy function, a print button 8B for selecting the print function, and a scan button 8C for selecting the scan function. The navigation bar 9 displays, for example, the name of the authenticated user, such as "User A," the name of the screen displayed on the operation panel 15, such as "Home," and information informing the user that the operation panel 15 is in contactless operation mode, such as "Touch Less."

[0076] When the user places their finger over copy button 8A, copy button 8A is selected. When copy button 8A is selected, the copy screen 30D is displayed on the control panel 15. The copy screen 30D displays buttons 8D to 8G for setting copy conditions and a copy start button 8H to start copying with the set copy conditions.

[0077] The copy screen 30D shown in Figure 5 displays, as examples of buttons 8 for setting copy conditions, a color mode button 8D for selecting the copy color, a duplex / single-sided selection button 8E for selecting the copy surface, an N-up button 8F for selecting the method of allocating the image to the recording medium, and a copy number button 8G for selecting the number of copies.

[0078] When the user places their finger over any of the buttons 8D to 8G for setting copy conditions, the button 8 corresponding to the user's operating position 6 is selected, and a screen 30 for setting copy conditions corresponding to the selected button 8 is displayed. If the double-sided / single-sided selection button 8E is selected on the copy screen 30D, the double-sided / single-sided selection screen 30G for selecting the copy surface is superimposed on the copy screen 30D and displayed on the operation panel 15.

[0079] The double-sided / single-sided selection screen 30G shown in Figure 5 displays, for example, a double-sided → double-sided selection button 8S that sequentially copies the contents of both sides of the document 11 to both sides of the recording medium, a single-sided → double-sided selection button 8T that sequentially copies the contents of the document 11, which has text etc. on only one side, to both sides of the recording medium, and a single-sided → single-sided selection button 8U that sequentially copies the contents of the document 11, which has text etc. on only one side, to one side of the recording medium.

[0080] On the double-sided / single-sided selection screen 30G, when the user places their finger over any of the buttons 8S to 8U, the button 8 corresponding to the user's operating position 6 is selected, and the copy surface corresponding to the selected button 8 is set. The example of the double-sided / single-sided selection screen 30G shown in Figure 5 shows the state where the user has selected double-sided → double-sided selection button 8S.

[0081] Once the copy surface is set on the double-sided / single-sided selection screen 30G, the copy screen 30D is displayed on the control panel 15. After the copy surface is set, the copy surface selected on the double-sided / single-sided selection screen 30G is displayed within the double-sided / single-sided selection button 8E on the copy screen 30D.

[0082] The above describes an example where the user selects the duplex / single-sided selection button 8E on the copy screen 30D. Even if the user selects the color mode button 8D, the N-up button 8F, or the number of copies button 8G on the copy screen 30D, a selection screen corresponding to each button 8, such as the duplex / single-sided selection screen 30G, will be displayed on the control panel 15.

[0083] When the user places their finger over the copy start button 8H on the copy screen 30D, the copy start button 8H is selected. When the copy start button 8H is selected, a copy process is executed to copy the contents of the original document 11 to the recording medium according to the set copy conditions. Before setting the copy conditions, the initial settings of the pre-configured copy conditions are displayed on buttons 8D to 8G on the copy screen 30D.

[0084] On the other hand, on the home screen 30B, when the user places their finger over the print button 8B, the print button 8B is selected. When the print button 8B is selected, the print screen 30E is displayed on the control panel 15.

[0085] The print screen 30E displays print information buttons 8J that show information about the image data to be printed, and a "Start Print All" button 8M that starts printing all image data corresponding to each print information button 8J. The example of the print screen 30E shown in Figure 5 represents the print screen 30E when two image data files to be printed have been received. In other words, the print screen 30E displays print information buttons 8J corresponding to each image data file, for each image data file received from the user as a print target.

[0086] If there are too many image data files to display all the corresponding print information buttons 8J on the print screen 30E at once, the user can make a gesture of moving their finger up or down on the print information buttons 8J. The operation panel 15 will then detect the movement of the operation position 6 and scroll the print information buttons 8J. This will display all the print information buttons 8J that were not previously visible on the print screen 30E.

[0087] Each print information button 8J displays the file name of the image data to be printed and the print conditions that the user has pre-set for the image data. For example, when a user sends image data from terminal 4 to image processing device 10, the print conditions that the user set from terminal 4 are displayed on the print information button 8J.

[0088] When the user places their finger over the All Print Start button 8M, the All Print Start button 8M is selected. When the All Print Start button 8M is selected, a print process is executed to print the image represented by the image data onto the recording medium, according to the set print conditions.

[0089] Furthermore, when a user places their finger over any of the print information buttons 8J, that button is selected. When a print information button 8J is selected, the print editing screen 30H is displayed on the operation panel 15. The print editing screen 30H shown in Figure 5 is an example of what is displayed when the user selects the print information button 8J corresponding to the image data of "Document B.pdf".

[0090] The print editing screen 30H displays, for example, a delete button 8V to delete the image data corresponding to the selected print information button 8J, a change button 8W to change the print conditions of the image data corresponding to the selected print information button 8J, and an individual print start button 8X to print only the image data corresponding to the selected print information button 8J. In the print editing screen 30H shown in Figure 5, a change button 8W is displayed as an example of a change button 8W, which changes the number of copies to print. Note that change buttons 8W to change other print conditions, such as the color of the image to be printed, are also displayed on the print editing screen 30H, but are not shown in the illustration.

[0091] On the other hand, on the home screen 30B, when the user places their finger over the scan button 8C, the scan button 8C is selected. When the scan button 8C is selected, the scan screen 30F is displayed on the operation panel 15.

[0092] The scan screen 30F displays a scan settings button 8N for setting scan conditions and a scan start button 8R for starting to scan the document 11 with the set scan conditions.

[0093] When the user places their finger over the scan setting button 8N, the scan setting button 8N corresponding to the user's operating position 6 is selected, and a selection screen (not shown) for selecting scan conditions corresponding to the selected scan setting button 8N is displayed. In other words, the user sets the scan conditions associated with the scan setting button 8N in the same manner as setting copy conditions from the copy screen 30D. When the user places their finger over the scan start button 8R, the scan start button 8R is selected. When the scan start button 8R is selected, a scanning process is executed to convert the contents of the document 11 into image data according to the set scan conditions.

[0094] On the other hand, on the home screen 30B, when the user places their finger over the navigation bar 9, the navigation bar 9 is selected. When the navigation bar 9 is selected, the authenticated user is logged out, and a message indicating that the logout is complete is displayed on the navigation bar 9.

[0095] Up to this point, we have described an example in which button 8 is selected by the user placing their finger over it. In the case of contactless operation, the finger does not touch the operation panel 15, so the finger may wobble. Therefore, if an icon image that includes the operation position 6 within the area is simply considered as the icon image selected by the user, then due to the wobbling of the finger, another icon image adjacent to the icon image being operated on may be mistakenly selected. Also, in the process of moving the finger to the icon image being operated on, the finger may pass over other icon images that are not the target of the operation, and in this case, another icon image may also be mistakenly selected.

[0096] Therefore, in this embodiment, if a finger is continuously held over an icon image for a predetermined period of time (a fixed period of time), the icon image over which the finger is held is determined to be an icon image that the user has intentionally selected. In other words, on the operation panel 15, if the user's operation position 6 remains within the area of ​​a specific icon image for a predetermined period of time (a fixed period of time), it is determined that the user has selected that icon image. In this embodiment, 3 seconds is applied as the predetermined period of time. However, this is not the only example. For example, a period other than 3 seconds may be applied as the predetermined period of time. Furthermore, the method for detecting the operation position 6 is not limited to detection by the operation panel 15, which is a capacitive touch panel. For example, a method using a ToF (Time of Flight) camera or the like may be applied as a method for detecting the operation position 6.

[0097] Figure 6 shows an example of a user selecting the copy button 8A, which is an example of an icon image, on the home screen 30B.

[0098] When a user places their finger over the copy button 8A, the operation position 6 is detected within the area of ​​the copy button 8A. This transition from a state where the operation position 6 is not detected within the area of ​​the icon image to a state where the operation position 6 is detected is called "selection start" or "hover". While the icon image is in the selection start state, the icon image is not yet selected.

[0099] When the user holds their finger over the copy button 8A and the detected operation position 6 remains within the area of ​​the copy button 8A for a predetermined time (a certain period of time), the copy button 8A is selected as shown in Figure 6, and the copy screen 30D is displayed on the operation panel 15. This confirmation of the selection of the icon image is called "selection complete" or "hold." The completion of the selection of an icon image is referred to as "the icon image has been selected."

[0100] Therefore, if the user's finger moves from the copy button 8A to another location while selection is in progress, the selection for the copy button 8A is canceled. This process, where the finger moves from the icon image to another location while selection is in progress, is called "deselection." If the selection for an icon image is canceled, the user can complete the selection of the deselected icon image by holding their finger over the deselected icon image again for a predetermined period of time (a certain amount of time).

[0101] Furthermore, each icon image on screen 30 is pre-associated with a process that will be executed when the copy start button 8H is selected, so that the copy process is executed when the icon image is selected. To inform the user what kind of process the icon image performs, information suggesting the content of the process to be executed when the icon image is selected is displayed, for example, "Copy" on the copy start button 8H. The user understands what kind of process the icon image performs by looking at the information that represents the content of the process to be executed when the icon image is selected, that is, the item associated with the icon image. In this way, the icon images are displayed on screen 30 in association with an item that represents the content of the process to be performed. Therefore, each icon image is an example of an "item displayed on the screen" according to this embodiment.

[0102] Next, the main components of the electrical system of the image processing device 10 will be described with reference to Figure 7. The image processing device 10 is configured, for example, using a computer 40.

[0103] In the computer 40, the CPU (Central Processing Unit) 41, RAM (Random Access Memory) 42, ROM (Read Only Memory) 43, non-volatile memory 44, and input / output interface (I / O) 45 are connected via the bus 46.

[0104] The CPU 41 is an example of a processor responsible for processing each functional unit of the image processing device 10 shown in Figure 4. The RAM 42 is an example of a storage medium used as a temporary workspace for the CPU 41. The ROM 43 is an example of a storage medium that stores information processing programs executed by the CPU 41. The non-volatile memory 44 is an example of a storage medium that maintains stored information even when the power supplied to the non-volatile memory 44 is cut off. For example, semiconductor memory is used, but a hard disk may also be used. The non-volatile memory 44 does not necessarily have to be built into the computer 40; for example, it may be a storage medium that can be attached to and detached from the computer 40, such as a memory card.

[0105] For example, the document scanning unit 12, image forming unit 14, input unit 31, display unit 32, and communication unit 33 are connected to I / O 45.

[0106] The document scanning unit 12 and the image forming unit 14 are devices that perform the operations as already described. The input unit 31 is a device that receives user instructions and user ID and notifies the CPU 21, and the touch panel and reader device 17 that make up the operation panel 15 are examples of the input unit 31. The display unit 32 is a device that visually displays the information processed by the CPU 41, and the display that makes up the operation panel 15 is an example of the display unit 32. The communication unit 33 is connected to the communication line 2 and has a communication protocol for communicating with the terminal 4. Note that the units connected to I / O 45 are not limited to the units exemplified in Figure 7. Depending on the functions of the image processing device 10, the units necessary for realizing those functions are connected to I / O 45.

[0107] By the way, as mentioned above, when determining the operation based on the distance from the screen, contactless operation involves operations in the air, making it difficult for the user to visually perceive the distance from the screen, and also difficult to understand the operation corresponding to that distance.

[0108] Therefore, in the image processing device 10 according to this embodiment, when a user's hand 3 performing a non-contact operation is detected on the screen 30, a first frame is displayed centered on the detected position according to the distance between the screen 30 and the user's hand 3, and a plurality of second frames, each with a different display format from the first frame, are displayed centered on the detected position, corresponding to a plurality of regions divided according to the distance from the screen 30. Different operation content is pre-assigned to each of the plurality of regions for non-contact operations.

[0109] Specifically, the CPU 41 of the image processing apparatus 10 according to this embodiment functions as the various parts shown in Figure 8 by writing the information processing program stored in the ROM 43 to the RAM 42 and executing it.

[0110] Figure 8 is a block diagram showing an example of the functional configuration of the image processing apparatus 10 according to the first embodiment.

[0111] As shown in Figure 8, the CPU 41 of the image processing apparatus 10 according to this embodiment functions as a detection unit 41A, a distance determination unit 41B, a conversion unit 41C, an operation processing unit 41D, and a display control unit 41E.

[0112] The non-volatile memory 44 stores operation assignment setting information 44A. Operation assignment setting information 44A indicates the number of areas to be divided and the operation content to be assigned to each area.

[0113] Figure 9 is a schematic diagram showing an example of a spatial region when the operation panel 15 according to this embodiment is viewed from the side.

[0114] In the example in Figure 9, the spatial region is divided into three regions: the first region R1 to the third region R3. The first region R1 is assigned a "pointing operation," the second region R2 is assigned a "click operation," and the third region R3 is assigned a "drag operation." In other words, the operation assignment setting information 44A above indicates the number of these region divisions and the operation content assigned to each region. A "pointing operation" is an operation to point to a certain position on the screen 30, a "click operation" is an operation to click on a certain position or icon image on the screen 30, and a "drag operation" is an operation to drag an icon image on the screen 30. Assignable operation content includes, for example, pointing operations, click operations, double-click operations, long-press operations, drag operations, and scroll operations. Also, the number of region divisions is not limited to three; it may be two, four or more, or any number of regions.

[0115] Furthermore, the first region R1 and the second region R2 are separated by the first threshold Th1, and the second region R2 and the third region R3 are separated by the second threshold Th2. The first threshold Th1 and the second threshold Th2 are thresholds for the distance from the screen 30 to the user's hand 3 (i.e., the operation distance D), and appropriate values ​​are set within a range that does not exceed the distance at which the user's hand 3 can be detected. These thresholds are also stored as operation assignment setting information 44A. In other words, the first region R1, the second region R2, and the third region R3 are separated according to the distance from the screen 30, and different operation contents are pre-assigned to non-contact operations. In addition, the operation contents assigned to each of the first region R1, the second region R2, and the third region R3 can be set by the user as appropriate. When there is no need to explain each of these first region R1, second region R2, and third region R3 separately, the regions are collectively referred to as "region R".

[0116] The detection unit 41A detects the user's hand 3 on the screen 30 and identifies the detected position. The detected position is identified as a coordinate point (x, y, z), as described above.

[0117] The distance determination unit 41B determines the distance (operating distance D) from the screen 30 to the user's hand 3 detected by the detection unit 41A. The distance (operating distance D) from the screen 30 to the user's hand 3 is determined, as described above, for example, by a change in capacitance.

[0118] The conversion unit 41C converts the distance determined by the distance determination unit 41B into a first frame. The conversion unit 41C also converts the multiple regions R obtained from the operation assignment setting information 44A into multiple second frames.

[0119] The operation processing unit 41D identifies the spatial region where the user's hand 3 is located based on the distance determined by the distance determination unit 41B, and identifies the operation content assigned to the identified spatial region from the operation assignment setting information 44A. The operation processing unit 41D then executes the operation corresponding to the identified operation content.

[0120] The display control unit 41E controls the display of the first frame and multiple second frames converted by the conversion unit 41C on the screen 30. The display control unit 41E also controls the display of the screen 30 according to the operation performed by the operation processing unit 41D.

[0121] When the display control unit 41E detects a user's hand 3 performing a non-contact operation on the screen 30, it controls the display of a first frame centered on the detected position, according to the distance (operation distance D) between the screen 30 and the user's hand 3. The display control unit 41E also controls the display of multiple second frames centered on the detected position, corresponding to multiple regions R divided according to the distance from the screen 30. Different operation content is pre-assigned to each of the multiple regions R for non-contact operations. The display format of the multiple second frames differs from that of the first frame. The first frame and the multiple second frames may be displayed simultaneously, the multiple second frames may be displayed after the first frame, or the first frame may be displayed after the multiple second frames.

[0122] Next, with reference to Figures 10(A) and 10(B), specific examples of the first frame and multiple second frames displayed on screen 30 will be explained.

[0123] Figure 10(A) schematically shows an example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 10(B) shows an example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front. Here, the multiple second frames W2 are shown as multiple second frames W21 to W23, but when there is no need to explain each of the multiple second frames W21 to W23 separately, each second frame is collectively referred to as "second frame W2".

[0124] As shown in Figures 10(A) and 10(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W23. The first region R1 is associated with the second frame W21, the second region R2 is associated with the second frame W22, and the third region R3 is associated with the second frame W23.

[0125] In the examples in Figures 10(A) and 10(B), the size of the first frame W1 displayed when the distance between the screen 30 and the user's hand 3 is the first distance is smaller than the size of the frame displayed when the distance is the second distance, which is longer than the first distance. In other words, the size of the first frame W1 decreases as the distance between the screen 30 and the user's hand 3 decreases. The size of the second frame W2 is smallest for the area closest to the screen 30 and largest for the area furthest from the screen 30. In other words, the second frame W23 corresponding to the third area R3 is smallest, and the second frame W21 corresponding to the first area R1 is largeest. The first frame W1 and the multiple second frames W21 to W23 are displayed concentrically around the position where the user's hand 3 is detected. Although the shape of the first frame W1 and the multiple second frames W21 to W23 are shown as circular, they may be other shapes, such as rectangles.

[0126] The first frame W1 and the multiple second frames W21-W23 differ in at least one of their line type and color. Furthermore, each of the multiple second frames W21-W23 differs in at least one of their line type and color depending on the corresponding region R. In the examples of Figures 10(A) and 10(B), the first frame W1 is represented by a solid black line, the second frame W21 by a dotted blue line, the second frame W22 by a dotted yellow line, and the second frame W23 by a dotted red line. The spacing and thickness of the dotted lines representing the second frames W21-W23 may also be varied. The second frames W21-W23 may also be represented by a gradient (for example, a change from light blue to dark blue).

[0127] Here, when displaying multiple second frames W21 to W23, it is also possible to selectively display text information indicating the operation content assigned to the region R where the user's hand 3 is detected, from among the multiple regions R1 to R3 corresponding to the multiple second frames W21 to W23. In the examples of Figures 10(A) and 10(B), since the user's hand 3 is detected in the first region R1, only the text information 51 representing "pointing operation," which is the operation content assigned to the first region R1, is selectively displayed.

[0128] Figure 11 shows another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0129] As shown in Figure 11, when displaying multiple second frames W21 to W23, text information indicating the operation content assigned to each of the multiple regions R1 to R3 corresponding to the multiple second frames W21 to W23 may be displayed. In the example in Figure 11, the user's hand 3 is detected in the first region R1, and text information 51 representing "pointing operation," which is the operation content assigned to the first region R1, is displayed, text information 52 representing "click operation," which is the operation content assigned to the second region R2, is displayed, and text information 53 representing "drag operation," which is the operation content assigned to the third region R3, is displayed.

[0130] Figure 12(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 12(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0131] In the examples in Figures 12(A) and 12(B), the size of the first frame W1 displayed when the distance between the screen 30 and the user's hand 3 is the first distance is larger than the size of the frame displayed when the distance is the second distance, which is longer than the first distance. In other words, the size of the first frame W1 increases as the distance between the screen 30 and the user's hand 3 decreases. The size of the second frame W2 is largest for the area closest to the screen 30 and smallest for the area furthest from the screen 30. In other words, the second frame W23 corresponding to the third area R3 is the largest, and the second frame W21 corresponding to the first area R1 is the smallest.

[0132] Next, referring to Figures 13(A) and 13(B), we will describe the configuration in which "point fine-tuning operation" or "drag fine-tuning operation" is assigned to region R.

[0133] Figure 13(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 13(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0134] As shown in Figures 13(A) and 13(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W22. The first region R1 is associated with the second frame W21, and the second region R2 is associated with the second frame W22.

[0135] "Point fine-tuning operation" and "Drag fine-tuning operation" are examples of operations in which the amount of movement per unit time of the first frame W1 corresponding to the detected position of the user's hand 3 is smaller than the amount of movement per unit time of the user's hand 3. In other words, "Point fine-tuning operation" is an operation in which the amount of movement per unit time of the first frame W1 (pointing position on screen 30) is smaller than the amount of movement per unit time of the user's hand 3, and "Drag fine-tuning operation" is an operation in which the amount of movement per unit time of the first frame W1 (drag position on screen 30) is smaller than the amount of movement per unit time of the user's hand 3.

[0136] The examples in Figures 13(A) and 13(B) illustrate the case of "point fine-tuning operation." Specifically, the amount of movement D2 per unit time of the first frame W1 corresponding to the position where the user's hand 3 is detected becomes smaller than the amount of movement D1 per unit time of the user's hand 3. However, the same applies to the case of "drag operation."

[0137] This makes it easier to perform fine operations that would be difficult with the user's hand raised. For example, it is suitable for tasks such as selecting text and drawing shapes.

[0138] Next, referring to Figures 14(A) and 14(B), we will describe a form of notification that occurs when the user's hand 3 approaches the screen 30, for example, from the standpoint of hygiene, safety, etc.

[0139] Figure 14(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 14(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0140] As shown in Figures 14(A) and 14(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W22. The first region R1 is associated with the second frame W21, and the second region R2 is associated with the second frame W22.

[0141] In the examples shown in Figures 14(A) and 14(B), if it is detected that the user's hand 3 is located in the region closest to the screen 30 (for example, the second region R2) among multiple regions R, a notification is issued that the user's hand 3 is located in a region close to the screen 30. For example, the notification may be made by voice, a warning sound, etc. Alternatively, the notification may be made by text, etc.

[0142] According to this, it is preferably used when the user's hand 3 does not touch the screen 30 from the standpoint of hygiene, safety, etc.

[0143] If you want to streamline the process by enabling the execution of multiple types of operations (for example, four or more), you can increase the number of regions R and shorten the width of each region R, as shown in Figures 15(A) and 15(B).

[0144] Figure 15(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 15(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0145] As shown in Figures 15(A) and 15(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W24. The first region R1 is associated with the second frame W21, the second region R2 is associated with the second frame W22, the third region R3 is associated with the second frame W23, and the fourth region R4 is associated with the second frame W24. In this case, the first region R1 is assigned to "pointing operation", the second region R2 is assigned to "point fine adjustment operation", the third region R3 is assigned to "click operation", and the fourth region R4 is assigned to "drag operation".

[0146] Furthermore, to reduce errors and ensure reliable operation, one example is to reduce the number of regions R and increase the width of each region R, as shown in Figures 16(A) and 16(B).

[0147] Figure 16(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 16(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0148] As shown in Figures 16(A) and 16(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W22. The first region R1 is associated with the second frame W21, and the second region R2 is associated with the second frame W22. In this case, the first region R1 is assigned to "pointing operation," and the second region R2 is assigned to "click operation."

[0149] Figure 17(A) schematically shows yet another example of multiple spatial regions R when the control panel 15 is viewed from the side. Figure 17(B) shows yet another example of the first frame W1 and multiple second frames W2 when the control panel 15 is viewed from the front.

[0150] As shown in Figures 17(A) and 17(B), the screen 30 displays a first frame W1 centered on the position where the user's hand 3 is detected, depending on the distance between the screen 30 and the user's hand 3, and displays multiple second frames W21 to W23. The first region R1 is associated with the second frame W21, the second region R2 is associated with the second frame W22, and the third region R3 is associated with the second frame W23.

[0151] In the examples of Figures 17(A) and 17(B), by aligning the first frame W1 and the multiple second frames W2 concentrically as described above, the first frame W1 and the multiple second frames W2 will not be completely hidden even if the position where the user's hand 3 is detected is at the edge of the screen 30. Furthermore, by aligning the first frame W1 and the multiple second frames W2 concentrically, the first frame W1 and the multiple second frames W2 will not be completely hidden by the user's hand 3.

[0152] Next, with reference to Figure 18, the operation of the image processing apparatus 10 according to the first embodiment will be described.

[0153] Figure 18 is a flowchart showing an example of the processing flow by the information processing program according to the first embodiment.

[0154] First, when contactless input is instructed on the control panel 15, the CPU 41 starts the information processing program and executes the following steps.

[0155] In step S101 of Figure 18, the CPU 41 reads operation assignment setting information 44A from the non-volatile memory 44.

[0156] In step S102, the CPU 41 determines whether or not it has detected non-contact input from the user. If it determines that, for example, the user's hand 3 has been detected as non-contact input from the user (positive determination), the process proceeds to step S103. If it determines that the user's hand 3 has not been detected (negative determination), the process remains in a waiting state in step S102.

[0157] In step S103, the CPU 41 controls the display of multiple second frames W2 on the screen 30, centered on the position where the user's hand 3 is detected, by associating multiple regions R, each with a different operation content pre-assigned to a non-contact operation by the user, as shown in Figures 10(A) and 10(B) above, for example.

[0158] In step S104, the CPU 41 determines the distance from the screen 30 to the user's hand 3.

[0159] In step S105, the CPU 41 controls the display of the first frame W1 centered on the position where the user's hand 3 was detected, according to the distance determined in step S104, as shown in Figures 10(A) and 10(B) above. The display timing of the multiple second frames W2 and the first frame W1 is not particularly limited and may be simultaneous.

[0160] In step S106, the CPU 41 updates the distance information from the screen 30 to the user's hand 3 in accordance with the movement of the position where the user's hand 3 was detected.

[0161] In step S107, the CPU 41 updates the display of the first frame W1 on the screen 30 according to the distance information updated in step S106.

[0162] In step S108, the CPU 41 performs an operation corresponding to the distance from the screen 30 to the user's hand 3.

[0163] In step S109, the CPU 41 determines whether or not the termination timing has arrived. If it determines that the termination timing has not arrived (negative determination), it returns to step S102 and repeats the process. If it determines that the termination timing has arrived (positive determination), it terminates the series of processes by this information processing program. The termination timing referred to here is assumed to be, for example, when the power is turned off or when the system enters sleep mode.

[0164] As described above, according to this embodiment, when an object to be operated on without contact is detected on the screen, a first frame is displayed centered on the detected position, according to the distance between the screen and the object. Multiple second frames, each with a different display format from the first frame, are displayed centered on the detected position, corresponding to multiple regions divided according to the distance from the screen. Therefore, when performing contactless operations on the screen, the user can visually grasp the correspondence between the distance from the screen and the operation content which varies depending on that distance.

[0165] [Second Embodiment] In the first embodiment described above, if the object moves from the original area to another area during non-contact operation, the operation corresponding to the other area to which it has moved may be executed. In contrast, the second embodiment describes a configuration in which, even if the object moves from the original area to another area during non-contact operation, the operation corresponding to the original area is executed, rather than the operation corresponding to the other area to which it has moved.

[0166] The CPU 41 of the image processing apparatus 10A according to this embodiment functions as the various parts shown in Figure 19 by writing the information processing program stored in the ROM 43 to the RAM 42 and executing it.

[0167] Figure 19 is a block diagram showing an example of the functional configuration of the image processing apparatus 10A according to the second embodiment.

[0168] As shown in Figure 19, the CPU 41 of the image processing apparatus 10A according to this embodiment functions as a detection unit 41A, a distance determination unit 41B, a conversion unit 41C, an operation processing unit 41D, and a display control unit 41E, as well as a shape determination unit 41F and a stop processing unit 41G. The same reference numerals are used for components identical to those in the image processing apparatus 10 described in the first embodiment, and repeated explanations are omitted.

[0169] The non-volatile memory 44 stores operation assignment setting information 44A and operation fixed shape setting information 44B.

[0170] The operation-fixing shape setting information 44B is information indicating the shape of the object for fixing the operation content according to region R, regardless of region R.

[0171] Figure 20 shows an example of the operation fixed shape setting information 44B according to the second embodiment.

[0172] As shown in Figure 20, operation-fixed shape setting patterns 61 and 62 are defined as operation-fixed shape setting information 44B. Here, the user's hand 3 is used as an example for explanation. In operation-fixed shape setting pattern 61, when the user's hand 3 changes from "two fingers" to "five fingers (hand extended)", the operation content is fixed, and further, when the user's hand 3 changes from "five fingers" to "two fingers", the fixation of the operation content is canceled. On the other hand, in operation-fixed shape setting pattern 62, when the user's hand 3 changes from "two fingers" to "three fingers", the operation content is fixed, and further, when the user's hand 3 changes from "three fingers" to "two fingers", the fixation of the operation content is canceled. Operation-fixed shape setting patterns 61 and 62 are just examples, and the operation-fixed shape setting patterns can be set as appropriate by the user.

[0173] The shape determination unit 41F determines the shape of the user's hand 3 on the screen 30. Here, the capacitance detected on the screen 30 differs depending on the shape of the user's hand 3. Therefore, by acquiring the correspondence between the shape of the user's hand 3 and the capacitance in advance, it is possible to identify the shape of the user's hand 3 from the capacitance detected when the user's hand 3 is held over the screen 30. Alternatively, the shape of the user's hand 3 may be identified from an image captured by a camera.

[0174] When the stop processing unit 41G detects a specific action by the user's hand 3 within any of the multiple regions R, it fixes the operation to the region where the specific action was detected, regardless of the distance between the screen 30 and the user's hand 3. For example, a specific action is one that changes the shape of the user's hand 3 from a first shape (e.g., "two fingers") to a second shape (e.g., "five fingers"), as shown in the operation-fixed-shape setting pattern 61 described above. Furthermore, for example, after detecting an action that changes the shape from the first to the second shape according to the operation-fixed-shape setting pattern 61 described above, the stop processing unit 41G releases the fix if it detects an action that returns the hand from the second shape back to the first shape.

[0175] When performing contactless operations, as shown in the comparative example in Figure 21, it is difficult to move the user's hand 3 steadily in the air, and the possibility of errors is particularly high with drag and flick operations. Also, dividing the space on the operation panel 15 into unnecessarily small sections may increase the possibility of errors.

[0176] Figure 21 shows the transitions of screen 30 in response to the user's hand 3 actions in a comparative example. In the example in Figure 21, the first area R1 is assigned to "pointing operation," and the second area R2 is assigned to "drag operation." Note that in the example in Figure 21, the second frame W23 corresponding to the third area R3 is not shown.

[0177] In (S1) of Figure 21, among the multiple first region R1, second region R2, and third region R3, the user's hand 3 is detected in the second region R2. Subsequently, in (S2), if the user's hand 3 is mistakenly moved to the first region R1, the "pointing operation" of the first region R1 is executed instead of the "drag operation" of the second region R2.

[0178] Figure 22 is a diagram showing an example of the transition of the screen 30 in response to the user's hand 3 movements according to the second embodiment. In the example of Figure 22, as in the example of Figure 21, the first area R1 is assigned to "pointing operation" and the second area R2 is assigned to "drag operation". Note that in the example of Figure 22, as in the example of Figure 21, the second frame W23 corresponding to the third area R3 is not shown.

[0179] In (S11) of Figure 22, the user's hand 3 is detected in the second region R2 among the multiple first region R1, second region R2, and third region R3. At this time, the user performs an action to change the shape of hand 3 from a first shape (e.g., "two fingers") to a second shape (e.g., "five fingers"). By changing the shape of the user's hand 3, the operation content assigned to the second region R2 is fixed. Subsequently, in (S12), even if the user's hand 3 is accidentally moved to the first region R1, the "drag operation" of the second region R2 is executed instead of the "pointing operation" of the first region R1.

[0180] In this embodiment, when the user's hand 3 performs a specific action within region R, the update of the distance between the screen 30 and the user's hand 3 is stopped, and the operation content of region R where the specific action was detected is fixed. Specifically, as shown in the example in Figure 22, if the user's hand 3 changes from "two fingers" to "five fingers" within the second region R2 to which a "drag operation" is assigned, even if the user's hand 3 moves to the first region R1 during the drag operation, the operation content remains fixed as a "drag operation".

[0181] Next, with reference to Figure 23, the operation of the image processing apparatus 10A according to the second embodiment will be described.

[0182] Figure 23 is a flowchart showing an example of the processing flow by the information processing program according to the second embodiment.

[0183] First, when contactless input is instructed on the control panel 15, the CPU 41 starts the information processing program and executes the following steps.

[0184] In step S111 of Figure 23, the CPU 41 reads operation assignment setting information 44A from the non-volatile memory 44.

[0185] In step S112, the CPU 41 reads the fixed operation shape setting information 44B from the non-volatile memory 44.

[0186] In step S113, the CPU 41 determines whether or not it has detected non-contact input from the user. If it determines that, for example, the user's hand 3 has been detected as non-contact input from the user (positive determination), the process proceeds to step S114. If it determines that the user's hand 3 has not been detected (negative determination), the process remains in a waiting state in step S113.

[0187] In step S114, the CPU 41 controls the display of multiple second frames W2 on the screen 30, centered on the position where the user's hand 3 is detected, by associating multiple regions R, each with a different operation content pre-assigned to a non-contact operation by the user, as shown in Figure 22 above, for example.

[0188] In step S115, the CPU 41 determines the distance from the screen 30 to the user's hand 3.

[0189] In step S116, the CPU 41 controls the display of the first frame W1 centered on the position where the user's hand 3 was detected, according to the distance determined in step S115, as shown in Figure 22 above. The display timing of the multiple second frames W2 and the first frame W1 is not particularly limited and may be simultaneous.

[0190] In step S117, the CPU 41 determines whether or not to fix the operation, that is, whether or not it has detected a specific action in which the user's hand 3 changes from a first shape (e.g., "two fingers") to a second shape (e.g., "five fingers"). If it determines that it has not detected a specific action by the user's hand 3 (negative determination), it proceeds to step S118. If it determines that it has detected a specific action by the user's hand 3 (positive determination), it proceeds to step S120.

[0191] In step S118, the CPU 41 updates the distance information from the screen 30 to the user's hand 3 in accordance with the movement of the position where the user's hand 3 was detected.

[0192] In step S119, the CPU 41 updates the display of the first frame W1 on screen 30 according to the distance information updated in step S118, and then proceeds to step S121.

[0193] Meanwhile, in step S120, the CPU 41 stops updating the distance information from the screen 30 to the user's hand 3 and proceeds to step S121.

[0194] In step S121, the CPU 41 performs an operation corresponding to the distance from the screen 30 to the user's hand 3. If a specific movement by the user's hand 3 is detected in step S117, the operation content assigned to the distance (region R) where the specific movement was detected will be executed, as shown in Figure 22 above as an example.

[0195] In step S122, the CPU 41 determines whether or not the termination timing has arrived. If it determines that the termination timing has not arrived (negative determination), it returns to step S113 and repeats the process. If it determines that the termination timing has arrived (positive determination), it terminates the series of processes by this information processing program. The termination timing referred to here is assumed to be, for example, when the power is turned off or when the system enters sleep mode.

[0196] As described above, according to this embodiment, when an object performs a specific action within a region, the update of the distance between the screen and the object is stopped, and the operation content of the region where the specific action was detected is fixed. Therefore, when performing contactless operations on the screen, the occurrence of errors is suppressed.

[0197] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0198] Furthermore, the processor operations in each of the above embodiments may not be performed by a single processor, but may also be performed by multiple processors located in physically separate locations working together. Also, the order of the processor operations is not limited to the order described in each of the above embodiments, but may be changed as appropriate.

[0199] The above description has illustrated an image processing device as an example of an information processing device according to the embodiment. The embodiment may also take the form of a program that causes a computer to execute the functions of the information processing device. The embodiment may also take the form of a non-temporary storage medium that is readable by a computer and stores these programs.

[0200] Furthermore, the configuration of the information processing device described in the above embodiment is merely an example, and may be modified as needed without departing from the main purpose.

[0201] Furthermore, the program processing flow described in the above embodiment is just one example, and unnecessary steps may be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0202] Furthermore, although the above embodiment describes a case in which the process according to the embodiment is realized by a software configuration using a computer by executing a program, the embodiment is not limited to this. The embodiment may also be realized by a hardware configuration or a combination of a hardware configuration and a software configuration. [Explanation of symbols]

[0203] 1. Information Processing System 2. Communication lines 3. User's hands 4 terminals 6 Operating position 8 buttons 8A Copy button 8B Print Button 8C Scan button 9 Navigation bar 10, 10A Image Processing Device 11 Manuscript 12 Document scanning unit 13 Operation Display Unit 14 Image forming unit 15. Control Panel 20 Control Unit 21 Reception Department 22 Display section 23. Document reading unit 24 Image forming unit 30 screens 30B Home Screen 31 Input Units 32 Display Units 33 Communication Unit 40 Computers 41 CPU 41A Detection Unit 41B Distance judgment section 41C Conversion Unit 41D Operation Processing Unit 41E Display Control Unit 41F Shape determination section 41G Stop Processing Unit 42 RAM 43 ROM 44 Non-volatile memory 44A Operation Assignment Setting Information 44B Fixed operation shape setting information 45 I / O

Claims

1. Equipped with a processor, The aforementioned processor, When an object to be operated on without contact is detected on the screen, a first frame is displayed centered on the detected position, according to the distance between the screen and the object. Centered on the detected position, a plurality of second frames with a different display format from the first frame are displayed. A plurality of regions are separated according to the distance from the screen, and each of the plurality of regions, each of which has different operation content pre-assigned to the non-contact operation, is associated with each of the plurality of second frames. If the object is continuously detected within the area of ​​the second frame for a predetermined period of time, it is determined that the operation content pre-assigned to the area corresponding to the second frame has been selected. Information processing device.

2. The first frame and the plurality of second frames differ in at least one of their line type and color. The information processing apparatus according to claim 1.

3. The aforementioned plurality of second frames differ in at least one of the line type and color depending on the corresponding area. The information processing apparatus according to claim 2.

4. The size of the first frame displayed when the distance between the screen and the object is a first distance is smaller than the size of the frame displayed when the distance is a second distance which is longer than the first distance. The size of the second frame is such that the smallest frame corresponds to the area closest to the screen, and the largest frame corresponds to the area furthest from the screen. The information processing apparatus according to any one of claims 1 to 3.

5. The size of the first frame displayed when the distance between the screen and the object is a first distance is larger than the size of the frame displayed when the distance is a second distance which is longer than the first distance. The size of the second frame is such that the frame size corresponding to the area closest to the screen is the largest, and the frame size corresponding to the area furthest from the screen is the smallest. The information processing apparatus according to any one of claims 1 to 3.

6. The first frame and the plurality of second frames are displayed concentrically around the detected position. The information processing apparatus according to claim 4 or claim 5.

7. The first frame is made movable in accordance with the movement of the object, The amount of movement per unit time of the first frame corresponding to the detected position is smaller than the amount of movement per unit time of the object. An information processing apparatus according to any one of claims 1 to 6.

8. When the processor detects that the object is located in the region closest to the screen among the multiple regions, it notifies that the object is located in the region close to the screen. The information processing apparatus according to any one of claims 1 to 7.

9. When the processor detects a specific action by the object within any of the multiple regions, it stops updating the operation content assigned to the region where the specific action was detected, regardless of the distance between the screen and the object. The information processing apparatus according to any one of claims 1 to 8.

10. The aforementioned specific operation is an operation that changes the shape of the object from a first shape to a second shape. After detecting the operation of changing from the first shape to the second shape, the processor If an operation to return from the second shape to the first shape is detected, the suspension of the update is released. The information processing apparatus according to claim 9.

11. When the processor displays the plurality of second frames, it displays character information indicating the operation content assigned to each of the plurality of areas corresponding to the plurality of second frames. The information processing apparatus according to any one of claims 1 to 10.

12. When the processor displays the plurality of second frames, it selectively displays text information indicating the operation content assigned to the region where the object was detected among the plurality of regions corresponding to the plurality of second frames. The information processing apparatus according to any one of claims 1 to 10.

13. When an object to be operated on without contact is detected on the screen, a first frame is displayed centered on the detected position, according to the distance between the screen and the object. Centered on the detected position, a plurality of second frames with a different display format from the first frame are displayed. A plurality of regions are separated according to the distance from the screen, and each of the plurality of regions, each of which has different operation content pre-assigned to the non-contact operation, is associated with each of the plurality of second frames. When the object is continuously detected within the area of ​​the second frame for a predetermined period of time, it is determined that the operation content pre-assigned to the area corresponding to the second frame has been selected. An information processing program designed to be executed by a computer.

Images