Information processing device and information processing program

Abstract

To present, when a non-contact UI detects a state in which a user is no aware of a function and an operation of a device, these information as guide information.SOLUTION: An information processing apparatus 10 detects a non-contact specific state to an operation surface by a user, and when detecting the specific state, presents an explanation of an operation method to the operation surface or an explanation of a function an object displayed on the operation surface.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing program.

Background Art

[0002] Conventionally, an information processing apparatus capable of performing a non-contact operation in which an operation on an operation surface is performed by an operation body without directly touching the operation surface with the operation body such as a finger or a stylus has been proposed.

[0003] For example, Patent Document 1 discloses an input processing apparatus including an input detection unit that detects proximity and contact of an object as an input, an event type determination unit that determines the type of an event input based on the coordinates of the detected input, a mode information acquisition unit that acquires information on a processing mode for an application, and a transmission unit that transmits data including the type of the event and the acquired processing mode to the information processing apparatus.

[0004] Further, Patent Document 2 discloses a technique for performing an operation of estimating a contact area on a touch screen by a finger or the like, determining whether a finger or the like has touched or approached the touch screen, and performing an enlargement process of an image suitable for the contact area at the time of proximity.

[0005] Further, Patent Document 3 discloses a user interface apparatus including a proximity / contact determination unit that detects the proximity and contact state of an object to a touch panel, a coordinate detection unit that detects relative proximity and contact position coordinates of the object to the touch panel when the proximity and contact state are detected, a proximity speed calculation unit that calculates the proximity speed of the object to the touch panel based on the time change of the proximity position coordinates, and a content control unit that controls the screen display of the display unit according to the contact position coordinates and the proximity speed.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Incidentally, in devices equipped with contact-type or non-contact-type user interfaces, there are cases where it is not clear what kind of processing or operations are possible with the device.

[0008] Furthermore, when using contact-based or contactless user interfaces, gesture-based input is commonly employed. This offers the advantage of intuitive and easy operation if the user is familiar with the gesture methods. On the other hand, if the user is unfamiliar with the available gestures, they may be unable to fully utilize the provided functions.

[0009] Thus, when the contactless UI detected a situation where the user was unaware of the device's functions or operation, it was not possible to properly convey this information to the user. [Means for solving the problem]

[0010] The invention according to claim 1 comprises a processor, the processor detects a specific non-contact state with respect to the operating surface by the user, and when such specific state is detected, presents an explanation of how to operate the operating surface or an explanation of the function of an object displayed on the operating surface. The content of the above explanation will differ depending on the trajectory in which proximity detection was performed with respect to the operating surface. It is an information processing device.

[0011] The invention according to claim 2 is an information processing device according to claim 1, wherein the specific state is a state in which proximity detection to the operating surface continues for a certain period of time while no operation is being performed on the object displayed on the operating surface.

[0013] The invention according to claim 4 is the information processing apparatus according to claim 1, wherein the operation method is an explanation of a gesture operation method corresponding to the user's movement on the operation surface.

[0014] The invention according to claim 5 is an information processing device according to claim 4, wherein the operation method is a description of a gesture operation method that is performed in a non-contact manner with respect to the operation surface and corresponds to the movement of a user.

[0015] The invention according to claim 6 is an information processing device according to any one of claims 1 to 5, which displays the description while avoiding the area where proximity detection has been performed on the operating surface.

[0016] The invention according to claim 7 causes a computer to detect a specific state of non-contact with an operating surface by a user, and when such specific state is detected, it presents an explanation of how to operate the operating surface or an explanation of the function of an object displayed on the operating surface. The content of the above explanation will differ depending on the trajectory in which proximity detection was performed with respect to the operating surface. It is an information processing program. [Effects of the Invention]

[0017] According to the inventions of claims 1 and 7, when a non-contact UI detects a state in which the user is unaware of the function or operation of the device, The system provides guide information by displaying different guide information depending on the trajectory where proximity detection occurred. It is possible.

[0018] According to the invention of claim 2, guide information can be provided when proximity detection to the operating surface continues for a certain period of time and no operation is being performed on the object displayed on the operating surface.

[0020] According to the invention of claim 4, a method of gesture operation that responds to the user's movements on the operating surface can be presented.

[0021] According to the invention of claim 5, a method for gesture operation that responds to user movements and is performed without contact with the operating surface can be presented.

[0022] According to the invention according to claim 6, it is possible to make it easier to grasp the guide information by displaying an explanation while avoiding an area where proximity detection is performed on the operation surface.

Brief Description of the Drawings

[0023] [Figure 1] It is a schematic configuration diagram of an information processing apparatus according to the present embodiment. [Figure 2] It is a diagram showing a space area facing the operation surface and the state of operations thereon. [Figure 3] It is a diagram showing a space area facing the operation surface and the state of operations thereon. [Figure 4] It is a flowchart showing an information processing method according to the present embodiment. [Figure 5] It is a diagram showing an example of a menu screen displayed on the operation surface. [Figure 6] It is a diagram showing an example of a menu screen on which guide information is displayed. [Figure 7] It is a diagram showing an example of a menu screen partially covered by a proximity detection area. [Figure 8] It is a diagram showing an example of a menu screen on which guide information is displayed while avoiding the proximity detection area. [Figure 9] It is a diagram showing another example of a menu screen on which guide information is displayed while avoiding the proximity detection area. [Figure 10] It is a diagram showing an example of a menu screen on which a hint button is displayed. [Figure 11] It is a diagram showing an example of a menu screen on which guide information corresponding to the hint button is displayed. [Figure 12] It is a diagram for explaining an example of a non-contact specific state that is not effective as a gesture operation. [Figure 13] It is a diagram showing an example of a menu screen on which guide information is displayed according to a non-contact specific state that is not effective as a gesture operation. [Figure 14] It is a diagram for explaining another example of a non-contact specific state that is not effective as a gesture operation. [Figure 15] This figure shows another example of a menu screen where guide information is displayed in response to specific non-contact states that are not valid as gesture operations. [Figure 16] This diagram illustrates an example of a specific non-contact state that is not valid as a gesture control. [Figure 17] This figure shows an example of displaying animated guide information in response to specific non-contact states that are not valid as gesture controls. [Figure 18] This figure shows another example of displaying animated guide information in response to specific non-contact states that are not valid as gesture operations. [Modes for carrying out the invention]

[0024] Figure 1 is a schematic diagram of the configuration of the information processing device 10 according to this embodiment. As will be described in detail later, the information processing device 10 is a device equipped with a user interface (UI) that enables contactless operation. Contactless operation is the operation of an operating object on an operating surface without directly touching the operating object to the operating surface. Contactless operation is also called hover operation. Contact operation is the operation of an operating object by directly touching the operating surface to the operating object. An operating object is an object used for operation, such as a user's finger or a stylus.

[0025] In this embodiment, the information processing device 10 is a multifunction device that has printing, copying, scanning, etc. functions and performs printing, copying, or scanning in response to processing commands (jobs) from the user. However, the information processing device 10 is not limited to this and may be any device as long as it is capable of contactless operation.

[0026] As shown in Figure 1, the information processing device 10 consists of a display 12, an object sensor 14, a memory 16, and a processor 18. Although not shown, the information processing device 10 may also include a communication interface (e.g., a NIC (Network Interface Card)) for communicating with other devices via a communication line such as a LAN (Local Area Network) or WAN (Wide Area Network), and a processing device (e.g., a printer or scanner) for performing print and scan operations.

[0027] The display 12 constitutes a display unit for displaying an operation screen. The display 12 is composed of, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. Various screens are displayed on the display 12 by the processor 18. For example, the display 12 displays an operation screen that includes objects that are the target of non-contact or contact operation. The objects can be operation icons or various buttons that the user operates.

[0028] The object sensor 14 constitutes a detection unit that detects the proximity or contact of an object. The object sensor 14 is a sensor that detects objects that are in contact with or near the display surface of the display 12, whether in contact or not. Specifically, the object sensor 14 detects the presence or absence of an object that has approached or come into contact with the display 12, and the position of that object. The position of the object includes the position in a plane parallel to the display 12 and the position in a direction perpendicular to the display 12. The object sensor 14 detects not only operating objects for non-contact and contact operations on the display 12, but also any object that approaches the display 12.

[0029] In this embodiment, the display surface of the display 12 corresponds to the operation surface that is the target of operation. That is, the operation screen, including operation icons and various buttons that the user operates, is displayed on part or all of the operation surface. In the following description, the operation surface will be simply referred to as the display 12.

[0030] Various known methods can be used to detect objects. For example, the object sensor 14 may be a capacitive sensor that detects changes in capacitance between the display 12 and an object. In this case, the processor 18 can detect the presence and position of an object according to the changes in capacitance between the display 12 and the object detected by the object sensor 14. Alternatively, the object sensor 14 may be an optical sensor that detects light. In this case, infrared light or laser light is emitted from a light source (not shown) in the direction of the display surface of the display 12, and the object sensor 14 detects the reflected light, especially the reflected light from an object. The processor 18 can detect the presence and position of an object based on the reflected light detected by the object sensor 14. By placing such an object sensor 14 in conjunction with the display 12, objects approaching or touching the display 12 can be detected.

[0031] A detection signal indicating that an object has been detected and the location of the detected object is transmitted from the object sensor 14 to the processor 18.

[0032] Memory 16 is composed of components such as an HDD (Hard Disk Drive), SSD (Solid State Drive), eMMC (embedded Multi Media Card), ROM (Read Only Memory), or RAM (Random Access Memory). Information processing programs for operating each part of the information processing device 10 are stored in memory 16. These information processing programs can also be stored on a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory or CD-ROM. The information processing device 10 can read and execute these information processing programs from such storage media.

[0033] The processor 18 refers to a processor in a broad sense and consists of at least one of the following: a general-purpose processor (e.g., a CPU (Central Processing Unit)) and a dedicated processing unit (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a programmable logic device). The processor 18 may not consist of a single processing unit, but rather of multiple processing units located in physically separate locations working together. As shown in Figure 1, the processor 18 performs its functions as an object detection unit 20, an input determination unit 22, an operation determination unit 24, a guide presentation processing unit 26, and an information processing unit 28, according to the information processing program stored in the memory 16.

[0034] The object detection unit 20 detects objects in the spatial region facing the display 12 based on detection signals from the object sensor 14. The processing of the object detection unit 20 will be specifically explained with reference to Figures 2 and 3. Figures 2 and 3 show the spatial region 30 facing the display 12, and the operating body 32, of which at least a portion is located in the spatial region 30. The direction parallel to the display 12 (horizontal direction of the display 12) is defined as the Xp axis, the direction parallel to the display 12 and perpendicular to the Xp axis (vertical direction of the display 12) is defined as the Yp axis, and the direction perpendicular to the display 12 is defined as the Zp axis.

[0035] The spatial region 30 opposite the display 12 is the region that the display 12 passes through when the display 12 is translated in the positive direction of the Zp axis, and is a region within a predetermined distance from the display 12 in the Zp axis direction. The predetermined distance, that is, the length of the spatial region 30 in the Zp axis direction, is determined according to the detectable range of the object sensor 14. In other words, the predetermined distance may be the detectable distance of an object by the object sensor 14.

[0036] As shown in Figure 2, when an object approaches the display 12, the object sensor 14 detects the operator 32. Specifically, the object sensor 14 detects various parts of the operator 32 used to operate the information processing device 10, such as the tip of the index finger or the tip of the thumb of the operator 32. Since objects in the spatial domain 30 can move, the object detection unit 20 detects the position of each part of the object within the spatial domain 30 at unit time intervals. Here, the unit time interval is, for example, a few milliseconds or less. The object detection unit 20 transmits detection signals indicating the position of each part of the object to the processor 18.

[0037] The input determination unit 22 determines that the position A on the display 12 corresponding to the position of the nearest object among the positions of each part of the object detected by the object detection unit 20 is the instruction input position due to user contact or non-contact operation. Based on the detection signal from the object sensor 14, the input determination unit 22 compares the distance Lv from the display 12 in the Zp axis direction of each part in the spatial region 30. Then, among the parts in the spatial region 30, the part with the smallest distance Lv is determined to be the nearest object. In the case of a contact operation, the distance Lv at the nearest object is 0.

[0038] In the example shown in Figure 2, the parts of the object in the spatial region 30 detected by the object sensor 14 are typically shown as the tip of the index finger 32a and the tip of the thumb 32b of the operating body 32. The object detection unit 20 compares the distance Lva in the Zp axis direction between part 32a and the display 12, the distance Lvb in the Zp axis direction between part 32b and the display 12, and the distance Lv in the Zp axis direction between other parts of the object in the spatial region 30 and the display 12. Since distance Lva is the smallest, it determines that part 32a, which corresponds to distance Lva, is the nearest part.

[0039] Position A on the display 12, which corresponds to the position of the nearest neighbor, is a point on the display 12 with the same Xp and Yp coordinates as the nearest neighbor. That is, if the position of the nearest neighbor is represented by the coordinates (Xp,Yp,Zp)=(x,y,z) in XpYpZp space, then the instruction input position will be represented by the coordinates (Xp,Yp)=(x,y) on the display 12.

[0040] As described above, since objects in the spatial region 30 can move, the input determination unit 22 determines the nearest point and its corresponding position A on the display 12 at unit time intervals based on the detection signal from the object sensor 14.

[0041] The operation determination unit 24 determines the content of the operation on the information processing device 10 according to the determination result of the input determination unit 22 regarding the nearest contact point and the corresponding position A on the display 12. The content of the operation refers to all aspects of the operation, such as whether a non-contact or contact operation was performed, and which operation icon displayed on the display 12 was operated on.

[0042] For example, if an operation icon associated with some process is displayed at the instruction input position on the display 12 where the contact operation was performed, the operation determination unit 24 can determine that a contact operation has been performed on that operation icon by determining the instruction input position. Furthermore, if gesture operation is possible, the operation determination unit 24 can determine that a gesture operation has been performed in accordance with the temporal change in the instruction input position.

[0043] A gesture operation is an operation that can be performed in response to the user's movements while in contact with the operating surface. A gesture operation means an operation that corresponds to the movement of the operating body 32, or in other words, the movement (operation pattern) of the instruction input position. That is, by pre-associating the operation pattern of the instruction input position with the processing content, the operation determination unit 24 can detect the operation pattern of the instruction input position and determine the instruction content according to the detected operation pattern. The processor 18 executes the processing associated with the identified gesture.

[0044] For example, if an operation is performed to continuously move the instruction input position on the display 12 that has been touched, so as to slide it, within a predetermined time, it can be determined that an operation to scroll the operation screen in the direction of the slide has been performed. The operation determination unit 24 may also be able to determine other gesture operations, such as pinch-in, which reduces the size of the operation screen in response to the movement of multiple instruction input positions, and pinch-out, which enlarges the operation screen in response to the movement of multiple instruction input positions.

[0045] Furthermore, the operation determination unit 24 can determine if a non-contact operation has been performed on an operation icon associated with some process is displayed at the instruction input position on the display 12 where a non-contact operation was performed, by determining the instruction input position. For example, if the instruction input position of the operator 32 is maintained within the area of ​​the operation icon displayed on the display 12, and the distance Lv between the display 12 and the operator 32 approaches at a predetermined speed or faster, it can be determined that a non-contact operation has been performed on the operation icon. In this way, the user can perform gesture-based instruction input by moving the operator 32 in the spatial area 30 without touching the display 12 to the operator 32.

[0046] Even in the case of non-contact operation, by pre-associating the operation pattern of the instruction input position with the processing content, the operation determination unit 24 can detect the operation pattern of the instruction input position and determine the instruction content according to the detected operation pattern. The processor 18 executes the processing associated with the identified gesture.

[0047] Furthermore, the operation determination unit 24 also determines a non-contact state with respect to the operating surface. A non-contact state means that an object that is not in contact with the operating surface has been detected, but the operation is not valid for the object displayed on the operating surface, or the movement has been detected is not valid. A non-contact state occurs when a user is attempting to perform some operation on the information processing device 10 with the operating body 32, but does not know how to perform the operation.

[0048] A non-contact state can be defined as a state in which, for example, an object is detected by the object sensor 14 without contact, but the state remains in a way that is not valid as an operation on the object displayed on the operating surface for a predetermined reference time or longer. For example, a user may be trying to perform some operation on the information processing device 10 and has brought the operating body 32 close to the object sensor 14, but does not know how to operate it and is thinking without moving the operating body 32. In this case, the operation determination unit 24 determines that the user has been in a state where the operating body 32 is in a way that is not valid as an operation for a predetermined reference time while it is close to the object sensor 14.

[0049] Here, the reference time can be set as appropriate. The reference time can be fixed to a certain value, for example, or it can be changed according to the number, type, function, and usage status of objects displayed on the operation screen. For example, the more objects displayed on the operation screen, the longer the reference time set for that operation screen may be. Also, for example, the more types of objects displayed on the operation screen, the longer the reference time set for that operation screen may be. Also, for example, the setting of the reference time may be changed according to the content of the functions of the objects displayed on the operation screen. Also, for example, the less frequently the objects displayed on the operation screen are used, the longer the reference time set for that operation screen may be. Furthermore, the reference time may differ for each operation screen. For example, for each operation screen, it may be the statistical average or median of the time during which the user remains in a state where the operation is not valid when they are unsure of what to do.

[0050] Furthermore, a non-contact state can be defined as a state in which, for example, an object has been detected by the object sensor 14 without contact, but a predetermined movement pattern has been detected that indicates it is not a valid operation on the information processing device 10. For example, the operation determination unit 24 determines a non-contact state to be one in which a user attempts to perform some operation on the information processing device 10, bringing the operating body 32 close to the object sensor 14, and moving the operating body 32 across the entire operating surface, or continuously moving it in small increments on a single operation icon. The predetermined movement pattern indicating it is not valid may be determined, for example, by the three-dimensional movement of the position (x,y,z) of the nearest part of the detected object, or by the two-dimensional movement of the coordinates (x,y) on the display 12.

[0051] The guide presentation processing unit 26 performs a process to present guide information related to the information processing device 10. The guide information is not particularly limited as long as it relates to the information processing device 10. The guide information can be an explanation of how to operate the operation surface or an explanation of the function of an object displayed on the operation surface. For example, it can be information about operating the operation surface displayed on the display 12, information about operations for transitioning from the current operation screen to another operation screen, or information about the function of an object displayed on the operation surface. In this embodiment, the guide presentation processing unit 26 displays the guide information on the screen of the display 12 when the operation determination unit 24 determines that a specific non-contact state has been reached. The guide presentation processing unit 26 also performs a process to erase the guide information displayed on the display 12 when predetermined conditions are met.

[0052] The information processing unit 28 performs normal information processing in the information processing device 10. When the operation determination unit 24 determines that a contact operation has been performed on an operation icon displayed on the display 12, the information processing unit 28 executes the information processing associated with that operation icon. In addition, when the operation determination unit 24 determines that a touch or non-touch gesture operation has been performed, the information processing unit 28 executes the information processing associated with that gesture operation.

[0053] The processing performed by the information processing unit 28 is not particularly limited and may include any processing provided by the information processing unit 10. For example, if the information processing unit 10 is a multifunction device, the information processing unit 28 may perform copying, faxing, scanning, printing, and the like.

[0054] [Guide presentation processing in a specific non-contact state] Figure 4 is a flowchart showing the guide presentation process in a non-contact specific state according to this embodiment. The processor 18 executes the information processing program stored in the memory 16 to realize the processing in each step. The guide presentation process in a non-contact specific state will now be described with reference to Figure 4.

[0055] In step S10, non-contact operation on the operating surface is detected. Through the processing in this step, the processor 18 functions as an object detection unit 20 and an input determination unit 22. Based on the detection signal from the object sensor 14, it detects objects in the spatial area on the operating surface. The object detection unit 20 also determines the position of each part of the detected object and the position A of the nearest part as the instruction input position.

[0056] In step S12, a determination is made as to whether or not a specific non-contact state is in place. Through the processing in this step, the processor 18 functions as an operation determination unit 24. If a specific non-contact state is achieved, the process moves to step S16; otherwise, if any other contact or non-contact operation is performed, the process moves to step S14.

[0057] For example, if an object is detected by the object sensor 14 without contact, but the state in which it is not valid as an operation to the information processing device 10 is maintained for a predetermined reference time or longer, it is determined that this is a specific non-contact state and the process proceeds to step S16. Also, for example, if a movement that is not valid as a gesture operation is detected without contact on the display area of ​​one operation icon, it is determined that this is a specific non-contact state and the process proceeds to step S16. Also, for example, if a movement that is not valid as a gesture operation is detected evenly in two dimensions across the entire area of ​​the operation surface without contact, it is determined that this is a specific non-contact state and the process proceeds to step S16. Also, for example, if the position (x,y,z) of the nearest part of the operation body 32 in the spatial area 30 on the operation surface of the display 12 remains within a predetermined range in three dimensions for a predetermined reference period, and a movement that is not valid as a gesture operation is maintained, it is determined that this is a specific non-contact state and the process proceeds to step S16.

[0058] In step S14, normal information processing is performed in response to the contact or non-contact operation performed in step S12. Through the processing in this step, the processor 18 functions as an information processing unit 28. For example, if a contact operation is performed on an operation icon, the information processing associated with that operation icon is executed. Also, for example, if a gesture operation is performed by contact or non-contact, the processing associated with that gesture operation is executed.

[0059] In step S16, a process is performed to display guide information corresponding to a specific non-contact state. Through the processing in this step, the processor 18 functions as a guide presentation processing unit 26. The processor 18 performs a process to display guide information, for example, as shown below.

[0060] Figure 5 shows an example of a simple menu screen 40 displayed on the operating surface of the display 12. The menu screen 40 displays a reset button 42 and several operation icons 44. The reset button 42 is an icon that instructs the system to reset the operation on the menu screen 40. The operation icons 44 are icons that instruct the system to perform information processing in the information processing device 10. For example, the operation icons 44 are icons for performing operations related to functions such as copying, faxing, scanning (sending via email), scanning (saving to PC), box operation, and address book.

[0061] When the menu screen 40 is displayed, if a non-contact object is detected but the operation is not valid, and this state persists for a predetermined period of time or longer, guide information explaining the available operations and functions will be displayed on the displayed menu screen 40. For example, as shown in Figure 6, a guide display area 46 is provided on the operating surface of the display 12, and guide information is displayed in the guide display area 46.

[0062] In the example of the menu screen 40 in Figures 5 and 6, a guide display area 46 is provided in advance adjacent to the operation surface, and guide information is displayed in that area. However, the method is not limited to this. Guide information may be displayed while avoiding areas where proximity has been detected by the object detection unit 20 and the input determination unit 22. For example, as shown in Figure 7, if there is a proximity detection area 48 on the operation surface of the display 12 where proximity has been detected, the operation icons 44, etc., may be displayed in a reduced size, and the guide display area 46 may be provided to avoid the proximity detection area 48 and display guide information, as shown in Figure 8. Alternatively, as shown in Figure 9, at least a part of the reset button 42 and the operation icons 44 may be slid to display them, and the guide display area 46 may be provided to avoid the proximity detection area 48 and display guide information.

[0063] In this way, when a non-contact state is determined, guide information regarding the menu screen 40 displayed on the display 12 can be shown. This allows the user to understand the operation and functions of the menu screen 40.

[0064] Furthermore, if a non-contact state is determined, hints regarding possible operations and functions may be displayed as guide information on the reset button 42 and operation icon 44. For example, as shown in Figure 10, hint buttons 50 for displaying hints regarding operations and functions may be displayed in accordance with the respective positions of the reset button 42 and operation icon 44. Guide information regarding operations and functions is displayed by performing a contact click, touch, or non-contact gesture operation on the hint buttons 50 displayed in accordance with the reset button 42 and operation icon 44 that require the display of guide information. For example, as shown in Figure 11, a speech bubble area 52 may be provided on the operation screen, and guide information regarding operations and functions may be displayed in the speech bubble area 52. Alternatively, as in Figure 6, a guide display area 46 may be provided in advance adjacent to the operation surface, and guide information may be displayed in that area. Mini icons for displaying guide information may also be displayed.

[0065] In this way, when a non-contact state is determined, the operation and functions related to the reset button 42 and operation icon 44 can be displayed according to the specifications for each of the reset button 42 and operation icon 44. This allows the user to understand the operations and functions of the menu screen 40 as needed.

[0066] Furthermore, as shown by the arrows in Figure 12, if the operating body 32 makes a movement that is not valid as a gesture operation in a non-contact state on the display area of ​​one operation icon 44, it may be determined that a non-contact state has been specified and the process may proceed to step S16. At this time, the content of the explanation as guide information may be different depending on the trajectory in which proximity detection occurred.

[0067] For example, it can be determined that a non-contact state has been achieved if a certain percentage or more of the trajectory length of the two-dimensional position A of the nearest contact detected over a predetermined reference time is within the display area of ​​one operation icon 44. For example, it can be determined that a non-contact state has been achieved if 80% or more of the trajectory length of the two-dimensional position A of the nearest contact detected over a 10-second period is within the display area of ​​one operation icon 44. When it is determined that a non-contact state has been achieved in this way, guide information related to the operation icon 44 is displayed. For example, as shown in Figure 13, a callout area 52 may be displayed for the operation icon 44, and guide information related to the operation icon 44 may be displayed in the callout area 52.

[0068] However, the baseline percentage is not limited to 80% and may be set as appropriate. The baseline percentage may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on menu screen 40. For example, the more objects displayed on menu screen 40, the lower the baseline percentage may be set. Also, for example, the more types of objects displayed on menu screen 40, the lower the baseline percentage may be set. Also, for example, the baseline percentage may be changed according to the function of the objects displayed on menu screen 40. Also, for example, the less frequently an object is used on menu screen 40, the lower the baseline percentage may be set. Furthermore, the baseline percentage may differ for each operation screen.

[0069] Furthermore, the reference time is not limited to 10 seconds and can be set as appropriate. The reference time may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on the menu screen 40. For example, the more objects displayed on the menu screen 40, the longer the reference time may be. Also, for example, the more types of objects displayed on the menu screen 40, the longer the reference time may be. Also, for example, the setting of the reference time may be changed according to the function of the objects displayed on the menu screen 40. Also, for example, the less frequently an object is used on the menu screen 40, the longer the reference time may be. In addition, the reference time may differ for each operation screen.

[0070] This allows the system to identify buttons, icons, etc., that users want to use but are unsure how to operate, and to provide guide information about the operations and functions associated with those buttons or icons. Therefore, users can accurately understand the guide information regarding the buttons or icons they are unsure how to use.

[0071] Furthermore, as shown by the arrows in Figure 14, if the operating body 32 exhibits movements that are not valid as gesture operations evenly across the entire area on the menu screen 40 while remaining in a non-contact state, it may be determined that a non-contact state has been established and the process may proceed to step S16. For example, it may be determined that a non-contact state has been established if the proportion of the trajectory length of the two-dimensional position A of the nearest point detected over a predetermined reference time that is on the display area of ​​each operation icon 44 is less than a reference proportion. For example, it may be determined that a non-contact state has been established if less than 10% of the trajectory length of the nearest point position A detected over a 10-second period is on each operation icon 44.

[0072] However, the baseline percentage is not limited to 10% and may be set as appropriate. The baseline percentage may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on menu screen 40. For example, the more objects displayed on menu screen 40, the lower the baseline percentage may be set. Also, for example, the more types of objects displayed on menu screen 40, the lower the baseline percentage may be set. Also, for example, the baseline percentage may be changed according to the function of the objects displayed on menu screen 40. Also, for example, the less frequently an object is used on menu screen 40, the lower the baseline percentage may be set. Furthermore, the baseline percentage may differ for each operation screen.

[0073] Furthermore, the reference time is not limited to 10 seconds and can be set as appropriate. The reference time may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on the menu screen 40. For example, the more objects displayed on the menu screen 40, the longer the reference time may be. Also, for example, the more types of objects displayed on the menu screen 40, the longer the reference time may be. Also, for example, the setting of the reference time may be changed according to the function of the objects displayed on the menu screen 40. Also, for example, the less frequently an object is used on the menu screen 40, the longer the reference time may be. In addition, the reference time may differ for each operation screen.

[0074] If it is determined that a contactless operation has occurred, guide information regarding operations and functions on the entire menu screen 40, and operations for transitioning to other menu screens, will be displayed. For example, as shown in Figure 15, a callout area 54 may be provided to display guide information regarding operations for scrolling the menu screen 40 and guide information regarding operations for transitioning from the menu screen 40 to other menu screens.

[0075] This allows us to provide guide information about the operations and functions of the entire menu screen 40. Therefore, users can accurately grasp the guide information when they want to perform an operation but don't know how, or when they can't find what they're looking for on the menu screen 40.

[0076] Furthermore, if non-contact gesture operation is possible, guide information indicating how to perform the gesture operation may be displayed when a specific non-contact state is determined. For example, as shown in Figure 16, in the spatial area 30 on the operating surface of the display 12, if the three-dimensional position (x,y,z) of the nearest part of the operating body 32 remains within a predetermined reference range (x±Δx,y±Δy,z±Δz) for a predetermined reference period, and a movement that is not effective as a gesture operation is maintained, a specific non-contact state is determined.

[0077] If a specific non-contact state is detected, guide information indicating what kind of non-contact gesture operations are possible is displayed on the menu screen 40. For example, if "air tapping," a non-contact gesture operation, is to be displayed as guide information, an animation is displayed in which the image 56 of a "hand," an example of the operating body 32, is repeatedly enlarged and reduced over time, as shown in Figure 17. Alternatively, as shown in Figure 18, the menu screen 40 may be switched to a three-dimensional representation, and an animation may be displayed in which the image 58 of a "hand," an example of the operating body 32, is brought closer to the operating surface of the display 12.

[0078] Furthermore, if the system is configured to allow users to register gesture operations, the registered gesture operations may be displayed one by one in sequence. For example, the possible gesture operations on the menu screen 40 currently displayed on display 12 may be displayed one by one in sequence.

[0079] However, the reference range (x±Δx, y±Δy, z±Δz) can be set as appropriate. The reference range may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on the menu screen 40. For example, the more objects displayed on the menu screen 40, the smaller the reference range may be. Also, for example, the more types of objects displayed on the menu screen 40, the smaller the reference range may be. Also, for example, the settings of the reference range may be changed according to the function of the objects displayed on the menu screen 40. Also, for example, the less frequently the objects displayed on the menu screen 40 are used, the smaller the reference range may be. Furthermore, the reference range may differ for each operation screen.

[0080] Furthermore, the reference time can be set as appropriate. The reference time may be changed according to, for example, the number, type, function, usage status, and size of the display area of ​​the objects displayed on menu screen 40. For example, the more objects displayed on menu screen 40, the longer the reference time may be. Also, for example, the more types of objects displayed on menu screen 40, the longer the reference time may be. Also, for example, the setting of the reference time may be changed according to the function of the objects displayed on menu screen 40. Also, for example, the less frequently an object is used on menu screen 40, the longer the reference time may be. Furthermore, the reference time may differ for each operation screen.

[0081] The method of presenting guide information is not limited to the examples above; any method that allows the user to understand the guide information is acceptable. For example, instead of displaying it as visual information, the guide information may be presented using auditory information such as sound or tactile information such as vibration.

[0082] Furthermore, the display of guide information may be stopped depending on predetermined conditions. For example, the display of guide information may be stopped when a contact operation is performed, or when an effective non-contact operation is performed on the operating surface. Also, for example, the display of guide information may be stopped when the specific non-contact state is resolved or when the object sensor 14 no longer detects an object.

[0083] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications are possible without departing from the spirit of the invention.

[0084] (Note) (((1))) Equipped with a processor, The aforementioned processor, The system detects a specific state of non-contact between the user and the operating surface. When the specific state is detected, the system will display an explanation of how to operate the operating surface or an explanation of the function of the object displayed on the operating surface. Information processing device. (((2))) The information processing apparatus according to (((1))), wherein the specified state is a state in which proximity detection to the operating surface continues for a certain period of time while no operation is being performed on the object displayed on the operating surface. (((3))) The information processing apparatus according to (((1))) or (((2))), wherein the content of the above description differs depending on the trajectory in which proximity detection was performed with respect to the operating surface. (((4))) The aforementioned operation method is an explanation of a gesture operation method corresponding to the user's movement on the operation surface, as described in any one of items (((1))) to (((3))). (((5))) The information processing apparatus described in (((4))) is an explanation of a method of performing gesture operations in response to user movements that are performed non-contact with respect to the operating surface. (((6))) An information processing device according to any one of (((1))) to (((5))), which displays the description while avoiding the area where proximity detection has been performed on the operating surface. (((7))) On the computer, The system detects when the user is not touching the operating surface. When the specific state is detected, the system will display an explanation of how to operate the operating surface or an explanation of the function of the object displayed on the operating surface. Information processing program.

[0085] According to the inventions described in (((1))) and (((7))), when a non-contact UI detects a state in which the user is unaware of the device's functions or operation, this information can be presented as guide information. According to the invention of (((2))), guide information can be provided when proximity detection to the operating surface continues for a certain period of time and no operation is being performed on the object displayed on the operating surface. According to the invention of (((3))), guide information can be provided by displaying the guide information differently depending on the trajectory in which proximity detection was performed. According to the invention of (((4))), a method of gesture operation that responds to the user's movements on the operating surface can be presented. According to the invention described in (((5))), a method of gesture operation that responds to user movements and is performed without contact with the operating surface can be presented. According to the invention of (((6))), the guide information can be made easier to understand by displaying the explanation on the operating surface while avoiding the area where proximity detection has been performed. [Explanation of symbols]

[0086] 10 Information processing device, 12 Display, 14 Object sensor, 16 Memory, 18 Processor, 20 Object detection unit, 22 Input determination unit, 24 Operation determination unit, 26 Guide presentation processing unit, 28 Information processing unit, 30 Spatial area, 32 Operation body, 32a, 32b Parts, 40 Menu screen, 42 Reset button, 44 Operation icon, 46 Guide display area, 48 Proximity detection area, 50 Hint button, 52, 54 Speech bubble area, 56, 58 Hand image.

Claims

1. Equipped with a processor, The aforementioned processor, The system detects a specific state of non-contact between the user and the operating surface. When the specific state is detected, the system will display an explanation of how to operate the operating surface or an explanation of the function of the object displayed on the operating surface. The content of the above explanation will differ depending on the trajectory in which proximity detection was performed to the aforementioned operating surface. Information processing device.

2. The information processing apparatus according to claim 1, wherein the specified state is a state in which proximity detection to the operating surface continues for a certain period of time while no operation is being performed on the object displayed on the operating surface.

3. The information processing apparatus according to claim 1, wherein the operation method is an explanation of a method of gesture operation in response to the user's movement on the operation surface.

4. The information processing apparatus according to claim 3, wherein the operation method is a description of a method of gesture operation that corresponds to the movement of a user and is performed without contact with the operation surface.

5. The information processing apparatus according to any one of claims 1 to 4, wherein the description is displayed on the operating surface while avoiding the area where proximity detection has been performed.

6. On the computer, The system detects when the user is not touching the operating surface. When the specific state is detected, the system will display an explanation of how to operate the operating surface or an explanation of the function of the object displayed on the operating surface. The content of the above explanation will differ depending on the trajectory in which proximity detection was performed to the aforementioned operating surface. Information processing program.

Images