Non-contact input apparatus and input assistance application
The non-contact input device allows three-dimensional input by using a detection unit to measure two-dimensional positions and distances, displaying a second pointer to facilitate three-dimensional interactions in devices limited to two-dimensional capabilities, thus overcoming the limitations of conventional devices.
Patent Information
- Application Number
- PCT/JP2024/007967
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-03-04
- Publication Date
- 2024-11-14
AI Technical Summary
Conventional input terminal devices require dedicated application programs to detect three-dimensional positions for non-contact input, limiting their functionality in devices equipped with only two-dimensional position handling capabilities.
A non-contact input device with an operation surface, a detection unit for two-dimensional position measurement, and a control device displaying a second pointer based on two-dimensional position and distance, enabling three-dimensional input in devices with two-dimensional position handling capabilities.
Enables non-contact input using three-dimensional movements in devices that only support two-dimensional positions, enhancing user interaction without requiring specialized application programs.
Smart Images

Figure JP2024007967_14112024_PF_FP_ABST
Abstract
Description
Non-contact input device and input support application
[0001] The present disclosure relates to a non-touch input device and an input assistance application.
[0002] Conventionally, input terminal devices that input user operations via a position input object include a position detection unit that detects the position of the position input object operated by the user in a non-contact manner, a display unit that displays a pointer based on the position detected by the position detection unit, and an operation processing control unit that executes corresponding operation processing based on the position detected by the position detection unit. The operation processing control unit has multiple operation modes for executing the operation processing, one of which includes a state in which no operation processing is executed other than moving the pointer according to the position of the position input object. The operation processing control unit switches between the multiple operation modes when the user performs a specific operation via the position input object (see, for example, Patent Document 1). It is also known to change the size of the pointer based on the three-dimensional spatial position of a hand approaching the operation screen, depending on the position of the hand on the operation screen and the distance between the hand and the operation screen (see, for example, Patent Document 2).
[0003] International Publication No. 2007-064771 Japanese Patent Application Laid-Open No. 2010-244422
[0004] However, conventional input terminal devices use a dedicated application program capable of detecting three-dimensional positions in order to detect the three-dimensional position of a position input object (object) without contact, and therefore cannot be realized with a device equipped with an application program that can only handle two-dimensional positions.
[0005] Therefore, the object is to provide a non-contact input device and an input assistance application that enable non-contact input with three-dimensional movements in a device equipped with an application program that can only handle two-dimensional positions.
[0006] A non-contact input device according to an embodiment of the present disclosure includes an operation surface, a detection unit that measures a two-dimensional position of an object that is operated on the operation surface without contact and a distance from the operation surface to the object, a display that is disposed on the back side of the operation surface, and a control device having an OS that uses a planar GUI and an input assistance application, wherein the input assistance application outputs to the OS a command to display a second pointer that is different from a first pointer that the OS displays on the display as an overlay on the display in accordance with the two-dimensional position and the distance, and outputs the two-dimensional position and a click event to the OS based on the two-dimensional position and the distance.
[0007] It is possible to provide a non-contact input device and an input assistance application that enable non-contact input with three-dimensional movements in a device equipped with an application program that can only handle two-dimensional positions.
[0008] 1 is a diagram illustrating an example of the configuration of a non-contact input device of an embodiment. FIG. 1 is a diagram illustrating an example of the configuration of an electrostatic sensor, a control device, etc. of a non-contact input device of an embodiment. FIG. 2 is a diagram illustrating an example of four regions according to the distance in the Z direction from the operation surface of the non-contact input device of an embodiment. FIG. 3 is a diagram illustrating an OS, an input assistance application, and a predetermined application program of a control unit of the non-contact input device of an embodiment. FIG. 4 is a diagram illustrating an index of the positional accuracy of an object in the non-contact input device of an embodiment. FIG. 5 is a diagram illustrating an index of the positional accuracy of an object in the non-contact input device of an embodiment. FIG. 6 is a diagram illustrating an example of a display on the display of the non-contact input device of an embodiment. FIG. 7 is a diagram illustrating an example of a display on the display of the non-contact input device of an embodiment. FIG. 8 is a diagram illustrating an example of a display on the display of the non-contact input device of an embodiment. FIG. 9 is a diagram illustrating an example of a display on the display of the non-contact input device of an embodiment. FIG. 10 is a diagram illustrating an example of a display on the display of the non-contact input device of an embodiment. 1 is a diagram illustrating an example of a modified example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 2 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 3 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 4 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 5 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 6 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment. FIG. 7 is a diagram illustrating an example of processing executed by an input assistance application of a non-contact input device of an embodiment.FIG. 1 is a diagram illustrating an example of processing executed by an input assistance application of the non-contact input device of an embodiment. FIG. 2 is a diagram illustrating an example of processing executed by an input assistance application of the non-contact input device of an embodiment. FIG. 3 is a diagram illustrating an example of processing executed by an input assistance application of the non-contact input device of an embodiment. FIG. 4 is a diagram illustrating an example of processing executed by an input assistance application of the non-contact input device of an embodiment. FIG. 5 is a diagram illustrating an example of a setting screen. FIG. 6 is a diagram illustrating an example of a folder configuration. FIG. 7 is a flowchart illustrating an example of setting processing.
[0009] Hereinafter, a non-contact input device and an embodiment to which an input assistance application according to the present disclosure is applied will be described.
[0010] 1 is a diagram showing an example of the configuration of a non-contact input device 100 according to an embodiment of the present invention, in which a display 110 displays an input image.
[0011] Fig. 2 is a diagram showing an example of the configuration of the electrostatic sensor 120, the control device 130, etc. of the non-contact input device 100. Fig. 3 is a diagram showing four regions according to the distance in the Z direction from the operation surface 105A of the non-contact input device 100.
[0012] In the following description, an XYZ coordinate system is defined. The X axis is the first axis, the Y axis is the second axis, and the Z axis is the third axis. The direction parallel to the X axis (X direction), the direction parallel to the Y axis (Y direction), and the direction parallel to the Z axis (Z direction) are perpendicular to each other. In the following description, the -Z direction is the direction approaching the electrostatic sensor 120, and the +Z direction is the direction away from the electrostatic sensor 120. In addition, a planar view refers to a view from the XY plane. In the following description, the length, width, thickness, etc. of each part may be exaggerated to make the configuration easier to understand.
[0013] The non-contact input device 100 may be, for example, a tablet-type input device that is placed in a store or facility and used by an unspecified number of users, or an input unit of an ATM (Automatic Teller Machine), or may be an input unit of a cooking appliance that needs to be kept clean.
[0014] <Overall Configuration of Non-Contact Input Device 100> The non-contact input device 100 includes a housing 101, a top panel 105, a display 110, an electrostatic sensor 120, an input sensor circuit 125A, an image display circuit 125B, and a control device 130. The input sensor circuit 125A calculates XYZ coordinates from the measurement results of the electrostatic sensor 120. Although the control device 130 (see FIG. 2) is omitted in FIG. 1, the control device 130 is, for example, provided below the display 110 and the electrostatic sensor 120 inside the housing 101. The non-contact input device 100 includes the electrostatic sensor 120 and the control device 130 shown in FIG. 2.
[0015] <Housing 101 and Top Panel 105> The housing 101 is a case made of resin, metal, or the like that houses the display 110, electrostatic sensor 120, and control device 130. The display 110 is disposed below the transparent electrostatic sensor 120, and is visible through an operation surface 105A, which is the upper surface of a transparent top panel 105 provided in an opening at the top of the housing 101. The electrostatic sensor 120 may be retrofitted to an existing display 110. The display and control device 130 may be integrated as in a tablet computer. The display and control device 130 may be separate as in a desktop computer.
[0016] <Four Areas of Non-Contact Input Device 100 and Operation Method> The non-contact input device 100 can be operated in both a state where a pointer such as a user's hand is not in contact with the operation surface 105A, and a state where a pointer such as a user's hand is in contact with the operation surface 105A. Here, the modes executed by the input assistance application of the non-contact input device 100 will be described together. The input assistance application has four modes: decision mode, selection mode, proximity mode, and standby mode.
[0017] The non-contact input device 100 is an input device that is operated by a user performing a pointing operation. The pointing operation is an operation performed by holding a finger substantially perpendicular to the operation surface 105A. The number of fingers used in the pointing operation may be multiple, but it is preferable to use one finger.
[0018] When performing such a pointing operation, if the finger is not approximately perpendicular to the operation surface 105A, the entire palm approaches the operation surface 105A, making it difficult to measure the position of the fingertip FT.
[0019] Unless otherwise specified, the following describes a case where a user performs a pointing operation with a fingertip FT as a pointer. In addition, hereinafter, performing an operation (proximity operation, selection operation, decision operation, or touch operation) with the fingertip FT will be simply expressed as an operation (proximity operation, selection operation, decision operation, or touch operation) with the fingertip FT.
[0020] There are three types of operation methods for the non-contact input device 100: a proximity operation, a selection operation, and a decision operation.
[0021] 3 to distinguish between the four operation methods. The four areas are, in order from the operation surface 105A side, a decision area, a selection area, a proximity area, and a standby area.
[0022] <Decision Area, Selection Area, Proximity Area, and Waiting Area> The decision area is a first area whose Z-directional distance from the operation surface 105A is shorter than Z1 (e.g., 2 cm), and is an area in which the input assistance application sets the mode of the input assistance application to decision mode. Z1 is an example of a first threshold. When the input assistance application determines that the fingertip FT is located within the decision area, a decision operation is performed. Note that a Z-directional distance of 0 cm (contact) from the operation surface 105A is also included in the decision area. Note that the relationship between Z-directional distance and capacitance is affected by individual differences and the installation environment. However, if the operator and the usage environment are the same, capacitance is inversely proportional to Z-directional distance. Therefore, if it is sufficient to measure the relative distance to the same operator, capacitance can be considered as distance. Note that the decision area and decision operation will be described in detail later.
[0023] Instead of the distance in the Z direction, the input assistance application uses a value (Z value) proportional to the capacitance between the fingertip FT and the electrostatic sensor 120. The input sensor circuit 125A calculates the position (XY coordinates) of the fingertip FT facing the operation surface 105A and a value (Z value) proportional to the capacitance from the operation surface 105A to the fingertip FT, based on the capacitance at multiple intersections between the multiple sensor electrodes 121X and the multiple sensor electrodes 121Y.
[0024] The decision area is an area where the Z value is greater than Cz1 (e.g., 60). Cz1 is a value proportional to the capacitance corresponding to the distance Z1. Cz1 is an example of a first threshold value. In this embodiment, the decision area also includes the case where the fingertip FT touches the operation surface 105A. However, it is also possible to determine from the Z value that the fingertip FT has touched the operation surface 105A. For example, it is possible to design the Z value to be 100 when the fingertip FT lightly touches the operation surface 105A. If the Z value is greater than 100, it is determined that the fingertip FT is in the contact area, and operation may be performed in contact mode.
[0025] The selection area is a second area whose distance from the operation surface 105A is equal to or greater than Z1 and shorter than Z2 (e.g., 4 cm), which is longer than Z1, and is an area in which the input assistance application sets the mode of the input assistance application to the selection mode. Z2 is an example of a second threshold. When the input assistance application determines that the fingertip FT is located within the selection area, a selection operation is performed. Details of the selection operation will be described later.
[0026] The selected region is a region where the Z value is greater than Cz2 (e.g., 40) and less than or equal to Cz1. Cz2 is a value smaller than Cz1. Cz2 and Cz1 are values proportional to the capacitances corresponding to the distances Z2 and Z1, respectively. The capacitance Cz2 is an example of a second threshold value.
[0027] The proximity area is a third area whose distance from the operation surface 105A is equal to or greater than Z2 and shorter than Z3 (e.g., 7 cm), which is longer than Z2, and is an area in which the input assistance application sets the mode of the input assistance application to proximity mode. Z3 is an example of a third threshold. When the input assistance application determines that the fingertip FT is located within the proximity area, a proximity operation is performed. The proximity area is the area farthest from the operation surface 105A among areas in which the XY coordinates of the target object can be calculated. Details of the proximity operation will be described later.
[0028] The proximity region is a region where the Z value is greater than Cz3 (e.g., 10) and less than or equal to Cz2. Cz3 is a value smaller than Cz2. Cz3 and Cz2 are values proportional to the capacitances corresponding to the distances Z3 and Z2, respectively. Cz3 is an example of a third threshold value.
[0029] The standby area is a fourth area whose distance from the operation surface 105A is longer than Z3, and is an area where the input assistance application sets the mode of the input assistance application to standby mode. The standby area is an area where the Z value is equal to or less than Cz3. Note that the capacitance value measured when there is no fingertip FT around the operation surface 105A is equal to the reference value. The Z value is calibrated so that it is 0 when there is no fingertip FT around the operation surface 105A. The input assistance API may turn off the power of the display in standby mode.
[0030] <Proximity operation, selection operation, decision operation, and contact operation> A proximity operation is an operation in which the fingertip FT is brought close to the operation surface 105A of the non-contact input device 100 without touching it, and is an operation for switching the display of the display 110 from the standby mode display to the proximity mode display.
[0031] A selection operation is an operation in which, after performing a proximity operation, the fingertip FT is brought even closer to the operation surface 105A of the non-contact input device 100 without touching the operation surface 105A, and a GUI button displayed on the display 110 is selected.
[0032] A confirmation operation is an operation in which, after a selection operation, the fingertip FT is brought closer to the operation surface 105A of the non-contact input device 100 without touching it, thereby issuing a click event. When a click event is issued on a GUI button, the OS 140 and the application program 160 decide the operation input. The confirmation operation is a non-contact operation input, in which the non-contact input device 100 is operated without touching the operation surface 105A with the fingertip FT. Operation inputs performed by non-contact selection operations and confirmation operations may be referred to as hover inputs or touchless inputs. Note that even when the fingertip FT touches the operation surface 105A of the non-contact input device 100, this may be treated as confirmation mode. Alternatively, when the fingertip FT touches the operation surface 105A of the non-contact input device 100, the input may be immediately confirmed. Alternatively, when the fingertip FT touches the operation surface 105A of the non-contact input device 100, a warning screen may be displayed.
[0033] <Display 110> The display 110 is a liquid crystal display, an organic electroluminescence (EL) display, or the like. The display 110 is a display for realizing a GUI (Graphical User Interface). The display 110 displays an image (desktop screen, etc.; not shown) provided by the graphical shell of the OS, a second pointer 112 overlaid by the input assistance application 150, and an application software image 115. The input assistance application 150 overlays the second pointer 112 by placing it at the top of the Z order (the order closest to the viewpoint). The application software image 115 includes an image of a GUI button 110A. The GUI button 110A is an operation unit, and is shaped, for example, like a push button. The application software is designed to be operated with a two-dimensional position input device such as a mouse.
[0034] FIG. 1 shows an example of the screen of an ordering terminal at a restaurant. The ordering screen shows a total of 17 GUI buttons 110A, including eight GUI buttons 110A for menu items and nine GUI buttons 110A in the form of a numeric keypad, such as numbers. The 17 GUI buttons 110A are arranged in three or four rows in the Y direction and five rows in the X direction. The rows extend in the X direction, and the Y direction extends in the column direction. Note that the GUI buttons 110A are not limited to the screen of an ordering terminal at a restaurant, and may be any other application as long as they are an operation screen for a touch panel.
[0035] <Electrostatic Sensor 120, Input Sensor Circuit 125A, Image Display Circuit 125B> The electrostatic sensor 120 is disposed on top of the display 110, and as shown in FIG. 2, has a plurality of sensor electrodes 121X extending in the X direction and a plurality of sensor electrodes 121Y extending in the Y direction. The electrostatic sensor 120 is also integrally provided with an input sensor circuit 125A. The display 110 is also integrally provided with an image display circuit 125B. The input sensor circuit 125A is connected between the wiring 122X, 122Y and the control device 130. The image display circuit 125B is connected between the display 110 and the control device 130.
[0036] The sensor electrodes 121X and 121Y are connected to the control device 130 via wiring 122X and 122Y and an input sensor circuit 125A. Such an electrostatic sensor 120 can be formed by forming a transparent conductive film such as ITO (Indium Tin Oxide) on the surface of transparent glass and patterning it into the sensor electrodes 121X and 121Y and wiring 122X and 122Y. The electrostatic capacitance detected by the electrostatic sensor 120 is input to the control device 130.
[0037] 2 shows a plurality of sensor electrodes 121X and a plurality of sensor electrodes 121Y. Note that it is preferable that the distance between the sensor electrodes 121X and the distance between the sensor electrodes 121Y are narrower than the distance between the GUI buttons 110A. In other words, it is preferable to use electrostatic sensors 120 that correspond to the distance between the GUI buttons 110A.
[0038] The input sensor circuit 125A is mounted on a wiring board. The input sensor circuit 125A is located between the wiring 122X, 122Y, and the control device 130. The input sensor circuit 125A sequentially selects the multiple wirings 122X and 122Y and performs analog-to-digital (AD) conversion of the electrostatic capacitance of the electrostatic sensor 120. The input sensor circuit 125A calculates the X and Y coordinates of the fingertip FT and a Z value proportional to the capacitance between the operation surface 105A and the fingertip FT from the capacitance values of each wiring. The input sensor circuit 125A is capable of outputting XY coordinates in the same format as a touch-operated digitizer, as well as its own XYZ output format. The input assistance application 150 sends a command to stop the digitizer-format XY output to the input sensor circuit 125A via a coordinate device driver. The input assistance application 150 performs processing based on the XYZ output. The image display circuit 125B is provided between the display 110 and the control device 130, and displays an image on the display 110 in accordance with image data sent from the control device 130.
[0039] <Control device 130> The control device 130 has a control unit 131 and a memory 132. The control device 130 is realized by a computer including a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), an input / output interface, an internal bus, etc. The control unit 131 represents the functions of the program executed by the control device 130 as functional blocks. The memory 132 functionally represents the memory of the control device 130.
[0040] The control unit 131 controls the operation of the non-contact input device 100. The control unit 131 receives XY coordinates and Z values input from the input sensor circuit 125A. The control unit 131 controls the display of images on the display 110 via the image display circuit 125B. The control unit 131 outputs commands corresponding to the operation content determined by the determination operation of the fingertip FT. For example, if the non-contact input device 100 is an ordering terminal in a restaurant, it transmits the names and quantities of dishes ordered by a customer to an order management terminal in the kitchen.
[0041] The control unit 131 has an OS that uses a planar GUI, a coordinate device driver, a display device driver, an input support application, and a predetermined application program. Here, Fig. 4 is used to explain the OS, input support application, and predetermined application program of the control unit 131. Fig. 4 is a diagram explaining the OS, input support application, and predetermined application program of the control unit 131.
[0042] 4 shows the OS 140, input assistance application 150, and application program 160 of the control unit 131. The OS 140 is an OS (Operating System) that uses a planar GUI, such as Windows (registered trademark) or Android (registered trademark). An OS that uses a planar GUI does not provide a UI (User Interface) that is suitable for three-dimensional input. The application program 160 is an application program for an ordering terminal at a restaurant, etc.
[0043] The OS 140 has a coordinate device driver 141 and a display device driver 142. The coordinate device driver 141 passes the X, Y coordinates and Z values input from the electrostatic sensor 120 to the input assistance application 150. When the X, Y coordinates and a click event are input from the input assistance application 150, the OS 140 causes the application program 160 to execute a command corresponding to the button on the input X, Y coordinates. The OS 140 also passes data of the image to be displayed, the X, Y position of the image to be displayed, and the Z order of the image to be displayed (order of distance from the viewpoint) to the display device driver 142. The input assistance application 150 places the second pointer 112 at the top of the Z order.
[0044] The display device driver 142 controls the display of the second pointer 112 at the X and Y coordinates input from the input assistance application 150, and controls the display of an image in accordance with a command input from the application program 160 when a confirmation operation is performed. The OS treats the second pointer 112 simply as an image.
[0045] The input assistance application 150 is an application that provides input assistance to a user using the non-contact input device 100. The input assistance application 150 may perform processing to calculate an index of positional accuracy of the fingertip FT. Based on the X and Y coordinates of the fingertip FT and the index of positional accuracy, the input assistance application 150 performs processing to cause the OS 140 to display an image of a second pointer 112, which is different from the first pointer 111 displayed by the OS 140 on the display 110. Furthermore, when a confirmation operation is performed, the input assistance application 150 outputs the X and Y coordinates at the time of the confirmation operation to the OS 140 as the position of the mouse pointer, and then issues a click event. The OS 140 notifies the application program 160 that a GUI button corresponding to the position of the mouse pointer when the click event is issued has been pressed. Other processing performed by the input assistance application 150 and details of each processing will be described later.
[0046] The application program 160 is designed to be operated using a normal mouse (which outputs two-dimensional coordinates) or a touch panel. When the application program 160 is notified by the OS 140 that a click event has been issued on a GUI button, the application program 160 performs processing according to the clicked GUI button.
[0047] Existing devices that do not support three-dimensional input (contactless input) may be used for the display 110, control device 130, OS 140, display device driver 142, and application program 160. By fixing an electrostatic sensor 120 capable of contactless input to the existing display 110 and installing a coordinate device driver 141 and an input assistance application 150 in the control device 130, the existing device becomes a contactless input device.
[0048] <Calculation of XY Coordinates and Z Values by Input Sensor Circuit 125A> The input sensor circuit 125A scans the plurality of sensor electrodes 121X row by row and the plurality of sensor electrodes 121Y column by column, converting the capacitances at the plurality of intersections between the plurality of sensor electrodes 121X and the plurality of sensor electrodes 121Y into digital values. The control unit 131 counts the change in the capacitance digital value output and calculates a difference value ΔAD at each intersection. The difference value ΔAD is a count value of the change in the output of the input sensor circuit 125A relative to a reference value. The reference value is a value proportional to the capacitance at each intersection of the sensor electrodes 121X and 121Y when there is no object such as a fingertip FT around the sensor electrodes 121X and 121Y. The input sensor circuit 125A calculates the XY coordinates on the display facing the fingertip FT from the difference values at each intersection, and selects the largest ΔAD among the difference values ΔAD at each intersection as the Z value. The Z value is a value proportional to the capacitance between the fingertip FT and the operation surface. The XY coordinates and the Z value may be calculated by the control unit 131.
[0049] It is also possible to use interpolation to increase the resolution beyond the spacing between the sensor electrodes 121X and the spacing between the sensor electrodes 121Y. In this case, the spacing between the sensor electrodes 121X and the spacing between the sensor electrodes 121Y may be wider than the spacing between the GUI buttons 110A. The coordinate device driver 141 may calculate the XY coordinates and Z values. When the coordinate device driver 141 calculates the XY coordinates and Z values, the hardware that performs the calculations is the control unit 131.
[0050] <Index of Positional Accuracy of Object> The input assistance application 150 calculates an index of positional accuracy of the object when displaying the second pointer 112 on the display 110. The object is an object whose XY coordinates and the distance in the Z direction from the operation surface 105A are calculated based on the capacitance between the electrostatic sensor 120 and the object, and is a part of the body such as the hand or fingertip of the user of the non-contact input device 100. Here, the index of positional accuracy of the object will be described.
[0051] 5A and 5B are diagrams illustrating an index of the positional accuracy of an object. The non-contact input device 100 uses the amount of variation in the X and Y coordinates (two-dimensional position) of the fingertip FT as an index of the positional accuracy of the object.
[0052] 5A shows an example of the distribution of capacitance in the X direction measured at times t1 and t2. Assume that the X coordinate of the fingertip FT at time t1 is calculated as X1, and the X coordinate of the fingertip FT at time t2 is calculated as X2.
[0053] Even if the fingertip FT is not moving, the X coordinate fluctuates due to noise. There is a difference between the X coordinate X1 at time t1 and the X coordinate X2 at time t2, and this coordinate difference appears as a fluctuation amount of the coordinate. The same applies to the Y coordinate.
[0054] The non-contact input device 100 uses the amount of fluctuation in coordinates measured within a certain period of time as an index of positional accuracy. More specifically, it uses the standard deviation of the measured coordinates as an index of positional accuracy. For example, if the second pointer 112 is a circle with a radius equal to 1σ of the standard deviation, the probability that a finger is located within the second pointer 112 is approximately 68%, and if the second pointer 112 is a circle with a radius equal to 2σ, the probability that a finger is located within the second pointer 112 is approximately 95%. In this way, by representing the positional accuracy by the size of the pointer, it is possible to provide intuitive feedback of the positional accuracy to the user.
[0055] Fig. 5B shows an example of the capacitance distribution in the X direction when a fingertip FT is held upright on the operation surface 105A and gradually brought closer to the operation surface 105A. The fingertip FT on the left side in Fig. 5B is the furthest from the operation surface 105A, and the fingertip FT on the right side in Fig. 5B is the closest to the operation surface 105A. The double-headed arrows in Fig. 5B indicate the range in which capacitance is detected. Note that while Fig. 5B shows the capacitance distribution in the X direction, as explained in Fig. 5C, the capacitance distribution in the Y direction is similar due to symmetry.
[0056] When the fingertip FT is far from the operation surface 105A (on the left side), the electrostatic sensor 120 is affected by the fingertip FT over a wide range, resulting in a flat capacitance distribution. If the magnitude of the noise is the distance between the two dotted lines, the range that can be detected as the X coordinate of the fingertip FT is the range with a larger capacitance than the lower dotted line. In other words, the range in the X direction in which the fingertip FT can be detected is the wide range indicated by the left-right arrow.
[0057] Furthermore, when the fingertip FT is close to the operation surface 105A (on the right side), the electrostatic sensor 120 is affected by the fingertip FT over a narrow range, resulting in a steep capacitance distribution. If the magnitude of the noise is the distance between the two dotted lines, the range that can be detected as the X coordinate of the fingertip FT is the range with a larger capacitance than the lower dotted line. In other words, the range in the X direction in which the fingertip FT can be detected is the narrow range indicated by the left-right arrow.
[0058] Furthermore, when the distance of the fingertip FT from the operation surface 105A is intermediate as shown in the center of FIG. 5B, the distribution of the capacitance in the X direction is intermediate between the gradual distribution on the left side and the steep distribution on the right side.
[0059] In this way, the range in which the measured XY coordinates fluctuate due to noise is larger when the fingertip FT is farther from the operation surface 105A (left side) than when the fingertip FT is closer to the operation surface 105A (right side). The larger the area over which the capacitance is detected, the larger the amount of fluctuation in the coordinates.
[0060] 5C shows an example of the capacitance distribution in the X direction when the fingertip FT is held upright on the operation surface 105A (left side) and when the fingertip FT is held horizontally on the operation surface 105A (right side). The double-headed arrows indicate the range in which capacitance is detected. Note that while FIG. 5C shows the capacitance distribution in the X direction, the capacitance distribution in the Y direction is similar.
[0061] As shown on the left side of Fig. 5C, when the fingertips FT are held upright during operation, the capacitance distribution becomes steeper and the range in which the fingertips FT are detected becomes narrower. As shown on the right side of Fig. 5C, when the fingertips FT are held down and the palm is close to the operation surface 105A during operation, the capacitance distribution becomes flatter and the range in which the fingertips FT are detected becomes wider.
[0062] In this way, the amount of fluctuation in the measured XY coordinates due to noise is greater when the fingertip FT is operated with its fingertip FT laid down, as shown on the right side of Figure 5C, than when the fingertip FT is operated with its fingertip FT upright, as shown on the left side of Figure 5C.
[0063] As explained above using FIGS. 5A, 5B, and 5C, the amount of fluctuation in the measured XY coordinates due to noise varies depending on the distance from the fingertip FT to the operation surface 105A, the orientation of the fingertip FT, and the like. Furthermore, the amount of fluctuation also changes depending on the magnitude of the noise. The amount of fluctuation also increases when the fingertip FT is actually moved. If the amount of fluctuation increases for any reason, the measurement accuracy is poor, so the second pointer 112 is displayed larger. Furthermore, if the amount of fluctuation decreases, the second pointer 112 is displayed smaller. In this way, the position accuracy is represented by the size of the second pointer, allowing the user to intuitively feedback the position accuracy.
[0064] 5C, the amount of variation in the measured XY coordinates is larger when the fingertips FT are laid down and the palm is close to the operation surface 105A than when the fingertips FT are held upright. Therefore, if the amount of variation in the XY coordinates is larger than a predetermined threshold, the non-contact input device 100 may determine that the fingertips FT are not held upright near the operation surface 105A and display a message urging the user to hold the fingertips FT upright near the operation surface 105A.
[0065] 5A to 5C, since the X and Y coordinates calculated by the input sensor circuit 125A have fluctuations, the input assistance application 150 calculates the X and Y coordinates of the fingertip FT as a moving average of the X coordinates acquired at multiple time points and a moving average of the Y coordinates acquired at multiple time points. The multiple time points are the most recent time points for calculating the moving average, and the interval between the time points is several milliseconds.
[0066] The input assistance application 150 outputs a command to the OS 140 to display an image of the second pointer at the X and Y coordinates calculated as the moving average.
[0067] The XY coordinates of the fingertip FT detected by the non-contact input device 100 using the electrostatic sensor 120 represent, for example, the XY coordinates of the position with the largest capacitance within the area where the fingertip FT is present. The center of gravity of a figure consisting of multiple points whose difference from the largest capacitance is less than a threshold value may be regarded as the XY coordinates of the fingertip FT. Alternatively, the X (or Y) coordinates of the fingertip FT may be regarded as the X (or Y) coordinates of the vertex obtained by fitting the largest capacitance position and the capacitances of the three points adjacent to it in the X (or Y) direction to a quadratic curve. Furthermore, capacitance is inversely proportional to distance. Since capacitance and distance correspond one-to-one, capacitance can be used instead of distance. Furthermore, in the non-contact input device 100 of the present invention, capacitance does not need to be expressed in units of farads [F]. In the present invention, a value proportional to capacitance is referred to as a Z value. The input sensor circuit 125A is designed to have a Z value of approximately 100 when the fingertip FT is in contact with the operation surface 105A, and a Z value of approximately 0 when the fingertip FT is not near the operation surface 105A.
[0068] In the proximity mode or the standby mode, the input assistance application 150 may stop calculating the moving average of the X and Y coordinates or the amount of fluctuation of the X and Y coordinates. In the standby mode, the power to the display may be turned off.
[0069] <Calculation of Index of Positional Accuracy of Object by Input Assistance Application 150> The input assistance application 150 calculates the standard deviation of the X and Y coordinates as an index of the positional accuracy of the object in order to indicate the magnitude of error due to the amount of fluctuation in the X and Y coordinates measured by the non-contact input device 100. A method for calculating the standard deviation of the X and Y coordinates will be described later with reference to FIG.
[0070] <Display Examples of the Non-Contact Input Device 100> Figures 6A to 6F are diagrams showing examples of the display on the display 110 of the non-contact input device 100. Figures 6A and 6B show the display 110 and the operation surface 105A superimposed on each other. The display on the display 110 is visible to the user through the operation surface 105A. Figures 6C to 6F show a portion of the image displayed on the display 110.
[0071] 6A shows the display 110 in standby mode. When the user's fingertip FT is not near the operation surface 105A, messages such as "Touchless operation is possible" and "Please place your finger on the screen" are displayed, as shown in FIG. 6A. This allows the user to guide the fingertip FT to the operation surface 105A even when using the non-contact input device 100 for the first time, leading to smooth use thereafter.
[0072] 6B shows an example of a message in proximity mode. In FIG. 6B, messages such as "The pointer will shrink depending on the position accuracy" and "If the pointer is too large, move your finger closer or raise your finger" are displayed. When the user brings the fingertip FT closer to the operation surface 105A, a second pointer 112 is displayed on the display 110. Note that the pointer is a graphic or symbol that indicates the current input position on the screen (image) displayed on the display 110.
[0073] The second pointer 112 is a graphic that the input assistance application 150 causes the OS 140 to display. The input assistance application 150 hides the first pointer (mouse pointer) 111 provided by the OS. The second pointer 112 may be a circular or elliptical graphic, in addition to the annular graphic shown in FIG. 6B . The input assistance application 150 increases the radius of the second pointer 112 as the amount of variation in the X and Y coordinates, which are indicators of positional accuracy, increases, and decreases the radius of the second pointer 112 as the amount of variation in the X and Y coordinates decreases. In other words, the input assistance application 150 causes the display 110 via the OS 140 to display a second pointer 112 with a larger radius as the amount of variation in the X and Y coordinates increases, and causes the display 110 via the OS 140 to display a second pointer 112 with a smaller radius as the amount of variation in the X and Y coordinates decreases.
[0074] 6C , a more specific operation when the user brings the fingertip FT close to the operation surface 105A to select the GUI button for the number 6 in the selection mode will be described. In FIG. 6C , the first pointer 111 is hidden, and only the second pointer 112 is shown. In the selection mode, the second pointer 112 is displayed in a circular shape.
[0075] 6C, the input assistance application 150 causes the OS 140 to display the second pointer 112 with a larger radius as the amount of variation in the X and Y coordinates increases. For example, when the fingertip FT is farther from the operation surface 105A, the amount of variation in the X and Y coordinates increases.
[0076] 6C , the input assistance application 150 causes the OS 140 to display a second pointer 112 with a smaller radius as the amount of variation in the X and Y coordinates decreases. For example, when the fingertip FT is closer to the operation surface 105A, the amount of variation in the X and Y coordinates decreases.
[0077] That is, when operating the GUI button for the number 6, if the fingertip FT is far from the operation surface 105A, the second pointer 112 with a large radius is displayed as shown on the left side of Fig. 6C . Then, if the fingertip FT is brought closer to the operation surface 105A, the radius of the second pointer 112 becomes smaller, and the second pointer 112 with a small radius is displayed as shown on the right side of Fig. 6C . In this way, if the fingertip FT is far from the operation surface 105A and the accuracy of the XY coordinates is low, the radius of the second pointer 112 is set to be large, and if the fingertip FT is close to the operation surface 105A and the accuracy of the XY coordinates is high, the radius of the second pointer 112 is set to be small. In this way, the non-contact input device 100 can intuitively provide feedback of position accuracy to the user.
[0078] FIG. 6D illustrates an operation performed in the confirmation mode when a user holds a fingertip FT close to the operation surface 105A to confirm an operation input for the GUI button number 6. The second pointer 112A has a different display color from the second pointer 112 shown in FIG. 6C . For this reason, the second pointer 112A is indicated by a double line in FIG. 6C . The input assistance application 150 intuitively indicates the change in input mode by changing the circular shape of the second pointer 112 to an arc shape of the second pointer 112A. Furthermore, the input assistance application 150 can emphasize the change in input mode by changing the color of the second pointer 112 from red (112) to yellow (112A). Note that the color of the second pointer is not limited to changing from red to yellow. The color of the second pointer (112, 112A) may be changeable in a setting screen described later. Instead of changing the color, the brightness may be changed, or both the color and the brightness may be changed.
[0079] FIG. 6D shows how the images change over time, starting from the top left and moving along the arrows to the top right, bottom left, and bottom right.
[0080] In the confirmation mode, an arc-shaped second pointer 112A is first displayed as shown in the upper left. As the fingertip FT is held close to the operation surface 105A, the arc extends as shown in the upper right, and then the arc of the second pointer 112A further extends as shown in the lower left. When a first predetermined time T1 has elapsed since the fingertip FT was brought to rest on the operation surface 105A, the second pointer 112A becomes a ring. When the second pointer 112A becomes a ring, the operation content is confirmed. In other words, the operation content is confirmed by continuing to point the fingertip FT at the number 6 until the second pointer 112A becomes a ring. In this way, the non-contact input device 100 allows the user to perform a non-contact click operation instead of an operation by touching the operation surface 105A.
[0081] When the operation content is determined, the input assistance application 150 outputs a click event. The input assistance application 150 issues the same event as when a mouse is clicked, by inputting an operation with the fingertip FT without contact. The input assistance application 150 outputs the X and Y coordinates of the fingertip FT to the OS 140, and also outputs a click event. The input assistance application 150 may also cause the OS 140 to play a click event output sound. The output of the X and Y coordinates without contact and the output of the click event correspond to the operation of touching the GUI button 110A on the operation surface 105A.
[0082] When the input assistance application 150 outputs a click event, it displays a circular second pointer 112B with a smaller radius than the annulus for a predetermined time (for example, about 1 to 2 seconds) as shown in the lower right of FIG. 6D.
[0083] The circular second pointer 112B is displayed for user confirmation at the XY coordinates (moving average) of the fingertip FT when the operation input is confirmed. For example, the second pointer 112B is displayed for a predetermined time (approximately 1 to 2 seconds) and then disappears after the predetermined time has elapsed. The second pointer 112B may have a different display color from the second pointers 112 and 112A. The input assistance application 150 intuitively indicates that the operation input has been confirmed by changing the second pointer 112B to a small circle immediately after the central angle of the second pointer 112A (arc) reaches 360 degrees. Furthermore, changing the color can emphasize that the operation input has been confirmed. Instead of changing the color, the brightness may be changed, or both the color and brightness may be changed.
[0084] 6C and 6D show an example in which the first pointer 111 of the OS 140 is erased, but as shown in Fig. 6E, both the first pointer 111 (mouse pointer) of the OS 140 and the second pointer 112 (pointer for a non-contact input device) of the input assistance application 150 may be displayed. When both the first pointer 111 and the second pointer 112 are displayed, the input assistance application 150 continues to output X and Y coordinates to the OS 140 in a selection mode and a decision mode, which will be described later. The first pointer 111 functions the same as a reticle that indicates the position of the X and Y coordinates with high precision.
[0085] <Display of Function Menu> Fig. 6F shows a function menu 112C. In the selection mode, when the fingertip FT is brought close to the operation surface 105A and the radius of the second pointer 112 is reduced as shown on the right side of Fig. 6C, and the XY coordinates of the fingertip FT remain stationary for a predetermined time, the input assistance application 150 causes the OS 140 to display the function menu 112C. This predetermined time is an example of a third predetermined time. Fig. 6E shows, as an example, the function menu 112C from which click mode, drag mode, and pinch-in can be selected.
[0086] 7A to 7C, 8A to 8F, and 9A to 9F are diagrams showing an example of processing executed by the input assistance application 150 of the non-contact input device 100. Figures 7A to 7C show an example of main processing, Figure 7D shows an example of a modified example of the processing shown in Figure 7C, and Figures 8A to 8F and 9A to 9F show an example of subroutine processing.
[0087] When the input assistance application 150 starts processing (see FIG. 7A), it sets initial values (step S1). Step S1 is a subroutine, the details of which will be described later with reference to FIG. 8A, in which initial values such as the number of data Nma for calculating the moving average are set.
[0088] The input assistance application 150 outputs a command to erase the first pointer 111 to the OS 140 (step S2). As a result, the first pointer 111 is erased from the display 110. Note that the first pointer 111 (mouse pointer) may be displayed without performing the processing of step S2 (see FIG. 6E). In this case, both the first pointer 111 and the second pointer 112 are displayed. In this case, the input assistance application 150 continues to output the XY coordinates of the fingertip FT to the OS. Because the position of the first pointer 111 and the center of the second pointer 112 coincide, the center of the second pointer 112 is easy to find even if the circle it represents is large.
[0089] The input assistance application 150 calculates the X and Y coordinates of the fingertip FT (step S3). Step S3 is a subroutine, the details of which will be described later, and calculates a moving average of the X and Y coordinates of the fingertip FT using the X and Y coordinates of the fingertip FT calculated at multiple points in time.
[0090] The input assistance application 150 determines whether the mode of the input assistance application 150 is the decision mode (step S4). For ease of understanding, the case where the mode of the input assistance application 150 is the decision mode will be described first.
[0091] If the input assistance application 150 determines that the mode of the input assistance application 150 is the decision mode (S4: YES), it determines whether the Z value is greater than Cz1 (step S5). If the Z value is greater than Cz1, the input assistance application 150 considers that the distance from the operation surface 105A to the fingertip FT is shorter than the first threshold value Z1. The input assistance application 150 determines from the Z value whether the fingertip FT is located within the decision area.
[0092] If the input assistance application 150 determines that the Z value is greater than Cz1 (S5: YES), it sets SelectionOffTime and ProximityOffTime to the current time (step S6). SelectionOffTime is updated to the time when it was determined that the Z value was greater than Cz1. Therefore, SelectionOffTime indicates the last time when it was determined that the Z value was greater than Cz1. Furthermore, as will be described later, ProximityOffTime is also updated when it is determined that the Z value is greater than Cz2. Therefore, ProximityOffTime indicates the last time when it was determined that the Z value was greater than Cz2.
[0093] The input assistance application 150 determines whether DecisionTH is 0 (S7). DecisionTH is a value that can be set by the user. If DecisionTH is 0, the input is confirmed immediately upon entering decision mode without displaying the arc cursor described below. On the other hand, if DecisionTH is other than 0, the input is confirmed after displaying the arc cursor described below.
[0094] If the input assistance application 150 determines that DecisionTH is not 0 (S7: No), it outputs to the OS 140 a command to display the arc-shaped second pointer 112A at the XY coordinates (moving average: Xave, Yave) calculated in step S3 (step S8). The command output to the OS 140 in step S8 is a command to display an arc with a radius of a predetermined value (fixed value) centered on the XY coordinates (moving average). The arc has a moving end that extends clockwise from a fixed end, the fixed end is located at a position corresponding to 12 o'clock on a clock, and the moving end extends clockwise from 12 o'clock. The input assistance application 150 calculates the angle of the arc between the fixed end and the moving end, for example, using the following formula (1): Arc angle = 360 degrees × (current time - DecisionTime) / DecisionTH (1)
[0095] Here, DecisionTime is the time when drawing of the arc begins, and DecisionTH is the time required from when drawing of the arc begins until a click event is output. A click event is an event that occurs when the operation content is decided by an operation input with the fingertip FT. The operation content is decided when the second pointer 112A changes from an arc to a ring. Once the operation content is decided, the input assistance application 150 outputs a click event. The user can decide the operation content by bringing the fingertip FT close to the operation surface 105A within the decision area and holding the fingertip FT until the second pointer 112A extends and becomes a ring.
[0096] The input assistance application 150 determines whether the relationship (current time - DecisionTime > DecisionTH) holds (step S9). That is, it determines whether the time elapsed since the time (DecisionTime) when the mode transitioned to the decision mode has exceeded DecisionTH. This is to determine whether the operation content has been decided, and whether to output a click event when the operation content has been decided. The time elapsed since the time (DecisionTime) when the mode transitioned to the decision mode is an example of a first predetermined time.
[0097] If the input assistance application 150 determines that the current time - DecisionTime > DecisionTH is not true (S9: NO), the flow returns to step S3. The processing of steps S3 to S8 is repeated, thereby extending the arc.
[0098] If the input assistance application 150 determines that the relationship Current Time - DecisionTime > DecisionTH is true (S9: YES), it proceeds with the process to output a click event. When the relationship Current Time - DecisionTime > DecisionTH is true, the arc cursor becomes a ring.
[0099] The input assistance application 150 erases the second pointer 112A (step S10).
[0100] The input assistance application 150 outputs the X and Y coordinates (moving averages) calculated in step S3 to the OS 140, and then outputs a click event to the OS 140. Furthermore, the input assistance application 150 causes the OS 140 to play a click event output sound (step S11). As a result, the click event output sound is output from the speaker. Furthermore, if a click event is issued when the X and Y coordinates overlap with the GUI button 110A of the application program 160, the application program 160 executes the processing indicated by the GUI button 110A.
[0101] If the input assistance application 150 determines that DecisionTH is 0 (S7: Yes), it executes the process of step S11 described above. That is, if the input assistance application 150 determines that DecisionTH is 0 (S7: Yes) or if it determines that the current time - DecisionTime > DecisionTH is true (S9: YES), it outputs the XY coordinates and the click event to the OS 140 and causes the OS 140 to play the click event output sound.
[0102] The input assistance application 150 causes the OS 140 to display the second pointer 112B centered on the XY coordinates (moving average) calculated in step S3 for a predetermined time (step S12).
[0103] The center of the second pointer 112B is the XY coordinate (moving average) calculated in step S3. The predetermined time is, for example, about 1 to 2 seconds, and when the predetermined time has elapsed, the second pointer 112B is hidden.
[0104] The input assistance application 150 determines whether ReClickTH=0 is true (step S13). ReClickTH is the time (click re-output time) until the second pointer 112A starts to be displayed again if the fingertip FT continues to be positioned within the decision area after a click event is output. The user of the non-contact input device 100 can set ReClickTH for the input assistance application 150 via the setting screen displayed on the display 110.
[0105] If the input assistance application 150 determines that ReClickTH=0 is true (S13: YES), it calculates the X and Y coordinates of the fingertip FT (step S14). Step S14 is a subroutine similar to step S3, and the details of which will be described later. The X and Y coordinates of the fingertip FT calculated at multiple points in time are used to calculate a moving average of the X and Y coordinates of the fingertip FT.
[0106] The input assistance application 150 determines whether the Z value is greater than Cz1 (step S15). After the operation content is determined, the input assistance application 150 determines whether the fingertip FT continues to be located within the determination area.
[0107] If the input assistance application 150 determines that the Z value is not greater than Cz1 (S15: NO), it sets the mode to selection mode (step S16). Then, the flow returns to step S3. If the Z value ΔAD is not greater than Cz1, this is the case when the fingertip FT has moved from the decision area to the selection area, the proximity area, or the like. If the fingertip FT has moved from the decision area to the selection area, the proximity area, or the like, the processing of step S14 is provided to enable re-input.
[0108] If the input assistance application 150 determines in step S15 that the Z value is greater than Cz1 (S15: YES), it causes the OS 140 to display the second pointer 112B at the XY coordinate (moving average) calculated in step S14 (step S17). The size of the circular second pointer 112B is a predetermined value.
[0109] If the click re-output time (ReClickTH) is set to 0, the input assistance application 150 will not re-output a click event unless the mode is changed from the decision mode to another mode, and will continue to display the second pointer 112B at the XY coordinate position of the fingertip FT.
[0110] When the input assistance application 150 finishes processing in step S17, it returns the flow to S14. If the operation content is decided as a result of YES in step S7 or YES in step S9, and the fingertip FT continues to be within the decision area after the second pointer 112B is displayed in step S12, the input assistance application 150 displays the second pointer 112B in step S17 and then returns the flow to step S14. That is, the second pointer 112B continues to be displayed while updating its position. After the operation content has been decided once, in order to decide the next operation content, it is sufficient to move the fingertip FT out of the decision area once.
[0111] <S5: NO> Furthermore, if the input assistance application 150 determines in step S5 that the Z value is not greater than Cz1 (S5: NO), it determines whether or not the relationship "current time - SelectionOffTime > SelectionOffTH" holds (step S18). That is, when the fingertip FT has been continuously positioned outside the decision area, it determines whether or not the elapsed time since the time (SelectionOffTime) when the fingertip FT was last determined to be positioned within the decision area has exceeded SelectionOffTH. This elapsed time is an example of a fourth predetermined time.
[0112] SelectionOffTH is the time it takes to transition from decision mode to selection mode when the fingertip FT is moved into the selection area. The user of the non-contact input device 100 can set SelectionOffTH for the input assistance application 150 through a setting screen displayed on the display 110. The time when the fingertip FT was last determined to be located within the decision area is the most recent time when the fingertip FT was determined to be located within the decision area.
[0113] If the input assistance application 150 determines that the current time - SelectionOffTime > SelectionOffTH is not true (S18: NO), it determines whether the Z value is greater than Cz2 (step S19). If the Z value is greater than Cz2, the input assistance application 150 determines that the fingertip FT is staying in the first area. If Z is greater than Cz2 (S19: YES), the flow returns to step S7. As a result, operation in the decision mode continues. The Z value does not use a moving average, and a measured value may be smaller than the actual value due to noise. In other words, even if the fingertip FT is located in the decision area, Z > Cz1 may not be true (S5: NO). Therefore, even if the Z value is not greater than Cz1, if the Z value is greater than Cz2, the input assistance application 150 continues drawing the second pointer 112A for a while (during SelectionOffTH). On the other hand, if the Z value is not greater than Cz2 (S19: NO), the input assistance application 150 sets the mode to selection mode and sets SelectionTime to the current time (step S20). If the Z value is not greater than Cz2, the fingertip FT is considered not to be in the decision area.
[0114] Furthermore, if the input assistance application 150 determines that the current time - SelectionOffTime > SelectionOffTH is true (S18: YES), it sets the mode of the input assistance application 150 to the selection mode and sets SelectionTime to the current time (step S20). If the state in which the Z value is not greater than Cz1 (step S5) continues longer than SelectionOffTH, it is determined that the fingertip FT is not in the decision area.
[0115] The input assistance application 150 causes the OS 140 to hide the second pointer 112A (step S21).
[0116] The input assistance application 150 displays the second pointer 112 on the OS 140 (step S22). The second pointer 112 is the second pointer for the selection mode shown in FIG. 6C . Before displaying the second pointer 112 on the OS 140, the input assistance application 150 hides the second pointer 112A. The center of the second pointer 112 is the XY coordinates (moving average) of the fingertip FT, and the radius of the second pointer 112 corresponds to the amount of fluctuation in the XY coordinates. In step S22, when displaying the second pointer 112 on the OS 140, the input assistance application 150 outputs the position of the second pointer 112 corresponding to the XY coordinates (moving average) of the fingertip FT to the OS 140. Furthermore, the input assistance application 150 adjusts the size of the second pointer 112 to correspond to the amount of fluctuation in the XY coordinates.
[0117] The input assistance application 150 executes a process for determining whether or not a function menu needs to be displayed (step S23). The process for determining whether or not a function menu needs to be displayed is a subroutine process, and will be described in detail later. After completing the process of step S23, the input assistance application 150 returns the flow to step S3.
[0118] <S12: NO> If the input assistance application 150 determines in step S13 that ReClickTH=0 is not true (S13: NO), it sets ClickTime to the current time (step S24). ClickTime is the time when a click event is output.
[0119] The input assistance application 150 determines whether the current time - ClickTime > ReClickTH is satisfied (step S25).
[0120] If the current time - ClickTime > ReClickTH is satisfied (S25: YES), the input assistance application 150 sets DecisionTime to the current time (step S26). Then, the flow returns to step S3. Therefore, DecisionTime becomes the time at the time of returning to step S3. When the fingertip FT remains in the decision area, a gradually longer arc is drawn again. The time from when a click event is output until the drawing of the second pointer 112A (arc) begins again is the sum of the "predetermined time" in step S12 and ReClickTH. It takes DecisionTimeTH from when the drawing of the second pointer 112A (arc) begins until the second pointer 112A becomes a ring. A click event is output every time the "predetermined time" in step S12, ReClickTH, and DecisionTimeTH have elapsed. That is, the second predetermined time is the sum of the "predetermined time" in step S12, ReClickTH, and DecisionTimeTH.
[0121] If the input assistance application 150 determines in step S25 that the current time - ClickTime > ReClickTH is not true (S25: NO), it calculates the X and Y coordinates of the fingertip FT (step S27). Step S27 is a subroutine similar to step S3, and the details of which will be described later. The X and Y coordinates of the fingertip FT calculated at multiple points in time are used to calculate a moving average of the X and Y coordinates of the fingertip FT.
[0122] The input assistance application 150 determines whether the Z value is greater than Cz1 (step S28) in order to determine whether the fingertip FT is located within the decision area.
[0123] If the input support application 150 determines that the Z value is greater than Cz1 (S28: YES), it displays the second pointer 112B at a position centered on the XY coordinates of the fingertip FT (step S29). If the input support application 150 remains in the determination mode, it causes the OS 140 to display the second pointer 112B until the click re-output time (ReClickTH) has elapsed.
[0124] If the input assistance application 150 determines in step S27 that the Z value is not greater than Cz1 (S28: NO), it sets the mode of the input assistance application 150 to the selection mode and sets the current time in SelectionTime (step S30). When the fingertip FT moves from the decision area to the selection area, the mode immediately transitions to the selection mode. A user who wants to perform the next operation input as quickly as possible can switch to the selection mode by slightly removing the fingertip FT from the operation surface 105A. When the input assistance application 150 completes the processing of step S30, it returns the flow to step S3.
[0125] <S4: NO> If the input assistance application 150 determines in step S4 that the mode is not the decision mode (S4: NO), it determines whether the Z value is greater than Cz1 (step S4A). That is, the input assistance application 150 determines whether the fingertip FT is located within the decision area. If the Z value is greater than Cz1, the fingertip FT is located within the decision area.
[0126] If the input assistance application 150 determines that the Z value is greater than Cz1 (S4A: YES), it sets SelectionOffTime and ProximityOffTime to the current time (step S5A). SelectionOffTime is updated to the time when it is determined that the Z value is greater than Cz1, so SelectionOffTime indicates the last time when it was determined that the Z value was greater than Cz1. As will be described later, ProximityOffTime is also updated when it is determined that the Z value is greater than Cz2. Therefore, ProximityOffTime indicates the last time when it was determined that the Z value was greater than Cz2.
[0127] The input assistance application 150 sets DecisionTime to the current time (step S6A). When the time during which the decision mode has continued reaches DecisionTH, the input assistance application 150 outputs a click event. Since it was determined in step S4 that the decision mode was not active and then in step S4A that the Z value was greater than Cz1, the time at which step S6A is executed is the time at which the decision mode was activated. In other words, DecisionTime set in step S6A is the time at which the decision mode was activated.
[0128] The input assistance application 150 sets the mode of the input assistance application 150 to the decision mode (step S7A). Furthermore, the input assistance application 150 erases the second pointer 112 for the selection mode (step S8A). After the processing of step S8A is completed, the flow proceeds to step S7.
[0129] The flow proceeds from step S4: NO to step S4A: YES when the fingertip FT enters the decision area from outside the decision area. In this way, when the fingertip FT approaches the operation surface 105A, the mode immediately transitions to decision mode. On the other hand, when the fingertip FT moves away from the operation surface 105A, the mode is not immediately changed. As described above, when the Z value becomes small (when the state in which the fingertip FT moves away from the operation surface 105A is measured), the mode is not immediately changed, thereby stabilizing operation. On the other hand, without using a moving average for the Z value, when the fingertip FT approaches the operation surface 105A, the mode immediately transitions to decision mode, thereby achieving quick operation. Note that when the decision mode is entered due to noise, the fingertip FT is not in the decision area but is near the decision area (selection area). This prevents the user from noticing a malfunction. This prevents both a decrease in response speed and erroneous input.
[0130] <S4A: NO> If the input assistance application 150 determines in step S4A that the Z value is not greater than Cz1 (S4A: NO), it determines whether the input assistance application 150 is in the selection mode (step S4B).
[0131] In the selection mode, the input assistance application 150 determines whether the Z value is greater than Cz2 (step S5B). That is, the input assistance application 150 determines whether the fingertip FT is located within the selection area.
[0132] If the input assistance application 150 determines that the Z value is greater than Cz2 (S5B: YES), it sets ProximityOffTime to the current time (step S6B). ProximityOffTime is the time when the fingertip FT was last determined to be located within the selection area when the fingertip FT has been continuously located within the selection area. When the fingertip FT is located within the selection area, ProximityOffTime is updated, so that ProximityOffTime becomes the most recent time when the fingertip FT was determined to be located within the selection area.
[0133] The input assistance application 150 executes a second pointer radius setting process to set the radius of the second pointer 112 according to the amount of variation in the X and Y coordinates (step S7B). The second pointer radius setting process is a subroutine process and will be described in detail later, but in the selection mode, the radius of the second pointer 112 (see FIG. 6C ) is set according to the amount of variation in the X and Y coordinates. After completing the process of step S7B, the input assistance application 150 advances the flow to step S21 and causes the OS 140 to display the second pointer 112.
[0134] <S5B: NO> If the input assistance application 150 determines in step S5B that the Z value is not greater than Cz2 (S5B: NO), it determines whether the relationship Current Time - ProximityOffTime > ProximityOffTH holds (step S8B). That is, when the fingertip FT is continuously positioned within the selection area, it determines whether the elapsed time from the time (ProximityOffTime) when the fingertip FT was last determined to be positioned within the selection area exceeds ProximityOffTH. ProximityOffTH is the time required to transition from selection mode to proximity mode. The user of the non-contact input device 100 can set ProximityOffTH for the input assistance application 150 via a setting screen displayed on the display 110.
[0135] If the input assistance application 150 determines that the relationship Current Time - ProximityOffTime > ProximityOffTH does not hold (S8B: NO), it executes a second pointer radius setting process to set the radius of the second pointer 112 according to the amount of fluctuation in the X and Y coordinates (step S9B). The second pointer radius setting process is a subroutine process, and details will be described later. In selection mode, the radius of the second pointer 112 (see FIG. 6C ) is set according to the amount of fluctuation in the X and Y coordinates. After completing step S9B, the input assistance application 150 advances the flow to step S21 and displays the second pointer 112 on the OS 140. The Z value does not use a moving average, and noise may result in a measured value that is lower than the actual value. However, a state in which Z > Cz2 does not hold even when the actual position of the fingertip FT is within the selection area is resolved in a time shorter than ProximityOffTH. Therefore, even if a small Z value is measured due to noise while the second pointer 112 is displayed, the second pointer 112 continues to be displayed. On the other hand, even if the fingertip FT is actually moved into the proximity area, the second pointer 112 continues to be displayed for a while. Most users do not perceive a slow response even if the time it takes for the second pointer 112 to disappear from the display is delayed. Therefore, the processing from S8B: NO to S9B can stabilize the operation without reducing the perceived response speed.
[0136] If the input assistance application 150 determines in step S8B that the current time - ProximityOffTime > ProximityOffTH is true (S8B: YES), the input assistance application 150 sets the mode of the input assistance application 150 to the proximity mode (step S10B).
[0137] The input assistance application 150 causes the OS 140 to display an image and a message for the proximity mode (see FIG. 6B ) (step S11B). As a result, as shown in FIG. 6B , the image and the message for the proximity mode are displayed on the display 110. After completing the processing of step S11B, the input assistance application 150 advances the flow to step S3.
[0138] <S4B: NO> If the input support application 150 determines in step S4B that the mode of the input support application 150 is not the selection mode (S4B: NO), the input support application 150 determines whether the Z value is greater than Cz2 (step S4C). That is, the input support application 150 determines whether the fingertip FT has entered the selection area again.
[0139] If the input assistance application 150 determines that the Z value is greater than Cz2 (S4C: YES), it sets ProximityOffTime to the current time (step S5C). As described above, if Z > Cz1 (S5 or S4A) is YES or Z > Cz2 (S5B or S4C) is YES, ProximityOffTime is updated to the current time. Therefore, the last time when Z was greater than Cz2 is stored as ProximityOffTime.
[0140] The input assistance application 150 sets the mode of the input assistance application 150 to the selection mode and sets the current time in SelectionTime (step S6C).
[0141] The input assistance application 150 performs a process to set the radius of the second pointer 112 to an initial value (step S7C). The process of setting the radius of the second pointer 112 to an initial value is a subroutine process, and will be described in detail later. However, this process sets the radius of the second pointer 112 to an initial value, rather than a radius corresponding to the amount of fluctuation in the X and Y coordinates. After completing the process of step S7C, the input assistance application 150 advances the flow to step S21 and displays the second pointer 112 on the OS 140. Note that even if the fingertip FT is actually in the proximity area, noise may cause the measured Z value to be larger than the actual value, resulting in the selection mode. However, since the selection mode is entered when the fingertip FT is relatively close, most users will not recognize this as a malfunction. Conversely, if the fingertip FT is in a distant position (standby area), the selection mode will not be entered even if a normally present level of noise is present.
[0142] <S4C: NO> If the input assistance application 150 determines in step S4C that the Z value is not greater than Cz2 (S4C: NO), it determines whether the mode of the input assistance application 150 is the proximity mode (step S4D).
[0143] If the input assistance application 150 determines that the mode of the input assistance application 150 is the proximity mode (S4D: YES), the input assistance application 150 determines whether the Z value is greater than Cz3 (step S5D). If the Z value is greater than Cz2, the input assistance application 150 considers that the distance from the operation surface 105A to the fingertip FT is shorter than the third threshold value Z3.
[0144] If the input assistance application 150 determines that the Z value is greater than Cz3 (S5D: YES), the flow proceeds to step S11B. As a result, in step S11B, an image and message for the proximity mode are displayed on the display 110, as shown in FIG. 6B.
[0145] <S5D: NO> If the input support application 150 determines in step S5D that the Z value is not greater than Cz3 (S5D: NO), it sets the mode of the input support application 150 to the standby mode (step S6D). If the Z value is not greater than Cz3, the input support application 150 considers that the distance from the operation surface 105A to the fingertip FT is longer than the third threshold value Z3.
[0146] The input assistance application 150 causes the OS 140 to display an image and message for standby mode (step S7D). As a result, the image and message for standby mode are displayed on the display 110. After completing the processing of step S7D, the input assistance application 150 advances the flow to step S3. Note that in standby mode, the display may be powered off (S7D' in FIG. 7D).
[0147] <S4D: NO> If the input assistance application 150 determines in step S4D that the mode of the input assistance application 150 is not the proximity mode (S4D: NO), the input assistance application 150 determines whether the Z value is greater than Cz3 (step S4E). The input assistance application 150 determines whether the fingertip FT has entered the proximity region again.
[0148] If the input assistance application 150 determines that the Z value is greater than Cz3 (S4E: YES), it sets the mode of the input assistance application 150 to the proximity mode (step S5E). The input assistance application 150 proceeds to step S11B and displays an image and message for the proximity mode.
[0149] If the input assistance application 150 determines in step S4E that the Z value is not greater than Cz3 (S4E: NO), the flow proceeds to step S7D. In this case, the input assistance application 150 is in standby mode, and therefore, in step S7D, an image and a message for standby mode are displayed on the display 110.
[0150] This completes the main flow. The input assistance application 150 repeatedly executes the main flow. Next, each subroutine will be described.
[0151] <FIG. 8A: Initial Value Setting> The input assistance application 150 sets the mode of the input assistance application 150 to the standby mode, and sets the flag null to YES (step S101).
[0152] The input assistance application 150 sets Nma to a predetermined integer (step S102). Nma is a calculated value of the number of multiple data items used when calculating the moving average, and is calculated as Nma = int(MATime / Nmea). The predetermined integer is obtained as int(MATime / Nmea). int(MATime / Nmea) represents an integer obtained by rounding down the decimal point of MATime / Nmea. MATime is the time used for the moving average (the time from the start to the end), and Nmea is the measurement interval. That is, Nma = int(MATime / Nmea). The user of the non-contact input device 100 can set MATime for the input assistance application 150 via a setting screen displayed on the display 110.
[0153] The input assistance application 150 compares Nma and Ndev to determine which is larger (step S103), where Ndev is the number of multiple data items used when calculating the index of position accuracy.
[0154] The input assistance application 150 sets the larger of Nma and Ndev as the number of arrays Num (steps S104 and S105). That is, the larger of the "number of data required to calculate the moving average" and the "number of data required to calculate the index of position accuracy" is set as the number of arrays Num. Since the number of elements in the array is determined, the array may be declared at this stage. After completing the processing of step S104, the input assistance application 150 ends the subroutine for setting initial values.
[0155] <FIG. 8B: Coordinate Calculation> The input assistance application 150 acquires the XY coordinates and Z value from the input sensor circuit 125A (step S111).
[0156] The input assistance application 150 acquires the current time from the OS 140 (step S112). The value acquired in step S112 is used as the "current time" used in the main loop described above.
[0157] The input assistance application 150 updates the array of X coordinates (step S113). Since step S113 is a subroutine process, details will be described later, but in step S113, the input assistance application 150 removes the oldest value from the array used to calculate the moving average of the X coordinates and the index of positional accuracy, and adds the latest value.
[0158] The input assistance application 150 updates the array of Y coordinates (step S114). Step S113 is a subroutine process and will be described in detail later, but in step S114, the input assistance application 150 removes the oldest value from the array used to calculate the moving average of the Y coordinates and the index of positional accuracy, and adds the latest value.
[0159] The input assistance application 150 calculates a moving average from the array of X coordinates updated in step S113 (step S115).
[0160] The input assistance application 150 calculates the moving average of the array of the Y coordinates updated in step S114 (step S116).
[0161] The input assistance application 150 determines whether the Z value acquired in step S111 is greater than Cz3 (step S117). If the Z value is not greater than Cz3, the Z value is within the waiting area, and the reliability of the latest XY coordinates acquired in step S111 and the XY coordinates acquired in steps S111 before that is low, and these are not coordinate values suitable for calculating the moving average of the XY coordinates, so the processing of step S117 is provided.
[0162] If the input assistance application 150 determines that the Z value acquired in step S111 is greater than Cz3 (S117: YES), it sets the flag null to NO (step S118). Because the latest X and Y coordinates are coordinate values suitable for calculating the moving average of the X and Y coordinates and the index of positional accuracy, the input assistance application 150 sets the flag null to NO.
[0163] If the input assistance application 150 determines that the Z value acquired in step S111 is not greater than Cz3 (S117: NO), it sets the flag "null" to "YES" (step S119). Since the latest X and Y coordinates are not coordinate values suitable for calculating the moving average of the X and Y coordinates, it sets the flag "null" to "YES."
[0164] This completes the coordinate calculation subroutine.
[0165] <FIG. 8C: Updating the X-Coordinate Array> FIG. 8C shows details of the subroutine process for updating the X-coordinate array in step S113 of FIG. 8B. Hereinafter, multiple X-coordinates acquired at multiple points in time will be distinguished as X(i). The multiple X-coordinates X(i) were acquired at different times and are multiple X-coordinates acquired continuously by the non-contact input device 100 at a predetermined sampling period. i=1 to Num, and X(1) is the most recent X-coordinate.
[0166] The input assistance application 150 determines whether the flag null is YES (step S121X) in order to determine whether the latest X coordinate is appropriate.
[0167] If the input assistance application 150 determines that the flag null is YES (S121X: YES), it executes a subroutine process of substituting the non-measured value into X(i) to create the X coordinate.
[0168] The input assistance application 150 selects Num-1 X coordinates (X(2) to X(Num)) one by one, where i is from 2 to Num, and assigns a non-measured value to each of them (step S122X). A non-measured value is a value that is not obtained by measurement, and is a dummy value. Here, 0xFFFF is used as an example of a non-measured value. After completing the subroutine processing including step S122X, the input assistance application 150 advances the flow to step S123X. Note that the non-measured value may be a value other than 0xFFFF, as long as it is a value that is not measured.
[0169] The input assistance application 150 assigns the latest X coordinate to X(1) (step S123X). In this way, the input assistance application 150 updates Num X coordinates X(1) to X(Num).
[0170] Furthermore, if the input assistance application 150 determines that the flag null is NO (S121X: NO), it selects Num-1 X coordinates (X(1) to X(Num-1)) where i is 1 to Num-1 one by one, increments the i number by 1, and moves to the X coordinate (X(2) to X(Num)) (step S124X).
[0171] After completing step S124X, the input assistance application 150 advances the flow to step S123X. The input assistance application 150 updates the array of X coordinates X(1) to X(Num) by adding the X coordinate X(1) to the X coordinates (X(2) to X(Num)) acquired in step S124X.
[0172] <FIG. 8D: Acquisition of Latest Num Y Coordinates> FIG. 8D shows details of the subroutine process for updating the Y coordinate array in step S114 of FIG. 8B. Hereinafter, multiple Y coordinates will be distinguished as Y(i). The multiple Y coordinates Y(i) were acquired at different times, and are multiple Y coordinates acquired consecutively by the non-contact input device 100 at a predetermined sampling period. The times at which multiple Y coordinates Y(i) were acquired are equal to the times at which multiple X coordinates X(i) were acquired. i = 1 to Num, and Y(1) is the latest Y coordinate.
[0173] The processing of steps S121Y to S124Y shown in FIG. 8D is processing executed by the input assistance application 150 for the Y coordinate Y(i), and is processing in which the processing of steps S121X to S124X for the X coordinate X(i) shown in FIG. 8C has been replaced with processing for the Y coordinate Y(i). Therefore, the description of FIG. 8D will be omitted here. Note that instead of the processing of steps S121X to S123X and S121Y to S123Y, processing may be performed to replace the oldest value in the array with the latest value. In this case, there is no need to copy each element of the array. However, a subscript (index) identifying the element storing the oldest value and a subscript (index) identifying the element storing the latest value are required. Furthermore, the loop processing for calculating the moving average and the amount of fluctuation, which will be described later, also needs to be modified.
[0174] <FIG. 8E: Moving Average of X Coordinate> FIG. 8E shows details of the subroutine process for calculating the moving average of the X coordinate in step S115 of FIG. 8B.
[0175] The input assistance application 150 resets the integrated value Xacc of the X coordinate to 0 (step S131X).
[0176] The input assistance application 150 executes the processes of steps S132X, S133X, and S134X included in the subroutine for accumulating the X coordinate. In this subroutine, the i in X(i) is accumulated one by one from 1 to Nma.
[0177] The input assistance application 150 determines whether X(i) is 0xFFFF (step S132X). 0xFFFF means that there is no measured value of the Z value to be used in calculating the X coordinate.
[0178] If the input assistance application 150 determines that X(i) is 0xFFFF (S132X: YES), it updates the integrated value Xacc to Xacc + X(1) (step S133X). That is, if dummy data is stored in X(i), the latest X coordinate (X(1)) is used to calculate the integrated value. Therefore, the calculation Xacc = Xacc + X(1) is performed.
[0179] Furthermore, if the input assistance application 150 determines in step S132X that X(i) is not 0xFFFF (S132X: NO), it updates the integrated value Xacc to Xacc + X(i) (step S134X). That is, it accumulates the previously measured X coordinates (X(i)) in order. Therefore, it performs the calculation Xacc = Xacc + X(i).
[0180] After repeatedly executing the processes of steps S132X, 133X, and 134X for i in X(i) from 1 to Nma, the input assistance application 150 ends the process of the subroutine that accumulates the X coordinate and advances the flow to step S135X.
[0181] The input assistance application 150 calculates the moving average of the X coordinate by dividing the integrated value Xacc by Nma (step S135X). That is, the moving average of the X coordinate Xave=Xacc / Nma.
[0182] <FIG. 8F: Moving Average of Y Coordinate> FIG. 8F shows details of the subroutine process for calculating the moving average of the Y coordinate in step S116 of FIG. 8B.
[0183] The processing of steps S131Y to S135Y shown in Fig. 8F is processing executed by the input assistance application 150 for the Y coordinate Y(i), and is processing in which the processing of steps S131X to S135X for the X coordinate X(i) shown in Fig. 8E is replaced with processing for the Y coordinate Y(i). For this reason, the description of Fig. 8F will be omitted here.
[0184] <FIG. 9A: Processing for Setting the Radius of the Second Pointer 112 to the Initial Value> FIG. 9A shows an example of processing for setting the radius of the second pointer 112 to the initial value in step S7C.
[0185] The input assistance application 150 sets the initial value of the radius "Radius" of the second pointer 112 to the maximum value "Rmax" (step S141A). That is, "Radius" = "Rmax." The user of the non-contact input device 100 can set Rmax for the input assistance application 150 via the setting screen displayed on the display 110.
[0186] <FIG. 9B: Processing for Setting the Radius of the Second Pointer 112 to the Initial Value> FIG. 9B shows another example of the processing for setting the radius of the second pointer 112 to the initial value in step S7C.
[0187] The input assistance application 150 calculates the amount of variation D of the X and Y coordinates as an index of positional accuracy (step S141B). Step S141B is a subroutine process, the details of which will be described later with reference to FIG. 9E. As an example, the input assistance application 150 calculates the amount of variation D when the selection mode is entered. The input assistance application 150 sets the maximum radius Rmax of the second pointer 112 to a value obtained by multiplying the amount of variation D of the X and Y coordinates by a constant (step S142B). That is, Rmax = D × constant. The predetermined constant is a predetermined value.
[0188] The input assistance application 150 sets the radius Radius of the second pointer 112 to the maximum value Rmax set in step S142B (step S143B). That is, Radius = Rmax = D × constant. Note that when performing the processing of FIG. 9B , the input assistance application 150 does not perform processing that enables the user to set the maximum value Rmax.
[0189] <FIG. 9C: Processing for Setting the Radius of the Second Pointer 112> FIG. 9C is a diagram showing an example of the processing for setting the radius of the second pointer in steps S7B and S9B.
[0190] The input assistance application 150 calculates the amount of variation D of the X and Y coordinates as an index of positional accuracy (step S141C). Step S141C is a subroutine process, the details of which will be described later with reference to FIG. 9E. As an example, the input assistance application 150 calculates the amount of variation D when the selection mode is entered.
[0191] The input assistance application 150 uses the amount of variation D calculated in step S141C to calculate the radius Radius of the second pointer 112 (step S142C). Radius = Rmin + (Rmax - Rmin) x (D / Rmax). Note that Rmin is the minimum value of the radius of the second pointer 112 and is, for example, a predetermined value.
[0192] The input assistance application 150 determines whether the radius Radius calculated in step S142C is greater than the maximum value Rmax (step S143C).
[0193] If the input assistance application 150 determines that the radius Radius is greater than the maximum value Rmax (S143C: YES), it sets the radius Radius to the maximum value Rmax (step S144C). That is, Radius=Rmax.
[0194] Furthermore, if the input assistance application 150 determines that the radius Radius is not greater than the maximum value Rmax (S143C: NO), it uses the calculated radius Radius as is.
[0195] As described above, the input assistance application 150, for example, calculates the variation D when the selection mode is entered. When the second pointer 112 starts to be displayed in the selection mode, the size of the second pointer 112 gradually changes, making it easier to visually recognize the second pointer 112 of a size according to the positional accuracy.
[0196] For example, the input assistance application 150 may calculate the amount of variation D when the proximity mode is selected. By calculating the positional accuracy before the second pointer 112 is displayed in the selection mode, the second pointer 112 can be displayed at a size according to the positional accuracy from the time the second pointer 112 starts to be displayed.
[0197] <Fig. 9D: Calculation Process of XY Coordinate Variation Amount D> Fig. 9D is a diagram showing an example of calculation process of XY coordinate variation amount D. Fig. 9D shows an example of calculation process of variation amount D in step S141C of Fig. 8C.
[0198] The input assistance application 150 resets the amount of variation DX of the X coordinate and the amount of variation DY of the Y coordinate (step S151A), i.e., DX=0, DY=0.
[0199] The input assistance application 150 performs a subroutine process to calculate the cumulative amount of variation D. This subroutine includes steps S152A, S153A, and S154A, and is performed by setting i from 1 to Ndev for the X coordinate X(i) and the Y coordinate Y(i). That is, the input assistance application 150 calculates the amount of variation D using the latest Ndev X and Y coordinates.
[0200] The input assistance application 150 determines whether the X coordinate X(i) is 0xFFFF (step S152A).
[0201] If the input assistance application 150 determines that the X coordinate X(i) is 0xFFFF (S152A: YES), it adds a constant to the amount of variation DX of the X coordinate and the amount of variation DY of the Y coordinate (step S153A). That is, DX = DX + constant, and DY = DY + constant.
[0202] Furthermore, if the input assistance application 150 determines that the X coordinate X(i) is not 0xFFFF (S152A: NO), it adds the square of the difference between the X coordinate moving average Xave and the Y coordinate moving average Xave to the X coordinate fluctuation amount DX and the Y coordinate fluctuation amount DY (step S154A). That is, DX = DX + (X(i) - Xave) 2 , DY=DY+(Y(i)-Yave) 2 This becomes:
[0203] The input assistance application 150 includes steps S152A, S153A, and S154A, and after setting i from 1 to Ndev for the X coordinate X(i) and the Y coordinate Y(i), the flow proceeds to step S155A.
[0204] The input assistance application 150 uses DX and DY calculated in the subroutine process to calculate the amount of change D of the XY coordinates according to the following equation (2) (step S155A).
[0205]
[0206] This completes the determination of the amount of variation D. By determining DX and DY in step S154A and the amount of variation D in step S155A, the standard deviation of the XY coordinates can be determined as an index of positional accuracy.
[0207] The input assistance application 150 determines whether the fingertip FT is near the operation surface 105A and is being operated with the finger held up, based on the amount of movement D (step S156A). If the amount of movement D is large, the fingertip FT is far from the operation surface 105A, the finger is not extended, or the finger is laid down. If the amount of movement D is greater than the palm threshold (S156A: YES), a warning is displayed (step S157A). The warning is a message such as "Please hold your finger up and close to the operation surface for operation." In addition, the entire screen may be colored red along with the warning.
[0208] If the fingertip FT is suddenly brought closer to the operation surface 105A from the standby mode, the amount of variation D will be large, resulting in poor positional accuracy. Furthermore, if the fingertip FT is kept close to the operation surface 105A, the dummy data substituted in step S122X will gradually be replaced with the measurement data, and the amount of variation D will become smaller. In this case, the positional accuracy will gradually improve. Furthermore, the amount of variation D may be calculated using the process shown in FIG. 9E.
[0209] <Fig. 9E: Modification of Calculation Process of Variation Amount D of XY Coordinates> Fig. 9E is a diagram showing an example of calculation process of variation amount D of XY coordinates. Fig. 9D shows an example of calculation process of variation amount D in step S141C of Fig. 8C.
[0210] The process shown in Fig. 9E is a modified version of step S155A of the process shown in Fig. 9D. Therefore, steps S151B to S154B in Fig. 9E are the same as steps S151A to S154A in Fig. 9D. Therefore, only step S155B will be described here.
[0211] The input assistance application 150 uses DX and DY calculated in the subroutine process to calculate the amount of change D of the XY coordinates according to the following equation (3) (step S155A).
[0212] D=DX+DY (3)
[0213] 9E, the amount of variation D is not divided by Ndev. For example, if Ndev is a constant, the sum (total) of DX and DY not divided by Ndev may be used as the amount of variation D. Furthermore, instead of the amount of variation D in the XY coordinates, either DX or DY may be used as the amount of variation D. For example, either the amount of variation DX / Ndev in the X coordinate or the amount of variation DY / Ndev in the Y coordinate may be used.
[0214] <Fig. 9F: Processing for Determining Whether Menu Display is Necessary> Fig. 9F is a diagram showing an example of processing for determining whether menu display is necessary. Fig. 9F shows the processing for determining whether menu display is necessary in step S23.
[0215] Here, Nmenu is the number of data items corresponding to the time slot for determining whether or not a menu needs to be displayed. Nmenu may be determined at the time of design, or may be set by the user on a setting screen. Furthermore, Dmenu is the distance corresponding to the threshold value for the amount of movement of the X and Y coordinates.
[0216] The input assistance application 150 performs the X coordinate determination process (determination X) while incrementing the i number of X(i) by 1 from 2 to Nmenu. The X coordinate determination process is a subroutine process.
[0217] The input assistance application 150 determines whether the absolute value of X(i)-X(1) is less than Dmenu (step S161), where X(1) is the latest X coordinate.
[0218] If the absolute values of the differences between X(2) to X(Nmenu) and X(1) are all less than Dmenu (all S161: YES), the input assistance application 150 executes the subroutine processing for the Y coordinate.
[0219] The input assistance application 150 performs the Y coordinate determination process (determination Y) while incrementing the i number of Y(i) by 1 from 2 to Nmenu. The Y coordinate determination process is a subroutine process.
[0220] The input assistance application 150 determines whether the absolute value of Y(i)-Y(1) is less than Dmenu (step S162), where Y(1) is the latest Y coordinate.
[0221] If the absolute values of the differences between Y(2) to Y(Nmenu) and Y(1) are all less than Dmenu (all S162: YES), the input assistance application 150 advances the flow to step S163.
[0222] The input assistance application 150 determines whether the current time - SelectionTime>SelectionTH is true (step S163). That is, if the time elapsed since the selection mode was entered exceeds the predetermined time, the flow proceeds to step S164.
[0223] The input assistance application 150 displays a menu (step S164). The menu is, for example, the function menu 112C (see FIG. 6E). If the finger is moving (S161: NO or S162: NO) or if the selection mode has just been entered (S163: NO), the function menu is not displayed.
[0224] The menu is displayed when the Nmenu number of X and Y coordinates are substantially stationary (when they are stationary) in the selection mode. That is, the input assistance application 150 causes the OS 140 to display the menu when the X and Y coordinates of the fingertip FT are stationary for a period of time during which the Nmenu number of X and Y coordinates are acquired in the selection mode. The period of time during which the Nmenu number of X and Y coordinates are acquired is an example of the third predetermined period of time.
[0225] <Settings Screen> Fig. 10 is a diagram showing an example of a settings screen. The input assistance application 150 causes the OS 140 to display the settings screen shown in Fig. 10 on the display 110. The settings screen is an image representing a screen on which various settings can be input.
[0226] 10 includes five text boxes for setting the mode switching threshold, messages to be displayed on the display 110 in the standby mode and the proximity mode, and text boxes for inputting the moving average of the position of the second pointer 112, the click event issuance time, the minimum and maximum values of the radius of the second pointer 112, and the click re-output time. The threshold for the proximity mode corresponds to the Z value Cz3, the threshold for the selection mode corresponds to the Z value Cz2, and the threshold for the decision mode corresponds to the Z value Cz1.
[0227] 11 is a diagram showing an example of the structure of folders. The folders are stored inside the memory 132 (see FIG. 2), and the input assistance application 150 can access the folders.
[0228] 11 shows the audio file (Click.wav) that can be selected on the setting screen and the background image files (Proximity.png, Waiting.png) for each mode. Note that although the audio file and background image file are shown in one folder here, they may be in separate folders.
[0229] <Flowchart of Setting Process> Fig. 12 is a flowchart showing an example of the setting process. When "Setting" is selected from the menu in the task tray, the input assistance application 150 starts a setting program and executes the process shown in Fig. 12.
[0230] The input assistance application 150 causes the OS 140 to display a setting screen on the display 110 (step S201). The user can input various values and the like into the setting screen shown in FIG.
[0231] The input assistance application 150 determines whether the apply button has been clicked (step S202).
[0232] If the input assistance application 150 determines that the Apply button has been clicked (S202: YES), it sets various values (step S203). Specifically, the input assistance application 150 inputs the value of the proximity mode setting box into ProximityTH, the value of the selection mode setting box into SelectionTH, the value of the decision mode setting box into DecisionTH, the value of the proximity mode OFF determination into ProximityOffTH, and the value of the selection mode OFF determination into SelectionOffTH. The input assistance application 150 also inputs the value of the click re-output time into ReClickTH, the time of the second pointer movement average into MATime, the value of the click event issuance time into DecisionTimeTH, the minimum radius into Rmin, the maximum radius into Rmax, saves a standby mode message, and saves a proximity mode message.
[0233] If the input support application 150 determines in step S202 that the Apply button has not been clicked (S202: NO), it determines whether the OK button has been clicked (step S204). The input support application 150 repeatedly executes the processes from step S202 onwards until the OK button is clicked.
[0234] If the input assistance application 150 determines that the OK button has been clicked (S204: YES), it causes the OS 140 to close the setting screen (step S205). After completing the processing of step S205, the input assistance application 150 ends the flow. Note that if the OK button has been clicked, the flow may end after setting each value, etc.
[0235] <Effects> The non-contact input device 100 includes an operation surface 105A, a detection unit (sensor electrodes 121X, 121Y, input sensor circuit 125A) that measures the two-dimensional position of an object operating on the operation surface 105A in a non-contact manner and the distance from the operation surface 105A to the object, a display 110 that is arranged on the back side of the operation surface 105A, and a control device 130 that has an OS 140 that uses a planar GUI and an input assistance application 150, and the input assistance application 150 outputs to the OS 140 a command to display, in an overlay on the display 110, a second pointer 112 that is different from a first pointer 111 that the OS 140 displays on the display 110, in accordance with the two-dimensional position and distance, and when it determines that an operation by the object has been determined based on the two-dimensional position and distance, it outputs to the OS 140 the two-dimensional position at the time the operation was determined and a click event.
[0236] Therefore, it is possible to provide a non-contact input device 100 that enables non-contact input with three-dimensional movements in a device equipped with an OS and application program that can only handle two-dimensional positions. Also, by having the application display a second pointer instead of the first pointer (mouse pointer) provided by the OS, it is possible to display a second pointer that corresponds to three-dimensional positions even in an OS that displays a first pointer (mouse pointer) that only corresponds to two-dimensional positions.
[0237] The input assistance application 150 may output to the OS 140 a command to hide the first pointer 111. Without making any changes to the OS 140, a second pointer whose size, arc length, etc. can be dynamically changed in response to an operation input can be displayed on the display 110 as a function not provided by the OS 140.
[0238] Furthermore, a decision area (first area) where the distance from the operation surface 105A to the fingertip FT is shorter than a first threshold is a decision area in which the input assistance application 150 executes the decision mode. The input assistance application 150 determines that the fingertip is in the decision area (first area) if the measured capacitance is greater than a first threshold Cz1. When the distance from the operation surface 105A to the fingertip FT (target object) is within the decision area (first area), the input assistance application 150 displays a second pointer 112 that moves in response to an operation performed by the fingertip FT (target object). When the distance from the operation surface 105A to the fingertip FT (target object) remains within the decision area (first area), the input assistance application 150 displays, as the second pointer 112A, an arc that lengthens over time and becomes a ring from stationary within a first predetermined time. The input assistance application 150 may determine that the operation content has been decided when the distance between the fingertip FT (object) and the operation surface 105A and the time during which the fingertip FT (object) remains within the decision area (first area) has elapsed for a first predetermined time, and may display a ring as the second pointer 112B, issue a click event to the OS 140, and output the two-dimensional position at the time the operation content was decided to the OS 140. The user can intuitively understand that the operation content is decided when the fingertip FT is stopped for a certain period of time in the decision area.
[0239] Furthermore, when the distance from the fingertip FT (object) to the operation surface 105A remains within a decision area (first area) after issuing a click event to the OS 140, the input assistance application 150 causes the OS 140 to display, as the second pointer 112A, an arc that lengthens over time and becomes a ring in a second predetermined time from the start of drawing the arc, and when the second predetermined time has elapsed since issuing the click event, causes the OS 140 to display the ring as the second pointer 112B and outputs the two-dimensional position and the click event to the OS 140. This enables continuous decision operations, enables non-contact input with three-dimensional movements, and provides a more user-friendly non-contact input device 100.
[0240] Furthermore, the input assistance application 150 may cause the OS 140 to display an arc that gets longer over time as the distance of the fingertip FT (target object) from the operation surface 105A moves out of the decision area (first area) and then returns to the decision area (first area) as the second pointer 112A. When successive decision operations are performed, the pointer can be displayed immediately.
[0241] Furthermore, a selection area (second area) in which the distance from the operation surface 105A to the fingertip FT is equal to or greater than the first threshold and shorter than a second threshold that is longer than the first threshold is a selection area in which the input assistance application 150 executes a selection mode. When the distance of the object is located within the second area, the input assistance application 150 causes the OS 140 to display a second pointer 112 that moves in accordance with the two-dimensional position of the object, and when the distance of the object remains within the second area for a third predetermined time and the two-dimensional position of the object becomes stationary, causes the OS 140 to display a function menu in an overlay. The input assistance application 150 may cause the OS 140 to display a second pointer 112A that moves in response to an operation performed by the fingertip FT (object) when the fingertip FT (object) is positioned within the selection area (second area) from the operation surface 105A, and may cause the OS 140 to display a function menu in an overlay when the fingertip FT (object) remains stationary within the selection area (second area) for a third predetermined time. As with the second pointer 112 (112A, 112B), the function menu can be displayed without making any changes to the OS 140.
[0242] Furthermore, when the distance from the operation surface 105A of the fingertip FT (object) is within the selection area (second area), the input assistance application 150 may calculate the amount of variation in the two-dimensional position calculated at multiple points in time as an index of the positional accuracy of the fingertip FT (object), and may cause the OS 140 to display the second pointer 112 smaller as the positional accuracy of the fingertip FT (object) improves. The second pointer 112 can be displayed smaller in accordance with the positional accuracy by an overlay without making any changes to the OS 140. Furthermore, by displaying the second pointer 112 smaller in accordance with the positional accuracy, the positional accuracy can be visually communicated to the user.
[0243] Furthermore, the input assistance application 150 may transition from the decision mode to the selection mode when a fourth predetermined time has elapsed since the fingertip FT (target object) moves from the decision area (first area) to the selection area (second area) from the operation surface 105A. This allows stable switching from the decision mode to the selection mode.
[0244] Furthermore, a proximity region (third region) in which the distance from the operation surface 105A to the fingertip FT is equal to or greater than a second threshold and shorter than a third threshold that is longer than the second threshold is a proximity region in which the input assistance application 150 executes the proximity mode. When the distance is within the third region, the input assistance application 150 outputs to the OS 140 a command to display an image or message for the proximity mode as an overlay on the display 110. When the two-dimensional position of the fingertip FT (target object) is within the proximity region (third region), the input assistance application 150 may output to the OS 140 a command to display an image or message for the proximity mode as an overlay on the display 110. Because the image or message for the proximity mode can be displayed as an overlay, it is possible to easily display the image or message for the proximity mode to the user without making any changes to the OS 140. Using an image showing an operation method as the image for the proximity mode allows the operation method to be shown at an appropriate time. Furthermore, by providing an operation screen on the input assistance application 150 side that allows images and messages to be easily replaced, the administrator of the non-contact input device 100 can display images and messages on the display 110 that are in line with the application program.
[0245] The fourth region, where the distance from the operation surface 105A to the fingertip FT is equal to or greater than a third threshold, is a standby region in which the input assistance application 150 does not calculate the two-dimensional position of the fingertip FT (object). The input assistance application 150 outputs an instruction to the OS to display an image or message for standby mode as an overlay on the display 110 when an object is not detected in the first region, the second region, or the third region. The input assistance application 150 may also output an instruction to the OS 140 to display an image or message for standby mode as an overlay on the display 110 when the fingertip FT (object) is not detected in the decision region (first region), the selection region (second region), or the proximity region (third region). Images and messages for standby mode can be easily displayed to the user without modifying the OS 140. Furthermore, images and messages can be easily replaced on the input assistance application 150 side, allowing images and messages consistent with the application program to be displayed on the display 110.
[0246] Furthermore, the input assistance application 150 may calculate a moving average of the two-dimensional positions acquired at multiple points in time as the two-dimensional position of the fingertip FT (object), and display the second pointer 112 (112A, 112B) at the calculated two-dimensional coordinate position on the OS 140. By calculating the moving average of the X and Y coordinates as the X and Y coordinates of the fingertip FT, it is possible to obtain the X and Y coordinates of the fingertip FT in which fluctuations due to external noise and hand movement are suppressed, and it is possible to display the second pointer 112 (112A, 112B) at highly accurate X and Y coordinates in which the influence of the amount of fluctuation is suppressed.
[0247] Furthermore, when the input assistance application 150 determines that the operation content by the fingertip FT (object) has been confirmed, it may output to the OS 140 the two-dimensional position of the fingertip FT (object) calculated as a moving average of the two-dimensional position when the operation content was confirmed, and a click event. The input assistance application 150 can output to the OS 140 that the confirmation operation has been performed using highly accurate X and Y coordinates calculated using the moving average, thereby allowing the content of the confirmation operation to be reflected with high precision. Furthermore, since the X and Y coordinates are not transmitted to the OS 140 until a click event is issued, the processing load on the OS 140 can be reduced.
[0248] The input assistance application 150 may also calculate the index of positional accuracy of the fingertip FT (object) by summing the amount of fluctuation of multiple two-dimensional positions using multiple points in time where the number of pieces of data to be used for calculation is set by the user. The index of positional accuracy can be calculated according to the number of two-dimensional positions set by the user. The user can adjust the level of the index of positional accuracy.
[0249] In addition, the input assistance application 150 outputs a command to the OS 140 to overlay the second pointer 112 on the display 110 in the size set by the user. The user can set the size of the second pointer 112 without making any changes to the OS 140. The user can set the size of the second pointer 112 via the input assistance application 150.
[0250] Furthermore, the input assistance application 150 may set the size of the second pointer to be proportional to the inverse of the Z value (a value proportional to capacitance). In this case, too, it is possible to encourage the user to bring the fingertip FT closer to the operation surface 105A. However, the size of the second pointer based on the Z value has no statistical significance.
[0251] The above describes a non-contact input device and an input assistance application according to exemplary embodiments of the present disclosure. However, the present disclosure is not limited to the specifically disclosed embodiments, and various modifications and variations are possible without departing from the scope of the claims.
[0252] This international application claims priority based on Japanese Patent Application No. 2023-078754, filed on May 11, 2023, the entire contents of which are incorporated herein by reference.
[0253] REFERENCE SIGNS LIST 100 Non-contact input device 101 Housing 105 Top panel 105A Operation surface 110 Display 120 Electrostatic sensor 121X, 121Y Sensor electrode 125A Input sensor circuit 125B Image display circuit 130 Control device 131 Control unit 132 Memory 140 OS 150 Input support application 160 Application program
Claims
1. A non-contact input device comprising: an operation surface; a detection unit that measures a two-dimensional position of an object operating on the operation surface without contact and a distance from the operation surface to the object; a display arranged on the rear side of the operation surface; and a control device having an OS that uses a planar GUI and an input assistance application, wherein the input assistance application outputs a command to the OS to cause the OS to display a second pointer different from a first pointer displayed on the display as an overlay on the display in accordance with the two-dimensional position and the distance, and when it is determined that an operation by the object has been determined based on the two-dimensional position and the distance, outputs the two-dimensional position at the time the operation was determined and a click event to the OS.
2. The non-touch input device according to claim 1, wherein the input assistance application outputs a command to the OS to make the first pointer invisible.
3. The non-contact input device according to claim 1 or 2, wherein the first area where the distance is shorter than a first threshold is a decision area where the input assistance application outputs a click event, and the input assistance application: when the distance of the object is located within the first area, causes the OS to display the second pointer which moves in accordance with an operation performed by the object; when the distance of the object remains within the first area, causes the OS to display as the second pointer an arc that lengthens with the passage of time and that becomes a ring a first predetermined time after the object enters the first area; and when the time that the distance of the object remains within the first area exceeds the first predetermined time, determines that the operation has been decided, causes the ring to be displayed as the second pointer via the OS, issues a click event to the OS, and outputs to the OS the two-dimensional position at the time the operation was decided.
4. The non-contact input device of claim 3, wherein the input assistance application causes the OS to display as the second pointer an arc that grows longer over time if the distance of the object continues to remain within the first area after a click event is issued to the OS, and that becomes a ring in a second predetermined time from the start of drawing the arc, and when the second predetermined time has elapsed since the click event was issued, causes the OS to display the ring as the second pointer, and outputs the two-dimensional position and the click event to the OS.
5. The non-contact input device according to claim 4, wherein the input assistance application causes the OS to display as the second pointer an arc that becomes longer with time when the distance of the object moves outside the first area and then back into the first area.
6. A non-contact input device according to any one of claims 3 to 5, wherein when the first predetermined time is set to 0, the input assistance application determines that the operation has been confirmed when the distance of the object is located within the first area, issues a click event to the OS, and outputs the two-dimensional position at the time the operation was confirmed to the OS.
7. The non-contact input device according to claim 3, wherein the input assistance application causes the OS to display a circle smaller than the annular ring as the second pointer after the click event is output.
8. A non-contact input device according to any one of claims 3 to 7, wherein a second region in which the distance is equal to or greater than the first threshold and shorter than a second threshold longer than the first threshold is a selection region in which the input assistance application executes a selection mode, and the input assistance application transitions from a decision mode in which the click event is output to the selection mode when a fourth predetermined time has elapsed since the distance of the object has moved from the first region to the second region.
9. The non-contact input device according to claim 8, wherein the input assistance application calculates the sum of the amounts of fluctuation in the two-dimensional positions calculated at multiple points in time when the object is located within the second area as an index of the object's positional accuracy, and causes the OS to display the second pointer smaller as the object's positional accuracy improves.
10. The non-contact input device according to claim 8 or 9, wherein the input assistance application causes the OS to display the second pointer that moves according to the two-dimensional position of the object when the distance of the object is located within the second area, and causes the OS to display a function menu in an overlay when the distance of the object remains within the second area for a third predetermined time and the two-dimensional position of the object becomes stationary.
11. A non-contact input device according to any one of claims 8 to 10, wherein a third region in which the distance is equal to or greater than the second threshold and shorter than a third threshold longer than the second threshold is a proximity region in which the input assistance application executes a proximity mode, and when the distance is within the third region, the input assistance application outputs to the OS a command to display an image or message for the proximity mode in an overlay on the display.
12. The non-contact input device according to claim 11, wherein a fourth region where the distance is equal to or greater than the third threshold is a standby region in which the input assistance application does not output the two-dimensional position of the object, and when the object is not detected in the first region, the second region, or the third region, the input assistance application outputs an instruction to the OS to display an image or message for a standby mode as an overlay on the display.
13. A non-contact input device according to any one of claims 9 to 12, wherein the input assistance application calculates a moving average of the two-dimensional positions obtained at multiple points in time as the two-dimensional position of the object, and displays the second pointer at the calculated two-dimensional coordinate position via the OS.
14. The non-contact input device according to claim 13, wherein when the input assistance application determines that an operation by the object has been confirmed, it outputs to the OS the two-dimensional position of the object calculated as a moving average of the two-dimensional position at the time the operation was confirmed, and the click event.
15. The non-contact input device according to claim 14, wherein the input assistance application calculates the sum of the amounts of variation in the multiple two-dimensional positions as an index of the positional accuracy of the object, using the multiple points in time at which the number of pieces of data to be used for calculation is set by the user.
16. The non-touch input device according to any one of claims 1 to 15, wherein the input assistance application outputs to the OS a command to display, in an overlay on the display, the size of the second pointer set by the user.
17. An input assistance application for operating an application program running on an OS that uses a planar GUI, using a detection unit that measures the two-dimensional position of an object performing a non-contact operation input on an operation surface and the distance from the operation surface to the object, wherein the input assistance application outputs to the OS a command to display, as an overlay on the display, a second pointer different from a first pointer displayed by the OS on the display, in accordance with the two-dimensional position and the distance, and when it determines that an operation by the object has been determined based on the two-dimensional position and the distance, outputs to the OS the two-dimensional position and a click event at the time the operation was determined.
Citation Information
Patent Citations
Three-dimensional instruction input device
JP2004303000A
Input assistance device, input assistance method, and program
JP2014219938A
Remote hover touch system and method
US20170364198A1
Three-dimensional display devices with out-of-screen virtual keyboards
US9411511B1
Input terminal device and operation input method
WO2017064771A1
