Terminal device, operation method, and program

The terminal device improves non-contact operation detection accuracy by using a capacitive touch sensor and a control unit to confirm user instructions with two operators, addressing the issue of unreliable detection in dirty or painted environments by differentiating between intended and unintended hand movements.

WO2024241952A1PCT designated stage expired Publication Date: 2024-11-28KYOCERA CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017689
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2024-05-13
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing non-contact operation detection techniques face challenges in accuracy, particularly in environments where user hands may be dirty or painted, leading to unreliable detection due to unintended hand movements and fluctuations.

Method used

A terminal device equipped with a capacitive touch sensor and a control unit that utilizes two operators in a hover state to confirm user instructions by detecting capacitance changes, differentiating between intended operations and unintended hand fluctuations through a calibration process and specific threshold settings.

Benefits of technology

Enhances the detection accuracy of non-contact operations by confirming user instructions based on the presence and positioning of both operators, reducing false positives from unintended hand movements and improving operational reliability in various environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017689_28112024_PF_FP_ABST
    Figure JP2024017689_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This terminal device is provided with a display and a control unit. The control unit determines, on the basis of information of a second operator and information of a first operator in a hover state with respect to the display, an instruction of a user with respect to an object displayed on the display.
Need to check novelty before this filing date? Find Prior Art

Description

Terminal device, operation method and program Cross-reference to related applications

[0001] This application claims priority to Japanese Patent Application No. 2023-084861, filed on May 23, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to a terminal device, an operation method, and a program.

[0003] Conventionally, there is known a technique for detecting a non-contact operation. For example, Patent Literature 1 describes a technique for identifying a moving direction of an operator and executing a process associated with the identified moving direction.

[0004] Japanese Patent Application Laid-Open No. 2019-212087

[0005] A terminal device according to one embodiment of the present disclosure includes a display and a control unit that confirms a user's instruction for an object displayed on the display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0006] An operating method according to one embodiment of the present disclosure includes determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0007] A program according to one embodiment of the present disclosure causes a computer to perform an operation including determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0008] FIG. 1 is an external view of a terminal device according to an embodiment of the present disclosure. FIG. 2 is a top view of a display unit shown in FIG. 1. FIG. 3 is a block diagram of the terminal device shown in FIG. 1. FIG. 4 is a flowchart illustrating an example procedure of a confirmation process according to an embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an example procedure of a confirmation process according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an XY coordinate system set for a display. FIG. 7 is a diagram illustrating an XY coordinate system set for a display. FIG. 8 is a diagram illustrating an example of a display mode of a display. FIG. 9 is a diagram illustrating an example of a display mode of a display. FIG. 10 is a block diagram of a terminal device according to another embodiment of the present disclosure. FIG. 11 is a diagram illustrating a state when the display is approximately parallel to the vertical direction. FIG. 12 is a diagram illustrating a state when the display is approximately parallel to the horizontal direction. FIG. 13 is a flowchart illustrating an example procedure of a notification process according to another embodiment of the present disclosure.

[0009] It would be useful if the detection accuracy of a non-contact operation could be improved. According to an embodiment of the present disclosure, the detection accuracy of a non-contact operation can be improved.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0011] As shown in Fig. 1, a terminal device 10 according to this embodiment includes a display unit 11. The display unit 11 displays various objects that can be operated by a user. The objects include, for example, icons, windows, toolbars, menus, buttons, sliders, etc. The terminal device 10 detects a non-contact operation by the user on the object.

[0012] The terminal device 10 may be used in any location. For example, the terminal device 10 may be used in a location where the user's hands may become dirty or wet. In this case, the terminal device 10 may be used in, for example, a kitchen, a bathroom, a factory, or the like.

[0013] Hereinafter, a user operation according to this embodiment for confirming an instruction to an object displayed on the display unit 11 will be described.

[0014] First, the user points the first operator 1 in a hover state toward an object that the user wants to execute among the objects displayed on the display unit 11. The user maintains the first operator 1 in a stationary state while hovering toward the object. The hover state refers to a state in which the distance between the operator and the display unit 11 is within a predetermined distance and the operator is not in contact with the display unit 11. The predetermined distance is, for example, a distance corresponding to a capacitance threshold value described below. The first operator 1 is, for example, the index finger of the user's hand. However, the first operator 1 does not have to be the index finger of the user's hand. The first operator 1 may be any finger of the user as long as it is pointed toward the object. For example, as shown in FIG. 1 , the user extends the index finger, which is the first operator 1, and grasps the other fingers. The user points the object that the user wants to execute with the first operator 1, which is the index finger, in a hover state.

[0015] When the user points the first operator 1 in a hover state toward an object, the object enters a selected state. When the user wants to confirm an instruction for the selected object, the user places the second operator 2 in a hover state on the first operator 1 in a stationary state, as shown in FIG. 2 . The second operator 2 is the finger next to the first operator 1. When the first operator 1 is the user's index finger, the second operator 2 is, for example, the user's middle finger. When the second operator 2 is placed on the first operator 1, the user's instruction for the selected object is confirmed.

[0016] 3, the terminal device 10 includes a storage unit 14 and a control unit 15 in addition to a display unit 11. The display unit 11 includes a display 12 and a detection sensor 13.

[0017] The display 12 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0018] The detection sensor 13 is capable of detecting capacitance. The detection sensor 13 is, for example, a capacitance-type touch sensor. The detection sensor 13 may be a self-capacitance type or a mutual-capacitance type. The detection sensor 13 is configured to include, for example, a transparent film and a transparent electrode such as ITO (Indium Tin Oxide). The transparent electrode may have a mesh pattern. Alternatively, strip-shaped transparent electrodes may be arranged in a grid pattern.

[0019] The detection sensor 13 is disposed on the display 12. By disposing the detection sensor 13 on the display 12, the detection sensor 13 can detect the capacitance of the display 12 in an XY coordinate system as shown in FIG. 6 (described later). The XY coordinate system is a two-dimensional coordinate system set with respect to the display 12. The X direction corresponds to the direction from left to right as viewed from the display 12. The Y direction corresponds to the direction from bottom to top as viewed from the display 12. However, the X and Y directions may be set arbitrarily with respect to the display 12. Each lattice point shown in FIG. 6 corresponds to each XY coordinate at which capacitance is detected by the detection sensor 13 in this embodiment. These lattice points are also referred to as "nodes."

[0020] The storage unit 14 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 14 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores data used in the operation of the terminal device 10 and data obtained by the operation of the terminal device 10.

[0021] The control unit 15 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 15 executes processing related to the operation of the terminal device 10 while controlling each unit of the terminal device 10.

[0022] <Confirmation Process> The control unit 15 confirms a user's instruction for an object displayed on the display 12, based on information about the first operator 1 and information about the second operator 2 that are in a hover state over the display 12. An example of this confirmation process will be described below with reference to FIGS. 4 and 5 .

[0023] 4 and 5 are flowcharts illustrating an example procedure of a determination process according to an embodiment of the present disclosure.

[0024] For example, when the terminal device 10 is installed, the control unit 15 executes a calibration process (step S1). The calibration process is a process in which the detection sensor 13 detects reference values ​​of capacitance at each X and Y coordinate of the display 12. The reference values ​​of capacitance at the X and Y coordinates are the capacitance at the X and Y coordinates when the user is not using the terminal device 10, i.e., when the user's hand or the like is not close to the display 12. The capacitance at each X and Y coordinate of the display 12 changes depending on the environment around the terminal device 10. Therefore, the control unit 15 executes the calibration process to detect the reference values ​​of capacitance at each X and Y coordinate of the display 12. The control unit 15 may store the detected reference values ​​of capacitance at each X and Y coordinate of the display 12 in the storage unit 14.

[0025] When the control unit 15 executes the calibration process, the detection sensor 13 can detect the change in capacitance at each X and Y coordinate of the display 12. The change in capacitance at each X and Y coordinate is the change from the reference value of the capacitance at that X and Y coordinate.

[0026] The control unit 15 detects the change in capacitance at each X and Y coordinate of the display 12 using the detection sensor 13 and determines whether the first operator 1 is present (step S2). As an example of this process, the control unit 15 may determine that the first operator 1 is present if the change in capacitance at each of multiple consecutive X and Y coordinates exceeds a capacitance threshold. The capacitance threshold may be set based on the distance between the hovered operator that the terminal device 10 wants to detect as a processing target and the display unit 11. For example, in FIG. 6, the change in capacitance at the X and Y coordinates corresponding to each lattice point within the circle 1A exceeds the capacitance threshold. In FIG. 6, the control unit 15 determines that the first operator 1 is present.

[0027] If the control unit 15 determines that the first operator 1 exists (step S2: YES), the control unit 15 proceeds to the process of step S3. If the control unit 15 does not determine that the first operator 1 exists (step S2: NO), the control unit 15 executes the process of step S2 again.

[0028] In the process of step S3, the control unit 15 acquires the position of the first operator 1 on the display 12. As an example of this process, the control unit 15 may acquire, as the position of the first operator 1 on the display 12, the XY coordinates at which the change in capacitance is greatest among the multiple XY coordinates corresponding to the first operator 1. For example, in FIG. 6 , among the change in capacitance at the XY coordinates corresponding to the lattice points within the circle 1A, the change in capacitance at coordinate 1B is greatest. In FIG. 6 , the control unit 15 acquires coordinate 1B as the position of the first operator 1 on the display 12. The control unit 15 may store the acquired position of the first operator 1 and the change in capacitance at the multiple XY coordinates corresponding to the first operator 1 in the storage unit 14.

[0029] The control unit 15 determines whether the position of the first operator 1 on the display 12 has changed within a predetermined time after determining in step S2 that the first operator 1 is present (step S4). The control unit 15 may determine whether the position of the first operator 1 has changed within the predetermined time by detecting a change in capacitance at each X and Y coordinate of the display 12 using the detection sensor 13. The predetermined time may be set based on the specifications of the terminal device 10, the user's usage pattern, etc.

[0030] If the control unit 15 determines that the position of the first operator 1 on the display 12 has changed within a predetermined time since determining that the first operator 1 is present (step S4: YES), the control unit 15 returns to the processing of step S2. If the control unit 15 determines that the position of the first operator 1 on the display 12 has not changed within a predetermined time since determining that the first operator 1 is present (step S4: NO), the control unit 15 proceeds to the processing of step S5.

[0031] In the processing of step S5, the control unit 15 selects the object pointed at by the first operator 1 in the hover state. The control unit 15 may identify the object to be selected based on the position of the first operator 1 on the display 12. For example, in FIG. 6 , the control unit 15 identifies the object to be selected based on coordinates 1B. As will be described later with reference to FIG. 9 , the control unit 15 may display the selected object on the display 12 in a display mode different from that of other objects.

[0032] In the process of step S5, if there is no object in the direction in which the first operator 1 in the hover state is pointed, the control unit 15 may return to the process of step S2.

[0033] In the process of step S6, the control unit 15 detects the change in capacitance at each X and Y coordinate of the display 12 using the detection sensor 13 and determines whether the second operator 2 is present. The control unit 15 may determine whether the second operator 2 is present within a predetermined range from the first operator 1. The predetermined range may be set based on the average distance between adjacent fingers on an open hand. Similarly to or similar to the first operator 1, the control unit 15 may determine that the second operator 2 is present if the change in capacitance at each of multiple consecutive X and Y coordinates exceeds a capacitance threshold. For example, in FIG. 7 , the change in capacitance at the X and Y coordinates corresponding to each lattice point within the circle 2A exceeds the capacitance threshold. Furthermore, each lattice point within the circle 2A is within a predetermined range from the coordinate 1B. The control unit 15 determines that the second operator 2 is present within the predetermined range from the first operator 1.

[0034] If the control unit 15 determines that the second operator 2 is present (step S6: YES), the control unit 15 proceeds to the process of step S7. Here, if the control unit 15 determines that the second operator 2 is present (step S6: YES), the control unit 15 may cause the display 12 to display an indicator such as that described later with reference to Fig. 9. If the control unit 15 determines that the second operator 2 is not present (step S6: NO), the control unit 15 returns to the process of step S3.

[0035] In the processing of step S7, the control unit 15 acquires the position of the second operator 2 on the display 12. In the same manner as or similar to the first operator 1, the control unit 15 may acquire, as the position of the second operator 2 on the display 12, the XY coordinates at which the change in capacitance at a plurality of XY coordinates corresponding to the second operator 2 is greatest. For example, in FIG. 7 , among the change in capacitance at the XY coordinates corresponding to the lattice points within the circle 2A, the change in capacitance at coordinate 2B is greatest. In FIG. 7 , the control unit 15 acquires coordinate 2B as the position of the second operator 2 on the display 12. The control unit 15 may store the acquired position of the second operator 2 and the change in capacitance at the plurality of XY coordinates corresponding to the second operator 2 in the storage unit 14.

[0036] In the processing of step S8, the control unit 15 sets a first line segment. The first line segment is a line segment connecting the position of the first operator 1 and the position of the second operator 2 on the display 12. In other words, the first line segment is a portion of a line passing through the positions of the first operator 1 and the second operator 2 on the display 12, between the positions of the first operator 1 and the second operator 2. For example, in FIG. 7 , the control unit 15 sets a first line segment 3. After setting the first line segment, the control unit 15 detects a change in capacitance at X and Y coordinates on or near the first line segment using the detection sensor 13. The X and Y coordinates near the first line segment may be, for example, X and Y coordinates at which capacitance can be detected by the detection sensor 13 and may be the X and Y coordinates closest to the first line segment. For example, in FIG. 7 , the control unit 15 detects a change in capacitance at X and Y coordinates on or near the first line segment 3.

[0037] In the process of step S9, the control unit 15 determines whether the change in capacitance at the XY coordinates on or near the first line segment detected in the process of step S8 exceeds a determination threshold. The determination threshold may be set based on the change in capacitance between two adjacent fingers when the user points the two fingers toward the display 12. The control unit 15 may determine whether all of the change in capacitance at the XY coordinates on or near the first line segment exceed the determination threshold. Alternatively, the control unit 15 may determine whether the minimum or representative value of the change in capacitance at the XY coordinates on or near the first line segment exceeds the determination threshold. The representative value may be, for example, an average value, a mode value, or a median.

[0038] If the control unit 15 determines that the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold (step S9: YES), the control unit 15 proceeds to the process of step S11 shown in Fig. 5. If the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold, it is highly likely that the second operator 2 is being brought closer to the first operator 1.

[0039] If the control unit 15 determines that the change in capacitance in the XY coordinates on or near the first line segment is equal to or less than the determination threshold (step S9: NO), the control unit 15 proceeds to the processing of step S10. If the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold, it is unlikely that the second operator 2 is being brought closer to the first operator 1.

[0040] In the process of step S10, the control unit 15 detects the change in capacitance at each X and Y coordinate of the display 12 using the detection sensor 13, and determines whether or not the second operator 2 is present. The control unit 15 may execute the process of step S10 in the same manner as or similar to the process of step S6. If the control unit 15 determines that the second operator 2 is present (step S10: YES), the control unit 15 returns to the process of step S7. If the control unit 15 determines that the second operator 2 is not present (step S10: NO), the control unit 15 returns to the process of step S3.

[0041] In the processing of step S11 shown in Figure 5, the control unit 15 detects the change in capacitance in the XY coordinates on or near the first line segment using the detection sensor 13, in the same or similar manner as the processing of step S8.

[0042] In the process of step S12, the control unit 15 selects target coordinates from among the coordinates on or near the first line segment based on the change in capacitance detected in the process of step S11. The target coordinates are used to set the second line segment, which will be described later. The control unit 15 may select any coordinates from the X and Y coordinates on or near the first line segment as the target coordinates. As an example, the control unit 15 may select, as the target coordinates, the coordinates at which the change in capacitance is smallest among the X and Y coordinates on or near the first line segment. For example, in FIG. 7 , the control unit 15 selects coordinate 4 on the first line segment 3 as the target coordinate.

[0043] In the processing of step S13, the control unit 15 sets a second line segment. The second line segment is a line segment that passes through the target coordinates and is perpendicular to the first line segment. For example, in FIG. 7 , the control unit 15 sets a second line segment 5. The length from the target coordinates to one end of the second line segment and the length from the target coordinates to the other end of the second line segment may be set based on the thickness of the user's finger, etc. The control unit 15 detects a change in capacitance at the XY coordinates on or near the second line segment using the detection sensor 13. For example, in FIG. 7 , the control unit 15 detects a change in capacitance at the XY coordinates on or near the second line segment 5 using the detection sensor 13.

[0044] In step S14, the control unit 15 determines whether the change in capacitance at the X and Y coordinates on or near the second line segment detected in step S13 exceeds a change threshold. The change threshold may be set based on the change in capacitance between two adjacent fingers when the user holds the two fingers together and points them toward the display 12. The control unit 15 may determine whether all of the change in capacitance at the X and Y coordinates on or near the second line segment exceeds the change threshold. Alternatively, the control unit 15 may determine whether the minimum or representative value of the change in capacitance at the X and Y coordinates on or near the second line segment exceeds the change threshold. The representative value may be, for example, an average, a mode, or a median. For example, in FIG. 8 , the control unit 15 determines that the change in capacitance at the X and Y coordinates on or near the second line segment 5 exceeds the change threshold.

[0045] If the control unit 15 determines that the change in capacitance at the XY coordinates on or near the second line segment exceeds the change threshold (step S14: YES), the control unit 15 proceeds to the process of step S15. If the control unit 15 determines that the change in capacitance at the XY coordinates on or near the second line segment is equal to or less than the change threshold (step S14: NO), the control unit 15 returns to the process of step S13.

[0046] In the process of step S15, the control unit 15 confirms the user's instruction for the selected object. The control unit 15 may generate a control signal so that a process corresponding to the object for which the instruction has been confirmed is executed. The control unit 15 may transmit the generated control signal to other components of the terminal device 10.

[0047] Through the processing of steps S14 and S15, if the control unit 15 determines that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, it can confirm the user's instruction to the object displayed on the display 12. The change threshold can also be called a distance threshold.

[0048] Here, if the control unit 15 determines that the second operator 2 is present within a predetermined range from the first operator 1 (step S6: YES), the control unit 15 may use the detection sensor 13 to detect a change in capacitance on the display 12 before and after determining that the second operator 2 is present. The control unit 15 may calculate an area on the display 12 where a change in capacitance on the display 12 exceeds a capacitance threshold before and after determining that the second operator 2 is present. The control unit 15 may calculate a difference between an area on the display 12 where a change in capacitance on the display 12 exceeds the capacitance threshold before determining that the second operator 2 is present and an area on the display 12 where a change in capacitance on the display 12 exceeds the capacitance threshold after determining that the second operator 2 is present. If the calculated difference exceeds a predetermined value, the control unit 15 may proceed to the processing of step S7. The predetermined value may be set based on an area on the display 12 where a change in capacitance on the display 12 caused by one finger exceeds the capacitance threshold. Here, an example of an operation using two operators is a pinch-in operation. A pinch-in operation is an operation in which two operators are brought closer to each other to narrow the distance between the two operators. Therefore, in a pinch-in operation, the area where the change in capacitance on the display 12 exceeds the capacitance threshold value does not change before and after it is determined that one operator is present within a predetermined range from the other operator. By proceeding to the processing of step S7 when the difference calculated as described above exceeds the predetermined value, the operation of confirming an instruction to an object can be more clearly distinguished from a pinch-in operation.

[0049] During the execution of the processes of steps S1 to S15, the control unit 15 may detect contact of the user's hand or the like with the display unit 11 using the detection sensor 13. In this case, the control unit 15 may cause a notice to be displayed on the display 12 to call attention. As an example, the control unit 15 may cause text information such as "Do not touch" to be displayed on the display 12. By displaying the notice to be displayed on the display 12, the user can remove their hand or the like from the display unit 11. This configuration makes it possible to avoid a situation in which the first operator 1 and the second operator 2 in the hover state cannot be detected.

[0050] <Display Processing> The control unit 15 may display a selected object from among multiple objects displayed on the display 12 in a display mode different from the other objects. As described above, in the processing of step S5, the control unit 15 selects the object toward which the first operator 1 in the hover state is pointed. In the processing of step S5, the control unit 15 may display the selected object in a display mode different from the other objects.

[0051] For example, objects 16a, 16b, 16c, 16d, 16e, and 16f are displayed on the display 12 shown in FIG. 9. The objects 16a to 16f are menus. In FIG. 9, the first operator 1 in a hover state is pointed at the object 16b. The control unit 15 places the object 16b in a selected state by the processing of step S5. The control unit 15 displays the selected object 16b in a display mode different from the other objects 16a and the like. In FIG. 9, the control unit 15 displays the object 16b on the display 12 in a mode in which it is surrounded by a frame. This configuration allows the user to easily recognize the selected object 16b.

[0052] When the control unit 15 determines that the second operator 2 is present within a predetermined range from the first operator 1 as described above (step S6: YES), the control unit 15 may display an indicator on the display 12. This indicator indicates the distance between the first operator 1 and the second operator 2. This indicator may indicate the distance between the first operator and the second operator in association with a distance threshold. The control unit 15 may calculate the distance between the first operator 1 and the second operator 2 based on the positions of the first operator 1 and the second operator 2 on the display 12. The control unit 15 may display the indicator near the selected object, or may display the indicator superimposed on the selected object.

[0053] For example, an indicator 17 is displayed on the display 12 shown in FIG. 9 . The indicator 17 is displayed near the selected object 16 b. The indicator 17 includes circles 17 a and 17 b. The radius of the circle 17 a corresponds to the distance between the first operator 1 and the second operator 2. The shorter the distance between the first operator 1 and the second operator 2, the larger the radius of the circle 17 a. The radius of the circle 17 b corresponds to the distance threshold. The center of the circle 17 a and the center of the circle 17 b are located at the same position. In other words, the circles 17 a and 17 b are concentric. When the radius of the circle 17 a becomes larger than the radius of the circle 17 b, the user's instruction for the selected object 16 b is confirmed. FIG. 10 shows the indicator 17 when the user's instruction for the object 16 b is confirmed. In FIG. 10 , the radius of the circle 17 a is larger than the radius of the circle 17 b. By displaying such an indicator 17, the user can easily understand how much the distance between the first operator 1 and the second operator 2 needs to be reduced before the user's instruction to the selected object 16b is confirmed.

[0054] In this manner, in the terminal device 10 according to this embodiment, the control unit 15 determines the user's instruction for the object displayed on the display based on the information of the first operator 1 and the information of the second operator 2 that are in a hover state over the display 12.

[0055] As a comparative example, assume that a user confirms an instruction to a selected object by moving the first operator 1 closer to or farther away from the selected object. The user's hand tends to fluctuate in the air. Therefore, it is difficult to maintain a constant distance between the first operator 1 and the display unit 11. In other words, the first operator 1 may be moved closer to or farther away from the selected object due to an unintentional fluctuating hand movement by the user. In this case, in the comparative example, the instruction to the selected object may be confirmed due to an unintentional fluctuating hand movement by the user.

[0056] In contrast to this comparative example, the control unit 15 according to this embodiment confirms a user's instruction for an object displayed on the display 12 based on information about the first operator 1 and information about the second operator 2. That is, in this embodiment, an instruction for an object is confirmed using information about the first operator 1 and the second operator 2, rather than only information about the first operator 1. By using information about the first operator 1 and information about the second operator 2 in this manner, this embodiment can avoid a situation in which an instruction for an object is confirmed due to an unintended movement of the user's hand, as in the comparative example.

[0057] Therefore, according to this embodiment, it is possible to improve the detection accuracy of the non-contact operation.

[0058] Furthermore, in this embodiment, when the control unit 15 determines that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, the control unit 15 may confirm a user's instruction for an object displayed on the display 12. Here, an example of an operation using two operators is a pinch-out operation. A pinch-out operation is an operation in which the two operators are moved away from each other to increase the distance between them. By confirming an instruction for an object when it is determined that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, the operation for confirming an instruction for an object is more clearly distinguished from a pinch-out operation. Therefore, an instruction for an object can be confirmed with high accuracy.

[0059] Furthermore, in this embodiment, the control unit 15 may determine whether or not the second operator 2 is present if the position of the first operator 1 on the display 12 does not change for a predetermined time after determining that the first operator 1 is present. Here, as described above, an example of an operation using two operators is the pinch-in operation. In this pinch-in operation, the two operators are brought closer to each other, so the positions of the two operators change. By determining whether or not the second operator 2 is present when the position of the first operator 1 does not change for a predetermined time, an operation for confirming an instruction to an object can be more clearly distinguished from a pinch-in operation.

[0060] Furthermore, in this embodiment, when the position of the first operator 1 on the display 12 does not change for a predetermined time, the control unit 15 may select an object, of the multiple objects displayed on the display 12, that the first operator is pointing at. The control unit 15 may display the selected object in a different display mode from the other objects. With this configuration, the user can easily identify the selected object, as described above with reference to FIG. 9 .

[0061] Furthermore, in this embodiment, when the control unit 15 determines that the second operator 2 is present, the control unit 15 may cause the display 12 to display an indicator indicating the distance between the first operator 1 and the second operator 2. This indicator may show the distance between the first operator 1 and the second operator 2 in association with a distance threshold. With this configuration, as described above with reference to FIGS. 9 and 10 , the user can easily grasp to what extent the distance between the first operator 1 and the second operator 2 needs to be reduced in order to confirm an instruction for a selected object.

[0062] Another Embodiment As shown in FIG. 11 , a terminal device 110 according to another embodiment includes a display unit 11 , a storage unit 14 , a control unit 15 , and an inertial sensor 18 .

[0063] The inertial sensor 18 can detect the tilt angle of the display 12. The inertial sensor 18 may be configured to include an acceleration sensor, an angular velocity sensor, or the like, or may be configured to include an inertial measurement unit (IMU).

[0064] Here, depending on the angle at which the display 12 is installed, even if the pose of the user's finger is the same, the part of the display 12 at which capacitance is detected may differ. This will be described with reference to FIGS. 12 and 13.

[0065] 12 shows a state where the display 12 is approximately parallel to the vertical direction. When the display 12 is approximately parallel to the vertical direction, when the user points the first operator 1 toward the display 12, the distance between the tip of the first operator 1 and the display 12 is shorter than the distance between the other fingers and the display 12. Therefore, the control unit 15 can detect the capacitance corresponding only to the first operator 1 using the detection sensor 13. The circle 1A shown in FIG. 12 corresponds to the circle 1A shown in FIG. 6.

[0066] FIG. 13 shows a state where the display 12 is approximately parallel to the horizontal direction. The pose of the user's fingers shown in FIG. 13 is the same as the pose of the user's fingers shown in FIG. 12. When the display 12 is approximately parallel to the horizontal direction, when the user points the first operator 1 toward the display 12, the distance between the tip of the first operator 1 and the display 12 is approximately the same as the distance between the other fingers and the display 12. Therefore, the control unit 15 detects the capacitance corresponding to the first operator 1 and the capacitance corresponding to the other fingers. For example, the control unit 15 detects the capacitance of the portion indicated by circle 1A1 and the capacitance indicated by circle 1A2. When the detection sensor 13 detects the capacitances of two portions of the display 12 in this way, it becomes difficult for the detection sensor 13 to detect the second operator 2.

[0067] Therefore, in another embodiment, the control unit 15 detects the tilt angle of the display 12 with respect to the horizontal direction using the inertial sensor 18. The control unit 15 does not accept a non-contact operation when the tilt angle of the display 12 with respect to the horizontal direction is equal to or less than a predetermined angle. The predetermined angle may be set based on the user's usage pattern or the capacitance detection accuracy of the detection sensor 13. The predetermined angle is, for example, 30 degrees. This configuration can further reduce false detection of a non-contact operation.

[0068] 14 is a flowchart illustrating an example procedure of a notification process according to another embodiment of the present disclosure. The control unit 15 starts the process of step S21, for example, when the terminal device 10 is turned on. For example, when installation of the display 12 is completed, the user may turn on the terminal device 10.

[0069] The control unit 15 detects the tilt angle of the display 12 with respect to the horizontal direction using the inertial sensor 18 (step S21). The control unit 15 determines whether the tilt angle of the display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle (step S22).

[0070] If the control unit 15 determines that the tilt angle of the display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle (step S22: YES), the control unit 15 determines that the non-touch operation is not accepted (step S23). The control unit 15 causes the display 12 to display a notification indicating that the non-touch operation is not accepted (step S24). After processing step S24, the control unit 15 may transition the operation mode of the terminal device 110 to another mode. The other mode is, for example, a mode in which touch input to the display unit 11 is accepted by the detection sensor 13.

[0071] If the control unit 15 determines that the tilt angle of the display 12 with respect to the horizontal direction exceeds the predetermined angle (step S22: NO), the control unit 15 determines that the non-contact operation is accepted (step S25). The control unit 15 causes the display 12 to display a notification indicating that the non-contact operation is accepted (step S26). After executing the process of step S26, the control unit 15 may proceed to the process of step S1 shown in FIG. 4.

[0072] As described above, in the terminal device 110 according to another embodiment, the control unit 15 does not accept a touchless operation when the tilt angle of the display 12 with respect to the horizontal direction is equal to or less than a predetermined angle. This configuration can further reduce false detection of a touchless operation. The control unit 15 also causes the display 12 to display a notification indicating that a touchless operation is not being accepted. By displaying this notification, the user can know that the terminal device 110 does not accept a touchless operation. The user can also adjust the tilt angle of the display 12 so that the terminal device 110 accepts a touchless operation.

[0073] Other configurations and effects of the terminal device 110 according to the other embodiments are the same as or similar to those of the terminal device 10.

[0074] While the present disclosure has been described based on various drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the functions included in each functional unit can be rearranged so as not to cause logical inconsistencies. Multiple functional units may be combined into one or divided. The above-described embodiments of the present disclosure are not limited to faithful implementation of each of the described embodiments, but may be implemented by combining features or omitting some features as appropriate. In other words, those skilled in the art can make various modifications and alterations to the contents of the present disclosure based on the present disclosure. Therefore, these modifications and alterations are within the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, or each step can be added to other embodiments so as not to cause logical inconsistencies, or can be replaced with each functional unit, each means, or each step of other embodiments. Furthermore, in each embodiment, multiple functional units, each means, or each step can be combined into one or divided. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but can also be implemented by combining each feature or omitting some of them as appropriate.

[0075] For example, if the terminal device 10, 110 described above does not employ a pinch-in operation, the operation for confirming an instruction to an object may be an operation of pointing the first operator 1 and the second operator 2 in a peace sign toward the object and then closing the peace sign. The peace sign is a sign in which the first operator 1 and the second operator 2 form a V shape by widening the distance between them. In this case, the control unit 15 does not need to execute the process of step S4 shown in FIG. 4 . That is, the control unit 15 may proceed to the process of step S5 after the process of step S3.

[0076] For example, an embodiment is also possible in which a general-purpose computer functions as the terminal device 10, 110 according to the above-described embodiments. Specifically, a program describing the processing content for realizing each function of the terminal device 10, 110 according to the above-described embodiments is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor or a non-transitory computer-readable medium storing the program.

[0077] In one embodiment, (1) a terminal device includes a display and a control unit that determines a user's instruction for an object displayed on the display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0078] (2) In the terminal device described in (1) above, the information about the first operator and the second operator may be based on the capacitance of the first operator and the second operator relative to the display.

[0079] (3) In the terminal device described in (1) or (2) above, when the control unit determines that the distance between the first operator and the second operator is less than a distance threshold based on information about the first operator and information about the second operator, the control unit may confirm a user's instruction regarding an object displayed on the display.

[0080] (4) In the terminal device described in any one of (1) to (3) above, the first line segment is a line segment connecting the position of the first operator and the position of the second operator on the display, the second line segment is a line segment passing through a point on the first line segment on the display and perpendicular to the first line segment, and when the control unit determines that the change value of the capacitance on or near the second line segment on the display exceeds a change threshold, the control unit may determine that the distance between the first operator and the second operator is less than the distance threshold.

[0081] (5) In the terminal device described in (4) above, when the control unit determines that the change value of the capacitance on the first line segment or near the first line segment on the display exceeds a judgment threshold, the control unit may determine whether the change value of the capacitance on the second line segment or near the second line segment on the display exceeds a change threshold.

[0082] (6) In the terminal device described in any one of (1) to (5) above, the control unit may determine whether the second operator is present or not if the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present.

[0083] (7) In the terminal device described in any one of (3) to (6) above, when the control unit determines that the second operator is present, the control unit may determine whether the distance between the first operator and the second operator is less than the distance threshold.

[0084] (8) In the terminal device described in any one of (1) to (7) above, when the control unit determines that the second operator is present, the control unit may cause the display to display an indicator indicating the distance between the first operator and the second operator.

[0085] (9) In the terminal device described in (8) above, the indicator may indicate the distance between the first operator and the second operator in association with the distance threshold.

[0086] (10) In the terminal device described in any one of (1) to (9) above, if the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present, the control unit may display, among the multiple objects displayed on the display, an object that the first operator is pointing at in a different display mode from other objects.

[0087] (11) In the terminal device described in any one of (1) to (10) above, the control unit may determine that non-contact operation is not accepted when the tilt angle of the display relative to the horizontal direction is equal to or less than a predetermined angle, and may cause the display to display a notification indicating that non-contact operation is not accepted.

[0088] (12) In the terminal device described in any one of (1) to (11) above, the control unit may determine that non-contact operation is accepted when the tilt angle of the display relative to the horizontal direction exceeds a predetermined angle, and cause the display to display a notification indicating that non-contact operation is accepted.

[0089] (13) In the terminal device described in any one of (1) to (12) above, the first manipulator and the second manipulator may be two adjacent fingers.

[0090] (14) In the terminal device according to any one of (1) to (13), the first operator may be an index finger, and the second operator may be a middle finger.

[0091] In one embodiment, (15) the operating method includes determining a user's instruction for an object displayed on the display based on information of a first operator and information of a second operator that are in a hover state over the display.

[0092] In one embodiment, (16) the program causes a computer to perform an operation including determining a user's instruction for an object displayed on the display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0093] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In this disclosure, the configurations distinguished by descriptions such as "first" and "second" can have their numbers exchanged. For example, the first operator can exchange the identifiers "first" and "second" with the second operator. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

[0094] REFERENCE SIGNS LIST 1 First operator 2 Second operator 1A, 1A1, 1A2, 2A Circle 1B, 2B Coordinates 3 First line segment 4 Coordinates 5 Second line segment 10, 110 Terminal device 11 Display unit 12 Display 13 Detection sensor 14 Storage unit 15 Control unit 16a, 16b Object 17 Indicator 17a, 17b Circle 18 Inertial sensor

Claims

1. A terminal device comprising: a display; and a control unit that determines a user's instruction for an object displayed on the display based on information about a first operator and information about a second operator that are in a hover state over the display.

2. The terminal device according to claim 1, wherein the information about each of the first and second operators is based on the electrostatic capacitance of each of the first and second operators relative to the display.

3. A terminal device as described in claim 1 or 2, wherein the control unit, when determining that the distance between the first operator and the second operator is less than a distance threshold based on information about the first operator and information about the second operator, confirms a user's instruction to an object displayed on the display.

4. A terminal device as described in any one of claims 1 to 3, wherein a first line segment is a line segment connecting the position of the first operator and the position of the second operator on the display, a second line segment is a line segment passing through a point on the first line segment on the display and perpendicular to the first line segment, and when the control unit determines that a change value in capacitance on the second line segment or in the vicinity of the second line segment on the display exceeds a change threshold, the control unit determines that the distance between the first operator and the second operator is less than the distance threshold.

5. The terminal device described in claim 4, wherein when the control unit determines that the change value of capacitance on the first line segment or near the first line segment on the display exceeds a judgment threshold, the control unit determines whether the change value of capacitance on the second line segment or near the second line segment on the display exceeds a change threshold.

6. A terminal device described in any one of claims 1 to 5, wherein the control unit determines whether or not the second operator is present if the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present.

7. A terminal device described in any one of claims 3 to 6, wherein when the control unit determines that the second operator is present, it determines whether the distance between the first operator and the second operator is less than the distance threshold.

8. A terminal device according to any one of claims 1 to 7, wherein the control unit, when determining that the second operator is present, causes the display to display an indicator indicating the distance between the first operator and the second operator.

9. The terminal device according to claim 8, wherein the indicator indicates the distance between the first operator and the second operator in association with the distance threshold value.

10. A terminal device as described in any one of claims 1 to 9, wherein, when the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present, the control unit displays an object, among a plurality of objects displayed on the display, that is pointed at by the first operator in a display manner different from that of other objects.

11. A terminal device described in any one of claims 1 to 10, wherein the control unit determines that non-touch operation is not accepted when the tilt angle of the display with respect to the horizontal direction is equal to or less than a predetermined angle, and causes the display to display a notification indicating that non-touch operation is not accepted.

12. A terminal device described in any one of claims 1 to 11, wherein the control unit determines that non-touch operation is accepted when the tilt angle of the display with respect to the horizontal direction exceeds a predetermined angle, and causes the display to display a notification indicating that non-touch operation is accepted.

13. A terminal device according to any one of claims 1 to 12, wherein the first operator and the second operator are two adjacent fingers.

14. A terminal device according to any one of claims 1 to 13, wherein the first operator is an index finger, and the second operator is a middle finger.

15. An operating method including determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

16. A program that causes a computer to perform operations including determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

Citation Information

Patent Citations

  • Information processing method, and equipment

    CN114077356A

  • Input device

    JP2015053034A

  • Input device, input assistance method, and program

    WO2013161171A1

  • Gesture input device

    WO2015004848A1

  • Electronic device control method and apparatus, and terminal and storage medium

    WO2022052774A1