Terminal device, operation method, and program

JP2024168309APending Publication Date: 2024-12-05KYOCERA CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2023084861
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-12-05

AI Technical Summary

Benefits of technology

【0009】 本開示の一実施形態によれば、非接触操作の検出精度を向上させることができる。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168309000001_ABST
    Figure 2024168309000001_ABST
Patent Text Reader

Abstract

To provide a terminal device, an operation method, and a program configured to improve the accuracy of detecting a non-contact operation.SOLUTION: A terminal device 10 includes a display 12, a display unit 11 having a detection sensor 13 which is a capacitive touch sensor, a storage unit and a control unit. The control unit determines, on the basis of information of a first operator 1, which may be an arbitrary finger of a user, and information of a second operator 2, which is a finger adjacent to the first operator 1, in a hover state with respect to the display 12, an instruction of the user with respect to an object displayed on the display 12.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a terminal device, an operation method, and a program. [Background technology]

[0002] Conventionally, there is known a technique for detecting a non-contact operation. For example, Patent Document 1 describes a technique for identifying a moving direction of an operator and executing a process associated with the identified moving direction. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-212087 A Summary of the Invention [Problem to be solved by the invention]

[0004] It would be useful if the accuracy of detecting a non-contact operation could be improved.

[0005] In view of the above, an object of the present disclosure is to improve the detection accuracy of a non-contact operation. [Means for solving the problem]

[0006] A terminal device according to an embodiment of the present disclosure includes: A display and a control unit that determines a user's instruction for an object displayed on the display based on information on a first operator and information on a second operator that are in a hover state over the display; Equipped with.

[0007] An operating method according to an embodiment of the present disclosure includes: The method includes determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0008] A program according to an embodiment of the present disclosure includes: On the computer, Based on information about a first operator and information about a second operator that are in a hover state over a display, an operation including determining a user's instruction for an object displayed on the display is executed. Effect of the Invention

[0009] According to an embodiment of the present disclosure, it is possible to improve the detection accuracy of a non-contact operation. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is an external view of a terminal device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a top view of the display unit shown in FIG. [Diagram 3] FIG. 2 is a block diagram of a terminal device shown in FIG. [Figure 4] 13 is a flowchart illustrating an example procedure of a determination process according to an embodiment of the present disclosure. [Diagram 5] 13 is a flowchart illustrating an example procedure of a determination process according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram showing an XY coordinate system set for a display. [Figure 7] FIG. 2 is a diagram showing an XY coordinate system set for a display. [Figure 8] FIG. 2 is a diagram showing an XY coordinate system set for a display. [Figure 9] FIG. 4 is a diagram showing an example of a display mode of a display. [Figure 10] FIG. 4 is a diagram showing an example of a display mode of a display. [Figure 11]FIG. 13 is a block diagram of a terminal device according to another embodiment of the present disclosure. [Figure 12] FIG. 13 is a diagram showing a state in which the display is approximately parallel to the vertical direction. [Figure 13] FIG. 13 is a diagram showing a state in which the display is approximately parallel to the horizontal direction. [Figure 14] 13 is a flowchart illustrating an example procedure of a notification process according to another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] As shown in Fig. 1, a terminal device 10 according to the present embodiment includes a display unit 11. Various objects to be operated by a user are displayed on the display unit 11. The objects are, for example, icons, windows, toolbars, menus, buttons, sliders, etc. The terminal device 10 detects a non-contact operation by the user on the object.

[0013] The terminal device 10 may be used in any location. As an example, the terminal device 10 may be used in a location where the user's hands are dirty or wet. In this case, the terminal device 10 is used in, for example, a kitchen, a bathroom, a factory, or the like.

[0014] Hereinafter, a user operation according to this embodiment for confirming an instruction for an object displayed on the display unit 11 will be described.

[0015] First, the user points the first operator 1 in a hover state toward an object that the user wants to execute among the objects displayed on the display unit 11. The user keeps the first operator 1 in a hover state toward the object in a stationary state. The hover state is a state in which the distance between the operator and the display unit 11 is within a predetermined distance and the operator and the display unit 11 are not in contact with each other. The predetermined distance is, for example, a distance corresponding to a capacity threshold value described later. The first operator 1 is, for example, the index finger of the user's hand. However, the first operator 1 does not have to be the index finger of the user's hand. The first operator 1 may be any finger of the user as long as it is a finger that is pointed toward the object. For example, the user extends the index finger, which is the first operator 1, and grasps the other fingers, as shown in FIG. 1. The user points the object that the user wants to execute with the first operator 1, which is the index finger in a hover state.

[0016] When the user points the first operator 1 in a hover state toward an object, the object becomes selected. When the user wants to confirm an instruction to the selected object, the user places the second operator 2 in a hover state on the first operator 1 in a stationary state, as shown in FIG. 2. The second operator 2 is the finger next to the first operator 1. When the first operator 1 is the user's index finger, the second operator 2 is, for example, the user's middle finger. When the second operator 2 is placed on the first operator 1, the user's instruction to the selected object is confirmed.

[0017] 3, the terminal device 10 includes a storage unit 14 and a control unit 15 in addition to a display unit 11. The display unit 11 includes a display 12 and a detection sensor 13.

[0018] The display 12 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0019] The detection sensor 13 is capable of detecting capacitance. The detection sensor 13 is, for example, a capacitance-type touch sensor. The detection sensor 13 may be a self-capacitance type or a mutual capacitance type. The detection sensor 13 is configured to include, for example, a transparent film and a transparent electrode such as ITO (Indium Tin Oxide). The transparent electrode may have a mesh-like pattern. Alternatively, strip-shaped transparent electrodes may be arranged in a lattice pattern.

[0020] The detection sensor 13 is disposed on the display 12. By disposing the detection sensor 13 on the display 12, the detection sensor 13 can detect the capacitance of the display 12 in an XY coordinate system as shown in FIG. 6 described later. The XY coordinate system is a two-dimensional coordinate system set for the display 12. The X direction corresponds to the direction from the left side to the right side of the display 12. The Y direction corresponds to the direction from the bottom to the top of the display 12. However, the X direction and the Y direction may be set arbitrarily for the display 12. Each lattice point shown in FIG. 6 corresponds to each XY coordinate at which the capacitance is detected by the detection sensor 13 in this embodiment. This lattice point is also called a "node."

[0021] The storage unit 14 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). The storage unit 14 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores data used in the operation of the terminal device 10 and data obtained by the operation of the terminal device 10.

[0022] The control unit 15 includes at least one processor, at least one dedicated circuit, or a combination of these. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 15 executes processes related to the operation of the terminal device 10 while controlling each part of the terminal device 10.

[0023] <Confirmation process> The control unit 15 confirms a user's instruction for an object displayed on the display 12, based on information about the first operator 1 and information about the second operator 2 that are in a hover state over the display 12. An example of this confirmation process will be described below with reference to Figs. 4 and 5.

[0024] 4 and 5 are flowcharts illustrating an example of a procedure of a determination process according to an embodiment of the present disclosure.

[0025] For example, when the terminal device 10 is installed, the control unit 15 executes a calibration process (step S1). The calibration process is a process in which the detection sensor 13 detects a reference value of the capacitance at each XY coordinate of the display 12. The reference value of the capacitance at the XY coordinate is the capacitance at the XY coordinate when the user is not using the terminal device 10, that is, when the user's hand or the like is not close to the display 12. The capacitance at each XY coordinate of the display 12 changes depending on the environment around the terminal device 10. Therefore, the control unit 15 executes a calibration process to detect the reference value of the capacitance at each XY coordinate of the display 12. The control unit 15 may store the detected reference value of the capacitance at each XY coordinate of the display 12 in the storage unit 14.

[0026] When the control unit 15 executes the calibration process, the control unit 15 can detect the change in capacitance at each X and Y coordinate of the display 12 by the detection sensor 13. The change in capacitance at the X and Y coordinates is the change from a reference value of the capacitance at the X and Y coordinates.

[0027] The control unit 15 detects the change in capacitance at each XY coordinate of the display 12 by the detection sensor 13, and determines whether or not the first operator 1 is present (step S2). As an example of this process, the control unit 15 may determine that the first operator 1 is present when the change in capacitance at each of a plurality of consecutive XY coordinates exceeds a capacitance threshold. The capacitance threshold may be set based on the distance between the hovered operator that the terminal device 10 wants to detect as a processing target and the display unit 11. For example, in FIG. 6, the change in capacitance at the XY coordinates corresponding to each lattice point in the circle 1A exceeds the capacitance threshold. In FIG. 6, the control unit 15 determines that the first operator 1 is present.

[0028] If the control unit 15 determines that the first operator 1 exists (step S2: YES), the process proceeds to step S3. On the other hand, if the control unit 15 determines that the first operator 1 does not exist (step S2: NO), the control unit 15 executes the process of step S2 again.

[0029] In the process of step S3, the control unit 15 acquires the position of the first operator 1 on the display 12. As an example of this process, the control unit 15 may acquire the XY coordinates at which the change value of the capacitance is maximum among the change values ​​of the capacitance at a plurality of XY coordinates corresponding to the first operator 1, as the position of the first operator 1 on the display 12. For example, in FIG. 6, among the change values ​​of the capacitance at the XY coordinates corresponding to each lattice point in the circle 1A, the change value of the capacitance at the coordinate 1B is maximum. In FIG. 6, the control unit 15 acquires the coordinate 1B as the position of the first operator 1 on the display 12. The control unit 15 may store the acquired position of the first operator 1 and the change values ​​of the capacitance at the plurality of XY coordinates corresponding to the first operator 1 in the storage unit 14.

[0030] The control unit 15 determines whether or not the position of the first operator 1 on the display 12 has changed within a predetermined time after determining in the process of step S2 that the first operator 1 is present (step S4). The control unit 15 may determine whether or not the position of the first operator 1 has changed within the predetermined time by detecting a change in capacitance at each XY coordinate of the display 12 using the detection sensor 13. The predetermined time may be set based on the specifications of the terminal device 10, the user's usage pattern, etc.

[0031] When the control unit 15 determines that the position of the first operator 1 on the display 12 has changed within a predetermined time since determining that the first operator 1 is present (step S4: YES), the process returns to step S2. On the other hand, when the control unit 15 determines that the position of the first operator 1 on the display 12 has not changed within a predetermined time since determining that the first operator 1 is present (step S4: NO), the process proceeds to step S5.

[0032] In the process of step S5, the control unit 15 selects the object pointed at by the first operator 1 in the hover state. The control unit 15 may identify the object to be selected based on the position of the first operator 1 on the display 12. For example, in FIG. 6, the control unit 15 identifies the object to be selected based on coordinates 1B. As will be described later with reference to FIG. 9, the control unit 15 may cause the display 12 to display the selected object in a display mode different from other objects.

[0033] In the process of step S5, if there is no object in the direction in which the first operating element 1 in the hover state is pointed, the control unit 15 may return to the process of step S2.

[0034] In the process of step S6, the control unit 15 detects the change value of the capacitance at each XY coordinate of the display 12 by the detection sensor 13, and determines whether or not the second operator 2 exists. The control unit 15 may determine whether or not the second operator 2 exists within a predetermined range from the first operator 1. The predetermined range may be set based on the average distance between adjacent fingers on an open hand. In the same or similar manner as the first operator 1, the control unit 15 may determine that the second operator 2 exists when the change value of the capacitance at each of a plurality of consecutive XY coordinates exceeds the capacitance threshold. For example, in FIG. 7, the change value of the capacitance at the XY coordinates corresponding to each lattice point in the circle 2A exceeds the capacitance threshold. Also, each lattice point in the circle 2A is within a predetermined range from the coordinate 1B. The control unit 15 determines that the second operator 2 exists within the predetermined range from the first operator 1.

[0035] If the control unit 15 determines that the second operator 2 exists (step S6: YES), the process proceeds to step S7. If the control unit 15 determines that the second operator 2 exists (step S6: YES), the control unit 15 may cause the display 12 to display an indicator as described below with reference to Fig. 9. On the other hand, if the control unit 15 determines that the second operator 2 does not exist (step S6: NO), the process returns to step S3.

[0036] In the process of step S7, the control unit 15 acquires the position of the second operator 2 on the display 12. In the same manner as or similar to the first operator 1, the control unit 15 may acquire the XY coordinates at which the change value of the capacitance is maximum among the change values ​​of the capacitance at the multiple XY coordinates corresponding to the second operator 2 as the position of the second operator 2 on the display 12. For example, in FIG. 7, among the change values ​​of the capacitance at the XY coordinates corresponding to each lattice point in the circle 2A, the change value of the capacitance at the coordinate 2B is maximum. In FIG. 7, the control unit 15 acquires the coordinate 2B as the position of the second operator 2 on the display 12. The control unit 15 may store the acquired position of the second operator 2 and the change values ​​of the capacitance at the multiple XY coordinates corresponding to the second operator 2 in the storage unit 14.

[0037] In the process of step S8, the control unit 15 sets a first line segment. The first line segment is a line segment that connects the position of the first operator 1 and the position of the second operator 2 on the display 12. In other words, the first line segment is a portion between the position of the first operator 1 and the position of the second operator 2 of a straight line that passes through the position of the first operator 1 and the position of the second operator 2 on the display 12. For example, in FIG. 7, the control unit 15 sets a first line segment 3. When the control unit 15 sets the first line segment, the control unit 15 detects a change in capacitance in XY coordinates on the first line segment or in the vicinity of the first line segment by the detection sensor 13. The XY coordinates in the vicinity of the first line segment may be, for example, XY coordinates at which the capacitance can be detected by the detection sensor 13 and may be the XY coordinates closest to the first line segment. For example, in FIG. 7, the control unit 15 detects a change in capacitance in XY coordinates on the first line segment 3 or in the vicinity of the first line segment 3.

[0038] In the process of step S9, the control unit 15 judges whether or not the change value of the capacitance in the XY coordinates on the first line segment or in the vicinity of the first line segment detected in the process of step S8 exceeds the judgment threshold. The judgment threshold may be set based on the change value of the capacitance between two adjacent fingers when the user points the two fingers toward the display 12. The control unit 15 may judge whether or not all the change values ​​of the capacitance in the XY coordinates on the first line segment or in the vicinity of the first line segment exceed the judgment threshold. Alternatively, the control unit 15 may judge whether or not the minimum value or the representative value of the change values ​​of the capacitance in the XY coordinates on the first line segment or in the vicinity of the first line segment exceeds the judgment threshold. The representative value is, for example, the average value, the mode value, the median value, or the like.

[0039] When the control unit 15 determines that the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold (step S9: YES), the control unit 15 proceeds to the process of step S11 shown in Fig. 5. When the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold, there is a high possibility that the second operator 2 is being brought closer to the first operator 1.

[0040] On the other hand, when the control unit 15 determines that the change in capacitance in the XY coordinates on or near the first line segment is equal to or less than the determination threshold (step S9: NO), the control unit 15 proceeds to the process of step S10. When the change in capacitance in the XY coordinates on or near the first line segment exceeds the determination threshold, it is unlikely that the second operator 2 is being brought closer to the first operator 1.

[0041] In the process of step S10, the control unit 15 detects the change value of the capacitance at each XY coordinate of the display 12 by the detection sensor 13, and determines whether or not the second operator 2 is present. The control unit 15 may execute the process of step S10 in the same manner as or similar to the process of step S6. If the control unit 15 determines that the second operator 2 is present (step S10: YES), the control unit 15 returns to the process of step S7. On the other hand, if the control unit 15 determines that the second operator 2 is not present (step S10: NO), the control unit 15 returns to the process of step S3.

[0042] In the process of step S11 shown in FIG. 5, the control unit 15 detects a change in capacitance in the XY coordinates on the first line segment or in the vicinity of the first line segment by the detection sensor 13 in the same or similar manner as the process of step S8.

[0043] In the process of step S12, the control unit 15 selects target coordinates from among the coordinates on the first line segment or in the vicinity of the first line segment based on the change in capacitance detected in the process of step S11. The target coordinates are used to set a second line segment, which will be described later. The control unit 15 may select any coordinates from among the XY coordinates on the first line segment or in the vicinity of the first line segment as the target coordinates. As an example, the control unit 15 may select, as the target coordinates, the coordinates at which the change in capacitance is the smallest among the XY coordinates on the first line segment or in the vicinity of the first line segment. For example, in FIG. 7, the control unit 15 selects the coordinate 4 on the first line segment 3 as the target coordinate.

[0044] In the process of step S13, the control unit 15 sets a second line segment. The second line segment is a line segment that passes through the target coordinates and is perpendicular to the first line segment. For example, in FIG. 7, the control unit 15 sets a second line segment 5. The length from the target coordinates to one end of the second line segment and the length from the target coordinates to the other end of the second line segment may be set based on the thickness of the user's finger, etc. The control unit 15 detects a change in capacitance at the XY coordinates on the second line segment or in the vicinity of the second line segment by the detection sensor 13. For example, in FIG. 7, the control unit 15 detects a change in capacitance at the XY coordinates on the second line segment 5 or in the vicinity of the second line segment 5 by the detection sensor 13.

[0045] In the process of step S14, the control unit 15 determines whether or not the change value of the capacitance in the XY coordinates on or near the second line segment detected in the process of step S13 exceeds the change threshold. The change threshold may be set based on the change value of the capacitance between two adjacent fingers when the user aligns the two fingers and points them toward the display 12. The control unit 15 may determine whether or not all of the change values ​​of the capacitance in the XY coordinates on or near the second line segment exceed the change threshold. Alternatively, the control unit 15 may determine whether or not the minimum or representative value of the change values ​​of the capacitance in the XY coordinates on or near the second line segment exceeds the change threshold. The representative value is, for example, the average value, the mode value, or the median value. For example, in FIG. 8, the control unit 15 determines that the change value of the capacitance in the XY coordinates on or near the second line segment 5 exceeds the change threshold.

[0046] When the control unit 15 determines that the change in capacitance in the XY coordinates on or near the second line segment exceeds the change threshold (step S14: YES), the control unit 15 proceeds to the process of step S15. On the other hand, when the control unit 15 determines that the change in capacitance in the XY coordinates on or near the second line segment is equal to or smaller than the change threshold (step S14: NO), the control unit 15 returns to the process of step S13.

[0047] In the process of step S15, the control unit 15 confirms the user's instruction for the selected object. The control unit 15 may generate a control signal so that a process corresponding to the object for which the instruction has been confirmed is executed. The control unit 15 may transmit the generated control signal to other components of the terminal device 10.

[0048] Through the processes of steps S14 and S15, when the control unit 15 determines that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, it can confirm the user's instruction to the object displayed on the display 12. The change threshold can also be called a distance threshold.

[0049] Here, when the control unit 15 determines that the second operator 2 exists within a predetermined range from the first operator 1 (step S6: YES), the control unit 15 may detect a change in capacitance on the display 12 before and after determining that the second operator 2 exists by using the detection sensor 13. The control unit 15 may calculate an area where the change in capacitance on the display 12 exceeds a capacitance threshold before and after determining that the second operator 2 exists. The control unit 15 may calculate a difference between an area where the change in capacitance on the display 12 exceeds a capacitance threshold before determining that the second operator 2 exists and an area where the change in capacitance on the display 12 exceeds a capacitance threshold after determining that the second operator 2 exists. When the calculated difference exceeds a predetermined value, the control unit 15 may proceed to the process of step S7. The predetermined value may be set based on an area where the change in capacitance on the display 12 by one finger exceeds a capacitance threshold. Here, an example of an operation using two operators is a pinch-in operation. The pinch-in operation is an operation in which the distance between the two operators is narrowed by bringing the two operators closer to each other. Therefore, in a pinch-in operation, the area in which the change in capacitance on the display 12 exceeds the capacitance threshold value does not change before and after it is determined that one operator is within a predetermined range from the other operator. By proceeding to the process of step S7 when the difference calculated as described above exceeds the predetermined value, the operation of confirming an instruction to an object can be more clearly distinguished from a pinch-in operation.

[0050] During the execution of the processes of steps S1 to S15, the control unit 15 may detect the contact of the user's hand or the like with the display unit 11 by the detection sensor 13. In this case, the control unit 15 may cause a notice to be displayed on the display 12 to call attention. As an example, the control unit 15 may cause text information saying "Do not touch" to be displayed on the display 12. By displaying the notice to be displayed on the display 12, the user can move his / her hand or the like away from the display unit 11. With this configuration, it is possible to avoid a situation in which the first operator 1 and the second operator 2 in the hover state cannot be detected.

[0051] <Display processing> The control unit 15 may display the selected object from among the multiple objects displayed on the display 12 in a display mode different from the other objects. As described above, the control unit 15 selects the object at which the first operator 1 in the hover state is pointed in the process of step S5. The control unit 15 may display the selected object in a display mode different from the other objects in the process of step S5.

[0052] For example, objects 16a, 16b, 16c, 16d, 16e, and 16f are displayed on the display 12 shown in FIG. 9. The objects 16a to 16f are menus. In FIG. 9, the first operator 1 in a hover state is directed toward the object 16b. The control unit 15 places the object 16b in a selected state by the process of step S5. The control unit 15 displays the selected object 16b in a display mode different from other objects 16a and the like. In FIG. 9, the control unit 15 displays the object 16b on the display 12 in a mode in which the object 16b is surrounded by a frame. With this configuration, the user can easily recognize the selected object 16b.

[0053] When it is determined that the second operator 2 is present within a predetermined range from the first operator 1 as described above (step S6: YES), the control unit 15 may display an indicator on the display 12. This indicator indicates the distance between the first operator 1 and the second operator 2. This indicator may indicate the distance between the first operator and the second operator in association with a distance threshold. The control unit 15 may calculate the distance between the first operator 1 and the second operator 2 based on the positions of the first operator 1 and the second operator 2 on the display 12. The control unit 15 may display the indicator near the selected object, or may display the indicator superimposed on the selected object.

[0054] For example, an indicator 17 is displayed on the display 12 shown in FIG. 9. The indicator 17 is displayed near the selected object 16b. The indicator 17 includes circles 17a and 17b. The radius of the circle 17a corresponds to the distance between the first operator 1 and the second operator 2. The shorter the distance between the first operator 1 and the second operator 2, the larger the radius of the circle 17a. The radius of the circle 17b corresponds to the distance threshold. The position of the center of the circle 17a is the same as the position of the center of the circle 17b. That is, the circles 17a and 17b are concentric. When the radius of the circle 17a becomes larger than the radius of the circle 17b, the user's instruction to the selected object 16b is confirmed. FIG. 10 shows the indicator 17 when the user's instruction to the object 16b is confirmed. In FIG. 10, the radius of the circle 17a is larger than the radius of the circle 17b. By displaying such an indicator 17, the user can easily understand to what extent the distance between the first operator 1 and the second operator 2 needs to be reduced in order for the user's instruction to be confirmed with respect to the selected object 16b.

[0055] In this manner, in the terminal device 10 according to this embodiment, the control unit 15 confirms a user's instruction for an object displayed on the display based on information about the first operator 1 and information about the second operator 2 that are in a hover state over the display 12.

[0056] As a comparative example, it is assumed that a user's instruction to a selected object is finalized by moving the first operator 1 closer to or farther away from the selected object. The user's hand is prone to fluctuating in the air. Therefore, it is difficult to maintain a constant distance between the first operator 1 and the display unit 11. In other words, the first operator 1 may be moved closer to or farther away from the selected object due to unintended fluctuating of the user's hand. In this case, in the comparative example, the instruction to the selected object may be finalized due to unintended fluctuating of the user's hand.

[0057] In contrast to such a comparative example, the control unit 15 according to this embodiment confirms a user's instruction to an object displayed on the display 12 based on information on the first operator 1 and information on the second operator 2. That is, in this embodiment, an instruction to an object is confirmed using information on the first operator 1 and the second operator 2, rather than only information on the first operator 1. By using information on the first operator 1 and information on the second operator 2 in this manner, this embodiment can avoid a situation in which an instruction to an object is confirmed due to an unintended shaking of the user's hand, as in the comparative example.

[0058] Therefore, according to this embodiment, it is possible to improve the detection accuracy of a non-contact operation.

[0059] Furthermore, in this embodiment, when the control unit 15 determines that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, the control unit 15 may confirm the user's instruction to the object displayed on the display 12. Here, an example of an operation using two operators is a pinch-out operation. The pinch-out operation is an operation in which the two operators are moved away from each other to increase the distance between the two operators. By confirming an instruction to an object when it is determined that the distance between the first operator 1 and the second operator 2 is less than the distance threshold, the operation of confirming an instruction to an object is more clearly distinguished from a pinch-out operation. Thus, the instruction to an object can be confirmed with high accuracy.

[0060] Furthermore, in this embodiment, the control unit 15 may determine whether or not the second operator 2 is present when the position of the first operator 1 on the display 12 does not change for a predetermined time after determining that the first operator 1 is present. Here, as described above, an example of an operation using two operators is the pinch-in operation. In this pinch-in operation, the two operators are brought closer to each other, so that the positions of each of the two operators change. By determining whether or not the second operator 2 is present when the position of the first operator 1 does not change for a predetermined time, the operation of confirming an instruction to an object can be more clearly distinguished from the pinch-in operation.

[0061] Furthermore, in this embodiment, when the position of the first operator 1 on the display 12 does not change for a predetermined time, the control unit 15 may select an object, of the multiple objects displayed on the display 12, at which the first operator is pointed. The control unit 15 may display the selected object in a display mode different from other objects. With this configuration, the user can easily recognize the selected object, as described above with reference to FIG. 9.

[0062] Furthermore, in this embodiment, when the control unit 15 determines that the second operator 2 is present, the control unit 15 may cause the display 12 to display an indicator indicating the distance between the first operator 1 and the second operator 2. This indicator may indicate the distance between the first operator 1 and the second operator 2 in association with a distance threshold. With this configuration, as described above with reference to Figs. 9 and 10, the user can easily grasp to what extent the distance between the first operator 1 and the second operator 2 needs to be reduced in order to confirm an instruction to an object in a selected state.

[0063] (Other embodiments) As shown in FIG. 11, a terminal device 110 according to another embodiment includes a display unit 11, a storage unit 14, a control unit 15, and an inertial sensor 18.

[0064] The inertial sensor 18 is capable of detecting the tilt angle of the display 12. The inertial sensor 18 may be configured to include an acceleration sensor or an angular velocity sensor, or may be configured to include an inertial measurement unit (IMU).

[0065] Here, even if the pose of the user's finger is the same, depending on the angle at which the display 12 is installed, the part of the display 12 at which the capacitance is detected may differ. This will be described with reference to Figs. 12 and 13.

[0066] 12 shows a state where the display 12 is approximately parallel to the vertical direction. When the display 12 is approximately parallel to the vertical direction, when the user points the first operator 1 toward the display 12, the distance between the tip of the first operator 1 and the display 12 is shorter than the distance between the other fingers and the display 12. Therefore, the control unit 15 can detect the capacitance corresponding only to the first operator 1 by the detection sensor 13. The circle 1A shown in FIG. 12 corresponds to the circle 1A shown in FIG. 6.

[0067] FIG. 13 shows a state where the display 12 is approximately parallel to the horizontal direction. The pose of the user's fingers shown in FIG. 13 is the same as the pose of the user's fingers shown in FIG. 12. When the display 12 is approximately parallel to the horizontal direction, when the user points the first operator 1 toward the display 12, the distance between the tip of the first operator 1 and the display 12 and the distance between the other fingers and the display 12 are approximately the same. Therefore, the control unit 15 detects the capacitance corresponding to the first operator 1 and the capacitance corresponding to the other fingers. For example, the control unit 15 detects the capacitance of the part indicated by the circle 1A1 and the capacitance indicated by the circle 1A2. When the capacitances of the two parts of the display 12 are detected by the detection sensor 13 in this way, it becomes difficult for the detection sensor 13 to detect the second operator 2.

[0068] Therefore, in another embodiment, the control unit 15 detects the tilt angle of the display 12 with respect to the horizontal direction by the inertial sensor 18. When the tilt angle of the display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle, the control unit 15 does not accept the non-contact operation. The predetermined angle may be set based on the usage mode of the user or the detection accuracy of the capacitance of the detection sensor 13. The predetermined angle is, for example, 30 degrees. With such a configuration, it is possible to further reduce erroneous detection of the non-contact operation.

[0069] 14 is a flowchart showing an example of a procedure of a notification process according to another embodiment of the present disclosure. For example, when the power of the terminal device 10 is turned on, the control unit 15 starts the process of step S21. For example, when the installation of the display 12 is completed, the user may turn on the power of the terminal device 10.

[0070] Control unit 15 detects the tilt angle of display 12 with respect to the horizontal direction by inertial sensor 18 (step S21). Control unit 15 determines whether the tilt angle of display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle (step S22).

[0071] When the control unit 15 determines that the tilt angle of the display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle (step S22: YES), it determines that the non-touch operation is not accepted (step S23). The control unit 15 causes the display 12 to display a notification indicating that the non-touch operation is not accepted (step S24). After the process of step S24, the control unit 15 may transition the operation mode of the terminal device 110 to another mode. The other mode is, for example, a mode in which a touch input to the display unit 11 is accepted by the detection sensor 13.

[0072] When the control unit 15 determines that the tilt angle of the display 12 with respect to the horizontal direction exceeds a predetermined angle (step S22: NO), the control unit 15 determines that the non-touch operation is to be accepted (step S25). The control unit 15 causes the display 12 to display a notice indicating that the non-touch operation is accepted (step S26). After executing the process of step S26, the control unit 15 may proceed to the process of step S1 shown in FIG. 4.

[0073] In this manner, in the terminal device 110 according to another embodiment, the control unit 15 does not accept a non-touch operation when the tilt angle of the display 12 with respect to the horizontal direction is equal to or smaller than a predetermined angle. With such a configuration, it is possible to further reduce erroneous detection of a non-touch operation. Furthermore, the control unit 15 causes the display 12 to display a notification indicating that a non-touch operation is not accepted. By displaying such a notification, the user can know that the terminal device 110 does not accept a non-touch operation. Furthermore, the user can adjust the tilt angle of the display 12 so that the terminal device 110 accepts a non-touch operation.

[0074] Other configurations and effects of the terminal device 110 according to the other embodiments are the same as or similar to those of the terminal device 10.

[0075] Although the present disclosure has been described based on the drawings and examples, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions, etc. included in each functional unit can be rearranged so as not to be logically inconsistent. Multiple functional units, etc. may be combined into one or divided. Each embodiment according to the present disclosure described above is not limited to being implemented faithfully to each of the embodiments described respectively, and may be implemented by combining each feature as appropriate or omitting a part. In other words, the contents of the present disclosure can be modified and corrected in various ways by a person skilled in the art based on the present disclosure. Therefore, these modifications and corrections are included in the scope of the present disclosure. For example, in each embodiment, each functional unit, each means, each step, etc. can be added to other embodiments so as not to be logically inconsistent, or replaced with each functional unit, each means, each step, etc. of other embodiments. In addition, in each embodiment, multiple functional units, each means, each step, etc. can be combined into one or divided. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but may be implemented by combining each feature or omitting some features as appropriate.

[0076] For example, if the pinch-in operation is not adopted in the above-mentioned terminal devices 10, 110, the operation for confirming an instruction to an object may be an operation of pointing the first and second operators 1 and 2 in a peace sign toward the object and then closing the peace sign. The peace sign is a sign in which the first and second operators 1 and 2 are made into a V shape by widening the distance between them. In this case, the control unit 15 does not need to execute the process of step S4 shown in FIG. 4. That is, the control unit 15 may proceed to the process of step S5 after the process of step S3.

[0077] For example, an embodiment is also possible in which a general-purpose computer functions as the terminal device 10, 110 according to the above-described embodiment. Specifically, a program describing the processing contents for realizing each function of the terminal device 10, 110 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor, or a non-transitory computer-readable medium that stores the program.

[0078] In one embodiment, (1) a terminal device includes: A display and a control unit that determines a user's instruction for an object displayed on the display based on information on a first operator and information on a second operator that are in a hover state over the display; Equipped with.

[0079] (2) In the terminal device according to (1), The information about each of the first and second operators may be based on electrostatic capacitance of each of the first and second operators with respect to the display.

[0080] (3) In the terminal device according to (1) or (2) above, When the control unit determines that the distance between the first operator and the second operator is less than a distance threshold based on information about the first operator and information about the second operator, it may confirm a user's instruction regarding an object displayed on the display.

[0081] (4) In the terminal device according to any one of (1) to (3) above, the first line segment is a line segment that connects a position of the first operator and a position of the second operator on the display; the second line segment is a line segment that passes through a point on the first line segment on the display and is perpendicular to the first line segment; When the control unit determines that the change in capacitance on the second line segment or near the second line segment on the display exceeds a change threshold, it may determine that the distance between the first operator and the second operator is less than the distance threshold.

[0082] (5) In the terminal device according to (4) above, When the control unit determines that the change value of capacitance on the first line segment or in the vicinity of the first line segment on the display exceeds a judgment threshold, it may determine whether the change value of capacitance on the second line segment or in the vicinity of the second line segment on the display exceeds a change threshold.

[0083] (6) In the terminal device according to any one of (1) to (5) above, The control unit may determine whether or not the second operator is present when a position of the first operator on the display does not change for a predetermined time after determining that the first operator is present.

[0084] (7) In the terminal device according to any one of (3) to (6) above, When determining that the second operator is present, the control unit may determine whether or not a distance between the first operator and the second operator is less than the distance threshold value.

[0085] (8) In the terminal device according to any one of (1) to (7) above, When it is determined that the second operator is present, the control unit may cause the display to display an indicator indicating a distance between the first operator and the second operator.

[0086] (9) In the terminal device according to (8), The indicator may indicate a distance between the first operator and the second operator in association with the distance threshold.

[0087] (10) In the terminal device according to any one of (1) to (9), If the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present, the control unit may display an object toward which the first operator is pointed, among a plurality of objects displayed on the display, in a display manner different from that of other objects.

[0088] (11) In the terminal device according to any one of (1) to (10), The control unit may determine that a non-touch operation is not accepted when a tilt angle of the display with respect to the horizontal direction is equal to or smaller than a predetermined angle, and cause the display to display a notification indicating that a non-touch operation is not accepted.

[0089] (12) In the terminal device according to any one of (1) to (11), The control unit may determine that a non-touch operation is accepted when a tilt angle of the display with respect to the horizontal direction exceeds a predetermined angle, and cause the display to display a notification indicating that the non-touch operation is accepted.

[0090] (13) In the terminal device according to any one of (1) to (12), The first manipulator and the second manipulator may be two adjacent fingers.

[0091] (14) In the terminal device according to any one of (1) to (13), the first operator is an index finger, The second operator may be a middle finger.

[0092] In one embodiment, the method of (15) comprises the steps of: The method includes determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

[0093] In one embodiment, the program (16) comprises: On the computer, Based on information about a first operator and information about a second operator that are in a hover state over a display, an operation including determining a user's instruction for an object displayed on the display is executed.

[0094] In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may have their numbers exchanged. For example, a first operator may exchange identifiers "first" and "second" with a second operator. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configurations or to justify the existence of identifiers with smaller numbers. [Explanation of symbols]

[0095] 1 1st controller 2 2nd controller 1A,1A1,1A2,2A Yen 1B,2B coordinates 3 First line segment 4. Coordinates 5 Second line segment 10,110 Terminal Equipment 11 Display section 12 Display 13 Detection sensor 14 Storage section 15 Control section 16a,16b Objects 17 Indicators 17a,17b yen 18 Inertial Sensors

Claims

1. A display and a control unit that determines a user's instruction for an object displayed on the display based on information on a first operator and information on a second operator that are in a hover state over the display; A terminal device comprising:

2. The terminal device according to claim 1 , wherein the information about each of the first and second operators is based on an electrostatic capacitance of each of the first and second operators with respect to the display.

3. The terminal device according to claim 1 , wherein the control unit, when determining that the distance between the first operator and the second operator is less than a distance threshold based on information about the first operator and information about the second operator, confirms a user's instruction for an object displayed on the display.

4. the first line segment is a line segment that connects a position of the first operator and a position of the second operator on the display; the second line segment is a line segment that passes through a point on the first line segment on the display and is perpendicular to the first line segment; The terminal device according to claim 3, wherein the control unit determines that the distance between the first operator and the second operator is less than the distance threshold when it determines that the change value of the capacitance on the second line segment or near the second line segment on the display exceeds a change threshold.

5. The terminal device of claim 4, wherein when the control unit determines that the change value of capacitance on the first line segment or near the first line segment on the display exceeds a judgment threshold, the control unit determines whether the change value of capacitance on the second line segment or near the second line segment on the display exceeds a change threshold.

6. The terminal device according to claim 3 , wherein the control unit determines whether or not the second operator is present when a position of the first operator on the display does not change for a predetermined time after determining that the first operator is present.

7. The terminal device according to claim 6 , wherein the control unit, when determining that the second operator is present, determines whether or not a distance between the first operator and the second operator is less than the distance threshold value.

8. The terminal device according to claim 6 , wherein the control unit, when determining that the second operator is present, causes the display to display an indicator indicating a distance between the first operator and the second operator.

9. The terminal device according to claim 8 , wherein the indicator indicates a distance between the first operator and the second operator in association with the distance threshold.

10. 4. The terminal device according to claim 3, wherein when the position of the first operator on the display does not change for a predetermined time after determining that the first operator is present, the control unit displays an object, among the multiple objects displayed on the display, that is pointed at by the first operator in a display manner different from other objects.

11. The terminal device according to claim 1 , wherein the control unit determines that non-touch operation is not accepted when a tilt angle of the display with respect to the horizontal direction is equal to or smaller than a predetermined angle, and causes the display to display a notification indicating that non-touch operation is not accepted.

12. The terminal device according to claim 1 , wherein the control unit determines that a non-touch operation is to be accepted when a tilt angle of the display with respect to a horizontal direction exceeds a predetermined angle, and causes the display to display a notification indicating that the non-touch operation is to be accepted.

13. The terminal device according to claim 1 , wherein the first operator and the second operator are two adjacent fingers.

14. the first operator is an index finger, The terminal device according to claim 13 , wherein the second operator is a middle finger.

15. An operating method including determining a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

16. On the computer, A program that executes an operation including confirming a user's instruction for an object displayed on a display based on information about a first operator and information about a second operator that are in a hover state over the display.

Citation Information

Patent Citations

  • Information processing method, and equipment

    CN114077356A

  • Input device

    JP2015053034A

  • Device, method and graphical user interface for interacting with a three-dimensional environment

    JP2022547930A

  • Input device, input assistance method, and program

    WO2013161171A1

  • Gesture input device

    WO2015004848A1