Input interface control method and system

Abstract

The invention discloses an input interface control method and system. The input interface control method includes: transmitting, by a radar unit, a transmit wave toward an identification space; receiving the reflected wave through a radar unit and obtaining a radar point cloud; capturing an image of the identification space through the camera unit; and identifying a target point from the radar point cloud and the captured image, wherein a target position of the target point is used as an input interface. Therefore, the accuracy of the input interface can be improved.

Description

Technical Field

[0001] This invention relates to an input interface control method and system, and more particularly to an input interface control method and system using a radar unit and a camera unit. Background Technology

[0002] With the increasing popularity of working from home and remote learning, the demand for portable electronic devices has grown significantly. However, the additional peripherals and control interfaces (such as mice) of portable electronic devices are still inconvenient to carry or use.

[0003] Furthermore, while there are various input control interfaces available on the market to replace the mouse, such as touchpads, pointing sticks, and visual virtual mice, they still have their limitations and drawbacks. Touchpads and pointing sticks need to be used within a specific small area, and prolonged use can strain the wrist. Visual virtual mice require an additional image recognition device placed on the front or back of the mouse and require specific gestures for assistance, forcing users to learn the relevant operating rules and adding another layer of burden.

[0004] Based on the above, in today's market for input control interfaces of portable electronic devices, there is an urgent need to develop a convenient, accurate input interface control method and system that reduces physical burden and increases user willingness. Summary of the Invention

[0005] This invention provides an input interface control method and system, which identifies target points by using radar point clouds and captured images, and uses the target position of the target point as the input interface. This allows for the convergence of point cloud data generated by random radar reflection waves through high-precision single-point visual identification, and solves the problem of visual identification being limited to a single plane by utilizing the multi-directionality of radar. This helps to improve the accuracy and identification efficiency of the input interface.

[0006] According to an embodiment of the present invention, an input interface control method is provided, comprising: transmitting a transmitted wave toward an identification space through a radar unit; receiving a reflected wave and obtaining a radar point cloud through a radar unit; capturing an image of the identification space through a camera unit; and identifying a target point from the radar point cloud and the captured image, wherein the target position of the target point is used as an input interface.

[0007] According to another embodiment of the present invention, an input interface control system is provided, comprising at least one control unit, a radar unit, and a camera unit. The control unit includes at least one processor and at least one storage medium, the storage medium including an input interface control module. The radar unit is communicatively connected to the at least one control unit. The camera unit is communicatively connected to the at least one control unit. Based on the input interface control module, the control unit is configured to: transmit a transmitted wave toward an identification space via the radar unit; receive reflected waves via the radar unit and obtain a radar point cloud; capture an image of the identification space via the camera unit; and identify a target point from the radar point cloud and the captured image, the target position of the target point being used as an input interface. Attached Figure Description

[0008] Figure 1 A flowchart illustrating the input interface control method of the first embodiment of the present invention is shown;

[0009] Figure 2A A block diagram illustrating the input interface control system according to a second embodiment of the present invention is shown;

[0010] Figure 2B Draw Figure 2A A schematic diagram showing the usage status of the input interface control system;

[0011] Figure 2C Draw Figure 2A Another schematic diagram illustrating the usage status of the input interface control system;

[0012] Figure 2D Draw Figure 2A A schematic diagram of the antenna configuration of the radar unit in the input interface control system; and

[0013] Figure 2E Draw Figure 2A A schematic diagram of another antenna configuration for the radar unit of the input interface control system.

[0014] Explanation of key component symbols:

[0015] 100 Input Interface Control Method

[0016] 200 Input Interface Control System

[0017] 210 Control Unit

[0018] 212 processor

[0019] 214 Storage Media

[0020] 216 Input Interface Control Module

[0021] 220 display screen

[0022] 222 cursor

[0023] 230 camera units

[0024] 240 radar units

[0025] 242 Antenna Plane

[0026] Transmitting antennas 251, 252, 261, and 262

[0027] Receiving antennas 256, 257, 258, 266, 267, 268

[0028] 300 electronic devices

[0029] 350 Keyboard

[0030] 360 Pivot

[0031] 700 reference plane

[0032] 800 radar point cloud

[0033] Target point 820

[0034] 900 users

[0035] 910 Hands

[0036] 920 fingers

[0037] a2, a3 center-to-center distance

[0038] d2 distance

[0039] s1 Recognition Space

[0040] x1 First direction

[0041] y1 Second direction

[0042] z1 Third direction

[0043] 110, 112, 120, 122, 124, 130, 140, 142, 150, 160, 170: Steps Detailed Implementation

[0044] Several embodiments of the present invention will now be described with reference to the accompanying drawings. For clarity, many practical details will be set forth in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, these practical details are not essential in the embodiments of the present invention. Furthermore, for the sake of simplicity in the drawings, some well-known and customary structures and elements will be illustrated in a simple schematic manner; and repeated elements may be denoted by the same reference numerals.

[0045] Furthermore, the terms "first" and "second" are used only to describe different components and do not restrict the components themselves. Therefore, the first component can also be referred to as the second component. Moreover, the combination of components in this article is not a combination that is generally known, conventional, or publicly known in this field. Whether the components themselves are publicly known cannot be used to determine whether their combination relationship is easily accomplished by a person skilled in the art.

[0046] Figure 1 A flowchart illustrating the input interface control method 100 according to the first embodiment of the present invention is shown. Figure 2A A block diagram illustrating the input interface control system 200 according to a second embodiment of the present invention is shown. Figure 2B Draw Figure 2A A schematic diagram showing the usage status of the input interface control system 200. Please refer to... Figure 1 , Figure 2A as well as Figure 2B The present invention is illustrated by the input interface control method 100 of the first embodiment and the input interface control system 200 of the second embodiment, and it is understood that the implementation of the input interface control method 100 of the present invention is not limited to the input interface control system 200, and the input interface control system 200 is not limited to using the input interface control method 100.

[0047] The input interface control method 100 of the first embodiment includes steps 110, 112, 120, and 122. Step 110 includes transmitting a transmitted wave towards the identification space s1 through the radar unit 240, and step 112 includes receiving the reflected wave through the radar unit 240 and obtaining a radar point cloud 800. Step 120 includes capturing an image of the identification space s1 through the camera unit 230. Step 122 includes identifying a target point 820 from the radar point cloud 800 and the captured image, wherein the target position of the target point 820 is used as the input interface. Thus, the input interface control method 100 combines the advantages of high precision and high stability of visual recognition with the advantages of multi-directionality and high-speed movement of radar, compensating for the shortcomings of using either alone, thereby improving the accuracy of the input interface.

[0048] The input interface control system 200 of the second embodiment includes at least one control unit 210, a radar unit 240, and a camera unit 230. The control unit 210 includes at least one processor 212 and at least one storage medium 214. The storage medium 214 includes an input interface control module 216, wherein the storage medium 214 is specifically a non-transitory computer-readable storage medium, and the input interface control module 216 is specifically program code. The radar unit 240 is communicatively connected to the control unit 210, and the camera unit 230 is communicatively connected to the control unit 210. Based on the input interface control module 216, the control unit 210 executes steps 110, 112, 120, and 122 of the input interface control method 100. Herein, by visually identifying high-precision single points to converge point cloud data generated by random radar reflections, and by utilizing the multi-directionality of radar to solve the problem of visual identification being limited to a single plane, the accuracy and identification efficiency of the input interface are improved.

[0049] In detail, steps 110 and 112 can precede step 120, meaning that obtaining the radar point cloud 800 can precede capturing the image. Thus, from the received radar reflected wave signal, multiple radar measurement points are obtained through the control unit 210. The radar measurement points and the last frame of image acquired by the camera unit 230 are processed to obtain the required radar point data. In other words, the input interface control system 200 combines three-dimensional point cloud data and two-dimensional image data, using the image data to determine the fingertip position as a single point to overlap and converge the radar point cloud data, thereby increasing the positional accuracy of the target point 820 in the three-dimensional radar detection space. When the input interface control system 200 replaces the mouse, the trajectory and direction of the cursor 222 become smoother. According to other embodiments of the present invention, obtaining the radar point cloud can be simultaneous with or later than capturing the image.

[0050] Figure 2C Draw Figure 2A Another schematic diagram showing the usage status of the input interface control system 200. Please refer to... Figure 1 , Figures 2A to 2C The input interface control method 100 may further include steps 124 and 130. Step 124 includes identifying the reference surface 700 from the radar point cloud 800 and the captured image, and step 130 includes calculating the distance d2 between the target point 820 and the reference surface 700. In this way, the input interface control method 100 can accurately obtain the required radar point data and then determine whether the target point 820 is on the reference surface 700 through calculation. Furthermore, it should be understood that... Figure 2B and Figure 2C It is a schematic diagram of the radar point cloud 800 obtained by combining the actual physical configuration with the radar unit 240 and the control unit 210.

[0051] Specifically, if the target point 820 has been identified from the radar point cloud 800 and the captured image in step 122, and the reference surface 700 has been identified from the radar point cloud 800 and the captured image in step 124, then the distance d2 between the target point 820 and the reference surface 700 is calculated in step 130. Conversely, if at least one of the target point 820 and the reference surface 700 cannot be identified from the radar point cloud 800 and the captured image in steps 122 and 124, then steps 110, 112, 120, 122, and 124 are returned to be executed until the target point 820 and the reference surface 700 can be identified from the radar point cloud 800 and the captured image in steps 122 and 124.

[0052] The input interface control method 100 may further include steps 140, 142, and 150. The input interface control system 200 may further include a display screen 220, which is communicatively connected to a control unit 210. Step 140 includes determining whether the distance d2 is less than or equal to a distance threshold. Step 142 includes calculating the cursor position on the display screen 220 of the electronic device 300 corresponding to the target position when the distance d2 is less than or equal to the distance threshold. Step 150 includes moving the cursor 222 on the display screen 220 to the cursor position. Therefore, actual point cloud data is prone to excessive divergence due to environmental and object movement influences, and its accuracy remains limited even through calculation and convergence. The input interface control method 100 obtains radar point cloud 800 and then captures images to assist in identification. It converts the three-dimensional radar signal into a two-dimensional plane signal, and the two-dimensional plane signal into the cursor position on the display screen 220. Based on the movement amount calculated by the system, it then operates the cursor 222 to move on the display screen 220, which can improve the identification accuracy of the target point 820 to within 0.5cm, thus effectively replacing the mouse.

[0053] The radar point cloud 800 may include the point cloud of the user's hand 910, the captured image may include the image of the hand 910, and the target point 820 may be the fingertip of the finger 920 of the hand 910. This helps to improve the user's willingness and convenience of using the input interface control system 200.

[0054] The reference surface 700 can be an operating plane, a supporting surface, or a contact surface, such as a desktop, and the distance threshold can be between 0.5cm and 7cm. In this way, the position of the finger 920 is identified by superimposing the radar reflection wave signal with the image. Only the radar reflection wave signal within a specific range on the fingertip of the finger 920 is used as the basis for moving the cursor 222. This effectively utilizes the image range to reduce the three-dimensional reflection wave error caused by environmental factors, movements, speed, etc., for example, reducing the position error from more than 2cm to 0.5cm, thereby increasing accuracy and accurately determining whether the fingertip has left the operating plane.

[0055] The input interface control method 100 may further include step 160. Step 160 includes determining whether the radar point cloud 800 corresponds to a specified gesture when the distance d2 is greater than a distance threshold, that is, when the distance d2 does not meet the requirement of being less than or equal to the distance threshold. This helps the input interface control system 200 to integrate more input control functions.

[0056] The input interface control method 100 may further include step 170. Step 170 includes causing the electronic device 300 to execute the corresponding specified function based on the specified gesture when it is determined that the radar point cloud 800 corresponds to a specified gesture. Therefore, the input interface control system 200 can simultaneously possess multiple input control functions and ease of use. Furthermore, if it is not determined in step 170 that the radar point cloud 800 corresponds to a specified gesture, then the process returns to steps 110, 112, 120, 122, and 124.

[0057] The designated function can be equivalent to one of the mouse's scroll wheel, left-click, and right-click functions. Therefore, when the radar wave detects that the target point 820 (fingertip) leaves the reference surface 700 (operation plane), the input interface control system 200 can begin to determine the button or scroll wheel function corresponding to the user's gesture 900. The gesture can be up and down tapping, left and right waving, etc., and the scroll wheel function can be replaced by the finger 920 drawing circles clockwise or counterclockwise.

[0058] The input interface control system 200 can be located in the electronic device 300, which is in Figure 2B Specifically, the input interface control system 200 of the present invention is a laptop computer and includes a keyboard 350, which is connected to a control unit 210. The radar unit 240 and the keyboard 350 can be arranged along a first direction x1, and the radar unit 240 and the camera unit 230 can be arranged along a second direction y1 and facing the recognition space s1, with the second direction y1 being parallel to the extension direction of the finger 920. Therefore, the input interface control system 200 according to the present invention has the advantage of miniaturization and can be built into existing portable electronic devices, such as a laptop computer with the radar unit 240 and camera unit 230 disposed on the side adjacent to the keyboard 350.

[0059] Electronic device 300 may further include pivot 360, which in Figure 2B Specifically, it is parallel to the first direction x1. The radar unit 240 and the camera unit 230 can be located adjacent to and positioned at one end of the parallel pivot 360 of the keyboard 350 of the laptop computer (i.e., Figure 2B(The positive or negative end of the first direction x1). This helps to effectively overlap radar reflection data and image data. Data outside a specific range (e.g., 0.5cm) of the fingertip of finger 920 is not used, reducing errors caused by radar reflections, making cursor 222 move more smoothly, and without drastically altering the user's original mouse usage habits, thus increasing user engagement. Furthermore, the radar unit and camera unit of the input interface control system according to the present invention can be a single device not built into a laptop computer or other electronic device. It can communicate with a laptop computer via wired or wireless means, such as a single device communicating with a laptop computer via a USB interface. The electronic device according to the present invention is not limited to a laptop computer. Moreover, the radar unit and camera unit of the input interface control system according to the present invention can also be located on the side of the keyboard facing the user, or on the edge of the display screen, and the present invention is not limited thereto.

[0060] Figure 2D Draw Figure 2A A schematic diagram of the antenna configuration of the radar unit 240 in the input interface control system 200. Please refer to... Figure 2B and Figure 2D The radar unit 240 may include at least one transmitting antenna and at least three receiving antennas. The antenna configuration in the radar unit 240 may be as follows: Figure 2D As shown, radar unit 240 includes two transmitting antennas and three receiving antennas, namely transmitting antennas 251 and 252 and receiving antennas 256, 257, and 258. Transmitting antennas 251 and 252 transmit transmitted waves, while receiving antennas 256, 257, and 258 receive reflected waves. Transmitting antennas 251 and 252 and receiving antennas 256, 257, and 258 are positioned on antenna plane 242, with the normal direction of antenna plane 242 parallel to a first direction x1. Receiving antennas 256 and 258 are arranged and aligned along a second direction y1, and the first direction x1 and the second direction y1 are perpendicular to each other. This improves radar resolution and ease of use.

[0061] Please refer to Figure 2D The operating frequency of radar unit 240 corresponds to a wavelength of λ, which can be less than 5 mm. Furthermore, radar unit 240 can be a millimeter-wave radar unit. The center-to-center distance between the projected positions of receiving antennas 256 and 257 in the second direction y1 is a2, which satisfies the following condition: 0.25 × λ ≤ a2 ≤ 0.5 × λ. This improves sensing accuracy and meets miniaturization requirements. Furthermore, the center-to-center distance between the projected positions of receiving antennas 257 and 258 in the second direction y1 can be equal to the aforementioned center-to-center distance a2.

[0062] In the second direction y1, among transmitting antennas 251 and 252 and receiving antennas 256, 257, and 258, the two closest ones are transmitting antenna 251 and receiving antenna 256. The center-to-center distance between the projected positions of transmitting antenna 251 and receiving antenna 256 in the second direction y1 is a3. That is, the center-to-center distance between the projected positions of receiving antenna 256 and receiving antennas 256, 257, and 258 in the second direction y1 is a3, which satisfies the following condition: a2 ≤ a3. This satisfies the sensing accuracy requirement and provides design flexibility.

[0063] Two of the receiving antennas 256, 257, and 258 (e.g., receiving antennas 256 and 257) may have different projection positions on the third direction z1, and the first direction x1, the second direction y1, and the third direction z1 are mutually perpendicular. In this way, receiving antennas 256 and 257 are offset from each other on the third direction z1, and receiving antennas 257 and 258 are also offset from each other on the third direction z1, so as to improve the accuracy of the 3D point cloud data by obtaining radar information along the third axis (i.e., the third direction z1).

[0064] Figure 2E Draw Figure 2A A schematic diagram of another antenna configuration for the radar unit 240 of the input interface control system 200. (Refer to...) Figure 2B and Figure 2E The antenna configuration in radar unit 240 can also be as follows: Figure 2E As shown, radar unit 240 includes transmitting antennas 261 and 262 and receiving antennas 266, 267 and 268. Other related details can be found in the foregoing. Figure 2D The contents of transmitting antennas 251, 252 and receiving antennas 256, 257 and 258 are included, and the antenna configuration in the radar unit of the present invention is not limited thereto.

[0065] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art should be able to make various modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the appended claims.

Claims

1. An input interface control method, the input interface control method comprising: A transmission wave is emitted into an identification space via a radar unit; The radar unit receives a reflected wave and obtains a radar point cloud. A captured image of the identification space is obtained through a camera unit; and A target point is identified from the radar point cloud and the captured image, wherein a target location of the target point is used as an input interface.

2. The input interface control method as described in claim 1, further comprising: A reference surface was identified from the radar point cloud and the captured image; as well as Calculate the distance between the target point and the reference surface.

3. The input interface control method as described in claim 2, further comprising: When the distance is less than or equal to a distance threshold, calculate the cursor position on a display screen of an electronic device corresponding to the target position; as well as Move a cursor on the display screen to the cursor position.

4. The input interface control method as described in claim 3, wherein the reference surface is a support surface, and the distance threshold is between 0.5cm and 7cm.

5. The input interface control method as described in claim 2, wherein the radar point cloud includes a point cloud of a user's hand, the captured image includes an image of the hand, and the target point is a fingertip of a finger of the hand.

6. The input interface control method as described in claim 5, further comprising: When the distance is greater than a distance threshold, it is determined whether the radar point cloud corresponds to a specified gesture.

7. The input interface control method as described in claim 6, further comprising: When it is determined that the radar point cloud corresponds to the specified gesture, an electronic device is caused to execute a specified function based on the specified gesture.

8. The input interface control method as described in claim 7, wherein the specified function is equivalent to one of a mouse's scroll wheel function, a left-click function, and a right-click function.

9. The input interface control method as described in claim 1, wherein obtaining the radar point cloud precedes acquiring the acquired image.

10. An input interface control system, the input interface control system comprising: At least one control unit, the at least one control unit including at least one processor and at least one storage medium, wherein the storage medium includes an input interface control module; A radar unit, which is communicatively connected to the at least one control unit; as well as A camera unit, which is communicatively connected to the at least one control unit; Based on the input interface control module, the at least one control unit is used to: The radar unit transmits a wave toward a recognition space. The radar unit receives a reflected wave and obtains a radar point cloud. The camera unit captures an image of the recognition space; and A target point is identified from the radar point cloud and the captured image, wherein a target location of the target point is used as an input interface.

11. The input interface control system of claim 10, wherein, based on the input interface control module, the at least one control unit is further configured to: A reference surface was identified from the radar point cloud and the captured image; and Calculate the distance between the target point and the reference surface.

12. The input interface control system as described in claim 11, further comprising: A display screen that is communicatively connected to the at least one control unit; Based on the input interface control module, the at least one control unit is further used to: When the distance is less than or equal to a distance threshold, calculate the cursor position on the display screen corresponding to the target position; and Move a cursor on the display screen to the cursor position.

13. The input interface control system of claim 12, wherein the reference surface is a support surface and the distance threshold is between 0.5 cm and 7 cm.

14. The input interface control system of claim 11, wherein the radar point cloud includes a point cloud of a user's hand, the captured image includes an image of the hand, and the target point is a fingertip of a finger of the hand.

15. The input interface control system of claim 14, wherein the radar unit includes at least one transmitting antenna and at least three receiving antennas, the at least one transmitting antenna transmitting the transmitted wave, the at least three receiving antennas receiving the reflected wave, the at least one transmitting antenna and the at least three receiving antennas being disposed on an antenna plane, a normal direction of the antenna plane being parallel to a first direction, two of the at least three receiving antennas being arranged and aligned along a second direction, and the first direction and the second direction being perpendicular to each other.

16. The input interface control system as described in claim 15, wherein the operating frequency of the radar unit corresponds to a wavelength λ, which is less than 5 mm, and the center-to-center distance between the projection positions of two of the at least three receiving antennas in the second direction is a2, which satisfies the following condition: 0.25×λ≤a2≤0.5×λ.

17. The input interface control system of claim 16, wherein the center-to-center distance between the projection positions of the at least one transmitting antenna and the closest of the at least three receiving antennas in the second direction is a3, which satisfies the following condition: a2≤a3.

18. The input interface control system of claim 15, wherein two of the at least three receiving antennas have different projection positions in a third direction, and the first direction, the second direction, and the third direction are perpendicular to each other.

19. The input interface control system of claim 15, wherein the input interface control system is disposed in a notebook computer, the notebook computer including a keyboard and communicatively connected to the at least one control unit; in, The radar unit and the keyboard are arranged along the first direction, and the radar unit and the camera unit are arranged along the second direction and facing the recognition space, and the second direction is parallel to an extension direction of the finger.

20. The input interface control system of claim 10, wherein the input interface control system is disposed in a notebook computer, and includes a pivot, wherein the radar unit and the camera unit are adjacent to and disposed at one end of a keyboard of the notebook computer parallel to the pivot.

Images