Point cloud-based pinch interaction method, head-mounted device, interaction system and storage medium

Abstract

The application discloses a pinching interaction method based on point cloud, a head-mounted device, an interaction system and a storage medium, and belongs to the technical field of smart glasses interaction. In the pinching interaction, the touch in touch control operation, mouse operation or keyboard input is determined as follows: a hand point cloud is output by using a skeleton joint model, the hand point cloud comprises an epidermis point cloud and a skeleton joint line, each point on the hand point cloud is tracked by a monocular camera in real time, and the XY position of each point is inferred; whether the epidermis point cloud of the trigger area exceeds the epidermis point cloud of the button area is determined; if yes, it is determined that the touch occurs. According to the application, the monocular camera can be used to accurately determine the touch, and the misjudgment caused by the fact that the binocular camera cannot simultaneously collect gesture actions can be avoided. The point cloud inference of the application can be realized in the smart glasses through a small neural network chip, and touch control, mouse or keyboard signals are output to an interaction terminal, so that the smart glasses can directly replace the mouse or the keyboard.

Description

Technical Field

[0001] This invention relates to the field of smart glasses interaction technology, specifically to a finger-tapping interaction method based on point clouds, a head-mounted display device, an interaction system, and a storage medium. Background Technology

[0002] There is an existing finger-pinching interaction method based on a hand joint model with neural network inference. Virtual buttons are drawn on the palm, and the fingertip of the thumb (or any finger) (or any part that can be focused as a cursor point) is defined as the "trigger fingertip". Virtual buttons are drawn on the three knuckles of each other finger and / or different areas of the palm. Each virtual button is defined as a different number key, letter key or function key. The human hand joint detection model is used to calculate the threshold position of whether the trigger fingertip touches the virtual button. At least two cameras are required to determine whether a touch has occurred.

[0003] However, in practical applications, users are accustomed to placing their hands on the lower sides of the smart glasses, such as when sitting in a chair, with their hands on their hip joints or the armrests. Although the camera is a 180° fisheye lens pointing downwards at 45°, the left and right cameras cannot simultaneously capture the same hand, thus making touch detection impossible.

[0004] Even if both cameras can see the same hand, the error is significant in actual operation. This is mainly because users don't intentionally raise their fingertip to touch another button area. Instead, they gently move their fingertip a few millimeters away from the button area and then move to another button area. Therefore, when a user's fingertip moves away, say, two millimeters and then double-clicks the button area, it cannot determine whether the fingertip clicked or double-clicked. Similarly, when the fingertip moves from one button area to a knuckle two button areas away, the system will mistakenly assume that the button area it crossed in between was also touched. Summary of the Invention

[0005] The purpose of this invention is to address the problem of misjudging whether a touch is real in existing gesture recognition and visual computing. It proposes a finger-pinching interaction method, head-mounted display device, interaction system, and storage medium based on point cloud. Point cloud technology is used to calculate touch judgment in finger pinching with millimeter-level precision. It can be achieved with only a single camera. Of course, if multiple cameras or SLAM-assisted technologies such as LiDAR can be added to increase accuracy, multiple cameras or LiDAR can be added.

[0006] This invention discloses a finger-pinching interaction method based on point clouds. The finger-pinching interaction involves defining a trigger fingertip and drawing virtual buttons on the knuckles of each of the other fingers and / or different areas of the palm. These virtual buttons are then defined to correspond to different number keys, letter keys, or function keys. Touch operations, mouse operations, or keyboard input are achieved by triggering the fingertip to touch the virtual buttons. The touch is determined as follows:

[0007] A trigger area is defined on the fingertip, and button areas are defined in different areas of the knuckles of each other finger and / or the palm. A hand point cloud is output using a skeletal joint model, which includes epidermal point cloud and skeletal joint lines. Each point on the hand point cloud is tracked and its XY position is inferred in real time by a monocular camera. It is determined whether the epidermal point cloud of the trigger area exceeds the epidermal point cloud of the triggered button area. If so, a touch is determined to have occurred.

[0008] The touch detection also includes the following steps:

[0009] Preprocess the skin point cloud output from the skeletal joint model to remove points below and above the skin:

[0010] Using a monocular camera, the point cloud and skeletal joint lines inferred from the skeletal joint model are observed. Each point in the point cloud is vertically projected onto the skeletal joint line. If there is only one point on the vertical line, that point is retained; if there are multiple points on the vertical line, only the point with the furthest distance is retained. After traversal, each remaining point is compared with its two adjacent points, and the vertical distances of these three points to the skeletal joint lines are analyzed. If the vertical distance of a point is the smallest and the difference between it and the other two points exceeds a threshold, then the point is determined to be below the epidermis and is deleted. Conversely, if the vertical distance of a point is the largest and the difference between it and the other two points exceeds a threshold, then the point is determined to be above the epidermis and is deleted. After traversal, the remaining points are the points on the epidermal point cloud.

[0011] Each point on the hand point cloud is tracked in real time by at least one camera to infer its XY or XYZ position.

[0012] The click action in the touch operation:

[0013] The XY position of each point in the point cloud is inferred. If any two points in the surface point cloud of the trigger area touch and intersect with any two points in the surface point cloud of the button area, it is determined that the trigger area has touched the button area. Within a preset time after the touch is determined, if the two intersecting points in the surface point cloud of the trigger area move away from the surface point cloud of the button area, it is determined that the click action is completed.

[0014] The XYZ position is obtained by reasoning for each point in the point cloud. If any three non-linear points in the surface point cloud of the trigger area touch and intersect with any three non-linear points in the surface point cloud of the button area, it is determined that the trigger area has touched the button area. Within a preset time after the touch is determined, if the three intersecting non-linear points in the surface point cloud of the trigger area move away from the surface point cloud of the button area, it is determined that the click action is completed.

[0015] In the mouse operation, a mouse cursor generation gesture is defined. When the system recognizes the mouse cursor generation gesture, it outputs the cursor. During the gesture holding phase, the cursor moves with the hand and the cursor movement speed is adjusted synchronously with the hand movement speed.

[0016] Define mouse left / right button gestures: Define the trigger area on the tip of the thumb, define the left mouse button area on the far knuckle of the index finger, and define the right mouse button area on the near knuckle of the index finger. Define the corresponding trigger command for single click, double click, or long press of the left / right mouse button area.

[0017] Define mouse wheel operation gestures: Define the button area where the middle joint of the index finger is the starting point of the scroll wheel movement. When the trigger area of ​​the thumb tip touches the button area where the scroll wheel movement starts, the mouse wheel operation is triggered. The scroll wheel movement can be defined as downward scrolling, upward scrolling, left scrolling, or right scrolling, depending on the direction of the thumb tip relative to the index finger.

[0018] A head-mounted display device includes: a camera, a display, a memory, and a processor including a neural network chip;

[0019] The camera is connected to the processor and is used to scan the user's real-world environment.

[0020] The display is connected to the processor and is used to display the processor's output.

[0021] This neural network chip is used to provide point cloud-based skeletal joint inference and output a hand point cloud to the processor, which includes skin point cloud and skeletal joint lines.

[0022] The memory is connected to the processor for storing computer programs and transferring the programs to the processor, which executes the computer programs to implement the method described in any of the above-mentioned cases.

[0023] An interactive system includes a head-mounted display device and an interactive terminal, wherein the head-mounted display device transmits touch, mouse, or keyboard signals to the interactive terminal to realize touch operation, mouse operation, or keyboard input.

[0024] A computer-readable storage medium storing a computer program thereon, the computer program implementing the method described in any one of the preceding methods when executed by a processor.

[0025] The technical solution of this invention outputs a point cloud including epidermal point cloud and skeletal joint lines through a skeletal joint model. Each point on the hand point cloud is tracked and its XY position is inferred in real time by a monocular camera. It determines whether the epidermal point cloud of the trigger area exceeds the epidermal point cloud of the button area. If so, a touch is determined. Only a monocular camera is needed for accurate touch detection, avoiding misjudgments caused by the inability of binocular cameras to simultaneously capture hand gestures. The point cloud inference of this invention can be implemented by installing a small neural network chip in smart glasses, directly calculating and outputting touch, mouse, or keyboard signals to an interactive terminal, which can be a mobile phone, tablet, computer, or laptop. Alternatively, the point cloud inference can be performed on the interactive terminal itself, meaning the camera needs to transmit the video stream to the interactive terminal for inference, and then the interactive terminal outputs the corresponding touch, mouse, or keyboard signals. The former has the advantage of saving the bandwidth of the camera's video stream transmission, allowing the smart glasses to directly replace a mouse or keyboard. This invention does not require every point in the trigger area or button area to remain permanently present during operation. Touch detection can be achieved simply by describing the points present in the trigger area and button area at the time of touch, resulting in good robustness. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a hand containing only the thumb and index finger, as seen through a monocular camera in this invention.

[0027] Figure 2 This is a schematic diagram of the hand point cloud inferred from the self-trained skeletal joint model in this invention.

[0028] Figure 3 This is a schematic diagram illustrating the triggering of a fingertip touch on a preset button area in this invention, wherein... Figure 3 (1) is a schematic diagram including the skin point cloud and bone joint lines inferred from the self-trained skeletal joint model. Figure 3 (2) is a schematic diagram of a hand as seen through a monocular camera;

[0029] Figure 4 This is a schematic diagram illustrating the process of determining the completion of a click action based on point clouds in this invention;

[0030] Figure 5 This is a schematic diagram of generating gestures from a mouse cursor in this invention. The left side of the diagram shows a gesture point cloud, and the right side shows a physical gesture.

[0031] Figure 6 This is a schematic diagram of the left mouse button operation gesture in this invention. The left side of the diagram is a gesture dot cloud, and the right side is a physical gesture.

[0032] Figure 7 This is a schematic diagram of the right-click operation gesture in the present invention. The left side of the diagram is a gesture dot cloud, and the right side is a physical gesture.

[0033] Figure 8 This is a schematic diagram of the mouse wheel operation gesture process in this invention. In the figure, (1) is the gesture point cloud of the start of the scroll wheel sliding, (2) is the physical gesture of the start of the scroll wheel sliding, (3) is the gesture point cloud of the scroll wheel scrolling upward, (4) is the physical gesture of the scroll wheel scrolling upward, (5) is the gesture point cloud of the scroll wheel scrolling downward, and (6) is the physical gesture of the scroll wheel scrolling upward.

[0034] Figure 9 This is a functional structural block diagram of a head-mounted display device according to the present invention;

[0035] Figure 10 This is a schematic diagram of the framework of the interactive system in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0038] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or scheme described as "exemplary" or "for example" in this embodiment of the invention should not be construed as being more preferred or advantageous than other embodiments or schemes. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] Explanation of the principle behind the technical implementation of this invention:

[0040] The inventors further improved the touch judgment method in the prior application ZL202311811248.1 "Implementation method of typing or touch with a sense of physicality". The finger pinching interaction defines the fingertip of the thumb (or any finger) (or any part that can be focused as a cursor point) as the "trigger fingertip" and draws virtual buttons on the three knuckles of each other finger and / or different areas of the palm. The virtual buttons are defined to correspond to different number keys, letter keys or function keys. The touch operation, mouse operation or keyboard input in the finger pinching interaction of this invention continues the definition in the prior application, and will not be repeated here.

[0041] This invention enables touch operations, mouse operations, or keyboard input by triggering a fingertip touch a virtual button. The touch is determined as follows:

[0042] Define the trigger area and button area: such as Figures 1 to 3 As shown, a trigger area 105 is defined on the triggering finger, and button areas 106 are defined on the three knuckles of each of the other fingers and / or different areas of the palm. In a two-dimensional world, the trigger area or button area must have at least two points; in a three-dimensional world, it must have at least three points that are not on the same straight line. Two-dimensional requires at least two points, and three-dimensional requires at least three non-linear points. The trigger area may consist of multiple points. Theoretically, the surface is a smooth convex surface, but in reality, it is a bumpy convex surface. This invention does not require a smooth surface. Each point in the point cloud has an XY position in the two-dimensional world and an XYZ position in the three-dimensional world. The position information can be based on the camera as the origin or on a point relative to a part of the hand, such as the XYZ position with the center of the wrist as the origin. With continuous hand operation, the surface points of each touch area should theoretically remain consistent; however, in reality, some points may disappear and new points may be generated over time. This invention does not require each point to exist permanently, but rather describes the trigger area and button area by the points where the trigger area 105 and button area 106 exist when touched.

[0043] This invention utilizes a self-trained skeletal joint model to output hand point clouds of the palm and each finger. These hand point clouds include epidermal point clouds and skeletal joint lines, such as... Figure 2 As shown, each point on the hand-held point cloud is tracked and its XY position is inferred in real time by a monocular camera, such as... Figure 1 As shown, after obtaining the accurate skin point cloud and the XY position of each point, it is possible to precisely determine whether the skin point cloud of the trigger area on the fingertip exceeds the skin point cloud of the button area, such as... Figure 3 As shown, if so, a touch intersection is determined to have occurred, that is, a fingertip touches the point cloud of the button area. Therefore, it is evident that if the skeletal joint model cannot output the skin point cloud, the touch detection of this invention cannot be achieved.

[0044] Each point on the hand point cloud is tracked in real time by at least one camera to infer its XY or XYZ position.

[0045] Figure 1 100 represents the hand seen by the monocular camera, containing only the thumb and index finger, while 101 represents the skeletal joint lines inferred by the skeletal joint inference model. The square dots represent joints containing the nail (fingertip), and the lines represent the bones connecting the joints. Figure 2 This is a schematic diagram of a hand point cloud derived from a self-trained skeletal joint model. 102 represents the skin point cloud of the hand, 101 represents the skeletal joint line (the skin point cloud is located on both sides of the joint), 105 represents the trigger area on the skin point cloud of the fingertip, and 106 represents the button area on the skin point cloud of the index finger. The hand point cloud includes not only the skin point cloud 102 but also non-skin points, such as 103 (points below the skin) and 104 (points above the skin). When determining a touch, this invention... Figure 3 As shown, only the surface point cloud is needed. To improve the accuracy of the judgment, the output surface point cloud needs to be preprocessed before determining the touch, removing points below and above the surface. The steps are as follows:

[0046] Using a monocular camera, the point cloud and skeletal joint lines inferred from the skeletal joint model are observed. Each point in the point cloud is vertically projected onto the skeletal joint line. If there is only one point on the vertical line, that point is retained; if there are multiple points on the vertical line, only the point with the furthest distance is retained. After traversal, each remaining point is compared with its two adjacent points, and the vertical distances of these three points to the skeletal joint lines are analyzed. If the vertical distance of a point is the smallest and the difference between it and the other two points exceeds a threshold, then the point is determined to be below the epidermis and is deleted. Conversely, if the vertical distance of a point is the largest and the difference between it and the other two points exceeds a threshold, then the point is determined to be above the epidermis and is deleted. After traversal, the remaining points are the points on the epidermal point cloud.

[0047] A hand point cloud is output using a skeletal joint model. This hand point cloud includes epidermal point cloud and skeletal joint lines. Each point on the hand point cloud is tracked and its XY position is inferred in real time by a monocular camera. The system determines whether the epidermal point cloud of the trigger area on the fingertip exceeds the epidermal point cloud of the triggered button area. Figure 3 As shown, if so, then a touch is determined to have occurred.

[0048] Define the click action of touch operation: Any three non-linear points (3D world) or any two points (2D world) in the skin point cloud of the trigger area 105 (such as the inner side of the thumb tip) touch and intersect with any three non-linear points (3D world) or any two points (2D world) in the skin point cloud of the button area 106. That is, any three non-linear points (3D world) or any two points (2D world) in the skin point cloud of the trigger area exceed any three non-linear points (3D world) or any two points (2D world) in the skin point cloud of the button area. The non-linear points refer to points that are not on the same straight line. Figure 4 In (1), 105 is the trigger area, which consists of points A to G in the epidermal point cloud on the inner side of the thumb tip. Figure 4 In (2), 106 is the button area, which is composed of points Q to W in the dot cloud of the epidermis on the inner side of the index finger joint. Figure 4 In (3) and (4), if any two points from A to G in the trigger area 105 do not exceed any two points from Q to W in the button area 106, it is determined that the trigger area does not touch the button area. Figure 4 In step (5), if any two points from A to G in the trigger area 105 have exceeded any two points from Q to W in the button area 106, it is determined that the trigger area has touched the button area. Figure 4 In step (6), if any two points from point A to point G in the trigger area 105 do not exceed any two points from point Q to point W in the button area 106 after the touch is determined, then the click action is determined to be completed.

[0049] Define cursor operations in mouse operations: This invention defines mouse cursor generation gestures, such as... Figure 5 For example, the index finger is bent or the tip of the thumb is above the side of the middle phalanx of the index finger (see...). Figure 5 When the system recognizes a mouse cursor gesture, it outputs the cursor. During the gesture holding phase, the cursor moves along with the hand, and its movement speed is adjusted synchronously with the hand's movement speed. If the origin of the point cloud XYZ position is the camera, the cursor movement distance or speed is calculated directly relative to the origin. If the origin is a part of the hand, such as the center of the wrist, the system calculates the relative movement speed or distance based on the position of the previous frame.

[0050] Define left / right mouse button gestures: Define the trigger area on the tip of the thumb, and define the left mouse button area on the distal phalanx of the index finger (see...). Figure 6 Define the left mouse button area near the knuckle of the index finger (see...). Figure 7 Define corresponding trigger commands for single clicks, double clicks, or long presses of the left / right mouse button area. If right-click operation is not needed, it can also be used as a back button command.

[0051] Define mouse wheel operation gestures: Scroll wheel operation requires sliding. Define the middle joint of the index finger as the button area where the scroll wheel slides. When the trigger area of ​​the thumb tip touches the middle joint of the index finger (the button area where the scroll wheel slides), the mouse wheel operation is triggered. The scroll wheel can be defined as scrolling down, scrolling up, scrolling left, or scrolling right, depending on the sliding direction of the thumb tip relative to the index finger. Figure 8 This is a schematic diagram of the mouse wheel operation gesture process in this invention. In the figure, (1) is the gesture point cloud of the start of the scroll wheel sliding, (2) is the physical gesture of the start of the scroll wheel sliding, (3) is the gesture point cloud of the scroll wheel scrolling upward, (4) is the physical gesture of the scroll wheel scrolling upward, (5) is the gesture point cloud of the scroll wheel scrolling downward, and (6) is the physical gesture of the scroll wheel scrolling upward.

[0052] Example 1

[0053] Embodiment 1 of this invention provides a finger-pinching interaction method based on point clouds. The finger-pinching interaction involves defining a trigger fingertip and drawing virtual buttons on the knuckles of each of the other fingers and / or different areas of the palm. These virtual buttons are then defined to correspond to different number keys, letter keys, or function keys. Touch operations, mouse operations, or keyboard input are achieved by triggering the fingertip to touch the virtual buttons. The touch is determined as follows:

[0054] A trigger area is defined on the fingertip, and button areas are defined in different areas of the knuckles of each other finger and / or the palm. A hand point cloud is output using a skeletal joint model, which includes epidermal point cloud and skeletal joint lines. Each point on the hand point cloud is tracked and its XY position is inferred in real time by a monocular camera. It is determined whether the epidermal point cloud of the trigger area exceeds the epidermal point cloud of the triggered button area. If so, a touch is determined to have occurred.

[0055] The touch detection process also includes the following steps:

[0056] Preprocess the skin point cloud output from the skeletal joint model to remove points below and above the skin:

[0057] Using a monocular camera, the point cloud and skeletal joint lines inferred from the skeletal joint model are observed. Each point in the point cloud is vertically projected onto the skeletal joint line. If there is only one point on the vertical line, that point is retained; if there are multiple points on the vertical line, only the point with the furthest distance is retained. After traversal, each remaining point is compared with its two adjacent points, and the vertical distances of these three points to the skeletal joint lines are analyzed. If the vertical distance of a point is the smallest and the difference between it and the other two points exceeds a threshold, then the point is determined to be below the epidermis and is deleted. Conversely, if the vertical distance of a point is the largest and the difference between it and the other two points exceeds a threshold, then the point is determined to be above the epidermis and is deleted. After traversal, the remaining points are the points on the epidermal point cloud.

[0058] Each point on the hand point cloud is tracked in real time by at least one camera to infer its XY or XYZ position.

[0059] The click action in the touch operation:

[0060] The XY position of each point in the point cloud is inferred. If any two points in the surface point cloud of the trigger area touch and intersect with any two points in the surface point cloud of the button area, it is determined that the trigger area has touched the button area. Within a preset time after the touch is determined, if the two intersecting points in the surface point cloud of the trigger area move away from the surface point cloud of the button area, it is determined that the click action is completed.

[0061] The XYZ position is obtained by reasoning for each point in the point cloud. If any three non-linear points in the surface point cloud of the trigger area touch and intersect with any three non-linear points in the surface point cloud of the button area, it is determined that the trigger area has touched the button area. Within a preset time after the touch is determined, if the three intersecting non-linear points in the surface point cloud of the trigger area move away from the surface point cloud of the button area, it is determined that the click action is completed.

[0062] In the mouse operation, a mouse cursor generation gesture is defined. When the system recognizes the mouse cursor generation gesture, it outputs the cursor. During the gesture holding phase, the cursor moves with the hand and the cursor movement speed is adjusted synchronously with the hand movement speed.

[0063] Define left / right mouse button gestures: Define the trigger area on the tip of the thumb, and define the left mouse button area on the distal phalanx of the index finger (see...). Figure 6 Define the left mouse button area near the knuckle of the index finger (see...). Figure 7 Define corresponding trigger commands for single clicks, double clicks, or long presses of the left / right mouse button area. If right-click operation is not needed, it can also be used as a back button command.

[0064] Define mouse wheel operation gestures: Scroll wheel operation requires sliding. Define the middle joint of the index finger as the button area where the scroll wheel slides. When the trigger area of ​​the thumb tip touches the middle joint of the index finger (the button area where the scroll wheel slides), the mouse wheel operation is triggered. The scroll wheel can be defined as scrolling down, scrolling up, scrolling left, or scrolling right, depending on the sliding direction of the thumb tip relative to the index finger.

[0065] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0066] Specifically, the steps of the method embodiments in this invention can be implemented by integrated logic circuits in the processor hardware and / or instructions in software form. The steps of the method disclosed in this invention can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software modules can reside in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, and registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0067] Example 2

[0068] Embodiment 2 of the present invention provides a head-mounted display device, such as Figure 9 As shown, the head-mounted display device 200 includes: a camera 201, a display 202, a memory 203, and a processor 204 including a neural network chip 205;

[0069] The camera 201 is connected to the processor 204 and is used to scan the user's real-world environment.

[0070] The display 202 is connected to the processor 204 and is used to display the output of the processor 204;

[0071] The neural network chip 205 is used to provide point cloud-based skeletal joint inference and output a hand point cloud to the processor 204, the hand point cloud including skin point cloud and skeletal joint lines;

[0072] The memory 203 is connected to the processor 204 and is used to store computer programs and transfer the programs to the processor 204. In other words, the processor 204 can call and run the computer programs from the memory 203 to implement the method in Embodiment 1 of this application.

[0073] In some embodiments of this application, the processor 204 may include, but is not limited to:

[0074] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0075] In some embodiments of this application, the memory 203 includes, but is not limited to, volatile memory and / or non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0076] The neural network chip 205 includes, but is not limited to, chips with point cloud-based skeletal joint inference capabilities, such as GPUs and NPUs.

[0077] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 203 and executed by the processor 204 to complete the method of Embodiment 1 provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the head-mounted display device 200.

[0078] It should be understood that the various components in the head-mounted display device 200 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0079] Example 3

[0080] Embodiment 3 of the present invention also provides an interactive system, such as Figure 10 As shown, the device includes the head-mounted display device 200 and the interactive terminal 300 described in Embodiment 3. The head-mounted display device 200 transmits touch, mouse, or keyboard signals to the interactive terminal 300 to realize touch operation, mouse operation, or keyboard input. The interactive terminal 300 includes, but is not limited to, mobile phones, tablets, computers, or laptops.

[0081] Example 4

[0082] Embodiment 4 of the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the method in Embodiment 1 above.

[0083] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for pinch interaction based on point cloud, the pinch interaction is to define a trigger finger tip and draw a virtual button on different area of the knuckles of other fingers and / or palm, define different digital keys, letter keys or function keys respectively, and realize touch operation, mouse operation or keyboard input by touching the virtual button with the trigger finger tip, characterized in that, The touch is determined as follows: Defining a trigger area on the trigger fingertip, defining button areas on the knuckles of the other fingers and / or different areas of the palm, outputting a hand point cloud including an epidermis point cloud and a skeletal joint line using a skeletal joint model, each point on the hand point cloud being tracked in real time by a monocular camera and reasoning its XY position, determining whether the epidermis point cloud of the trigger area exceeds the epidermis point cloud of the triggered button area, and if so, determining that a touch has occurred; The touch determination further includes the following steps: Pretreating the epidermis point cloud output by the skeletal joint model to remove points below and above the epidermis: Seeing the point cloud and the skeletal joint line reasoned by the skeletal joint model through the monocular camera, vertically projecting each point in the point cloud onto the skeletal joint line, if there is only one point on the vertical line, retaining this point, if there are multiple points on the vertical line, retaining only the point farthest away; after traversal, comparing the vertical distance of each remaining point to the skeletal joint line with the vertical distance of the adjacent two points, if the vertical distance of the point is the smallest and the difference with the other two points exceeds the threshold, it is determined that the point is below the epidermis and is deleted; otherwise, if the vertical distance of the point is the largest and the difference with the other two points exceeds the threshold, it is determined that the point is above the epidermis and is deleted; After traversal, the remaining points are the points on the epidermis point cloud.

2. The point cloud based pinch interaction method of claim 1, wherein, Each point on the hand point cloud is tracked in real time by at least one camera, and its XY position or XYZ position is reasoned.

3. The point cloud based pinch interaction method of claim 2, wherein, The click action in the touch operation: If any two points in the epidermis point cloud of the trigger area and any two points in the epidermis point cloud of the button area intersect, it is determined that the trigger area has touched the button area, and within a preset time after the determination that the trigger area has touched the button area, if the two intersecting points in the epidermis point cloud of the trigger area move away from the epidermis point cloud of the button area, it is determined that the click action is complete. If any three non-linear points in the epidermis point cloud of the trigger area and any three non-linear points in the epidermis point cloud of the button area intersect, it is determined that the trigger area has touched the button area, and within a preset time after the determination that the trigger area has touched the button area, if the three non-linear points in the epidermis point cloud of the trigger area move away from the epidermis point cloud of the button area, it is determined that the click action is complete.

4. The point cloud based pinch interaction method of claim 1, wherein In the mouse operation, a mouse cursor generation gesture is defined, the system outputs a cursor when the mouse cursor generation gesture is recognized, and the cursor moves following the movement of the hand in the gesture maintaining stage, and the cursor moves at a speed synchronized with the movement speed of the hand; Defining a mouse left / right button operation gesture: defining a trigger area on the index fingertip, defining a mouse left button area on the distal phalanx of the index finger, and defining a mouse left button area on the proximal phalanx of the index finger, and defining corresponding trigger instructions for single-clicking, double-clicking or long-pressing the mouse left / right button area of the trigger area. The mouse wheel operation gesture is defined: a button area where the middle segment of the index finger is the starting point of the wheel sliding is defined, when the trigger area of the thumb tip touches the button area of the starting point of the wheel sliding, the mouse wheel operation is triggered, and according to the sliding direction of the thumb tip relative to the index finger, downward scrolling, upward scrolling, left wheel or right wheel of the wheel is defined.

5. A head-mounted device, comprising: The head-mounted device comprises a camera, a display, a memory and a processor comprising a neural network chip; The camera is connected with the processor and is used for scanning a real environment where a user is located; The display is connected with the processor and is used for displaying output content of the processor; The neural network chip is used for providing point cloud-based skeleton joint inference and outputting a hand point cloud to the processor, the hand point cloud comprising an epidermis point cloud and a skeleton joint line; The memory is connected with the processor and is used for storing a computer program and transmitting the program to the processor, the processor being used for executing the computer program to realize the method in any one of claims 1 to 4.

6. An interactive system, characterized by The head-mounted device and an interactive terminal are provided, the head-mounted device transmits touch, mouse or keyboard signals to the interactive terminal to realize touch operation, mouse operation or keyboard input.

7. A computer readable storage medium characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to realize the method in any one of claims 1 to 4.

Images