Naked hand operation method and system in augmented reality

Abstract

The present invention provides a bare - hand operation method in augmented reality. Based on the depth information of a real hand, the spatial coordinates of a virtual hand in a VR system are obtained, and then tracking balls are bound to the virtual hand based on the spatial coordinates. Among them, a palm ball is bound to the palm of the virtual hand, and fingertip balls are bound to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls. Then, the straight - line distance between the fingertip balls and the palm ball is obtained in real time. If the straight - line distance is less than a preset trigger threshold, the first trigger condition is established, and thus VR interaction operations are performed according to the first trigger condition. This bare - hand operation method has higher stability and lower precision requirements because of the introduction of small balls that move with the hand, thereby reducing manpower and financial resources. Moreover, due to the lower precision requirements, it is convenient to perform click operations, greatly improving the user's interaction experience.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and more specifically, to a bare - hand operation method and system in augmented reality. Background Art

[0002] Due to the progress of technology and the diversified development of market demands, virtual reality systems are becoming increasingly common and are applied in many fields, such as computer games, health and safety, industry, and education and training. For example, hybrid virtual reality systems are being integrated into various aspects of life, such as mobile communication devices, game consoles, personal computers, cinemas, theme parks, university laboratories, student classrooms, hospital exercise and fitness rooms, etc.

[0003] With the development of the field of artificial reality, the interaction between users and content in VR, AR, and MR scenarios is essential. At the same time, convenient "bare - hand" gesture interaction has become the trend of development today. Most of the existing gesture interactions in this scenario are single - finger index finger "click" gestures, thumb - and - index - finger "pinch" gestures, fist - clenching "confirm" gestures, etc. However, for simple interaction gestures such as single - finger index finger "click", thumb - and - index - finger "pinch", and fist - clenching for "confirmation", the accuracy requirements for gesture recognition are relatively high. Therefore, the investment in human and financial resources is bound to be large. And when collecting the virtual coordinates of the hand, due to the high accuracy requirements, the positioning of the important joints of the hand is usually unstable, resulting in relatively poor interaction accuracy and experience.

[0004] Therefore, there is an urgent need for a bare - hand operation method in augmented reality that can reduce human and financial investment, improve gesture recognition accuracy, and improve the stability of gesture operations. Summary of the Invention

[0005] In view of the above problems, the purpose of the present invention is to provide a bare - hand operation method in augmented reality to solve the problems that for the existing simple interaction gestures such as single - finger index finger "click", thumb - and - index - finger "pinch", and fist - clenching for "confirmation", the accuracy requirements for gesture recognition are relatively high, so the investment in human and financial resources is bound to be large, and when collecting the virtual coordinates of the hand, due to the high accuracy requirements, the positioning of the important joints of the hand is usually unstable, resulting in relatively poor interaction accuracy and experience.

[0006] A bare - hand operation method in augmented reality provided by the present invention includes:

[0007] Obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand;

[0008] Binding tracking balls to the virtual hand based on the spatial coordinates; wherein, binding a palm ball to the palm of the virtual hand, binding fingertip balls to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls.

[0009] Obtain the straight-line distance between the fingertip ball and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is satisfied.

[0010] Perform VR interaction operations according to the first trigger condition.

[0011] Preferably, the process of obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand includes:

[0012] Obtain the relative distance between the real hand and the sensor in the VR system through a depth camera;

[0013] Obtain the real position of the wrist of the real hand based on the position of the sensor and the relative distance;

[0014] Taking the virtual coordinates of the sensor as a reference, map the real position of the wrist into the virtual space to form the wrist spatial coordinates;

[0015] Perform calculation and filling based on the wrist spatial coordinates and pre-obtained hand joint information to form a virtual hand, and obtain the spatial coordinates of the virtual hand in the VR system.

[0016] Preferably, the process of binding a palm ball to the palm of the virtual hand and binding fingertip balls to the fingertips of the virtual hand includes:

[0017] Obtain the virtual position coordinates of the palm and the virtual position coordinates of the fingertips of the virtual hand;

[0018] Set a palm ball at the virtual position coordinates of the palm and set fingertip balls at the virtual position coordinates of the fingertips; and,

[0019] The palm ball always moves with the movement of the virtual position coordinates of the palm;

[0020] The fingertip ball always moves with the movement of the virtual position coordinates of the fingertips.

[0021] Preferably, the fingertip balls include a little finger fingertip ball, a ring finger fingertip ball, and a middle finger fingertip ball.

[0022] Preferably, the process of obtaining the straight-line distance between the fingertip ball and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is satisfied includes:

[0023] Obtain the fingertip ball - palm ball straight-line distance in real time, where the fingertip ball - palm ball straight-line distance represents the straight-line distance formed between the fingertip ball and the palm ball when the fingers except the thumb move towards the palm to make a fist.

[0024] Compare the straight-line distance between the fingertip ball and the palm ball with a preset trigger threshold. If the straight-line distance between the fingertip ball and the palm ball is less than the trigger threshold, the first trigger condition is satisfied.

[0025] Preferably, it further includes: binding a thumb ball to the thumb of the virtual hand and binding a finger cuboid to the index finger of the virtual hand; wherein,

[0026] The process of binding a thumb ball to the thumb of the virtual hand and binding a finger cuboid to the index finger of the virtual hand includes:

[0027] Obtain the virtual position coordinates of the thumb and the virtual position coordinates of the index finger;

[0028] Set a thumb ball at the virtual position coordinates of the thumb and set a finger cuboid at the virtual position coordinates of the index finger; and,

[0029] The thumb ball always moves with the movement of the virtual position coordinates of the thumb;

[0030] The finger cuboid always moves with the movement of the virtual position coordinates of the index finger.

[0031] Preferably, the distance between the thumb ball and the finger cuboid is obtained in real time, wherein the distance between the thumb ball and the finger cuboid represents the distance formed between the thumb ball and the finger cuboid when the thumb moves closer to the index finger to perform a pinching action;

[0032] When the distance between the thumb ball and the finger cuboid is less than a preset pinching threshold, the second trigger condition is satisfied;

[0033] Trigger an operation cursor at the corresponding position of the virtual hand on the display in the VR system according to the second trigger condition to perform VR interaction operations.

[0034] The present invention also provides a bare hand operating system in augmented reality, which implements the bare hand operation method in augmented reality as described above, including:

[0035] A coordinate corresponding module for obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand;

[0036] A tracking and binding module for binding a tracking ball to the virtual hand based on the spatial coordinates; wherein, a palm ball is bound to the palm of the virtual hand, a fingertip ball is bound to the fingertip of the virtual hand, and the palm ball is larger than the fingertip ball;

[0037] A condition triggering module for obtaining the straight-line distance between the fingertip ball and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is satisfied;

[0038] An interactive execution module, configured to perform VR interaction operations according to the first trigger condition.

[0039] Preferably, the coordinate corresponding module includes:

[0040] A real position calculation unit, configured to obtain the relative distance between the real hand and the sensor in the VR system through a depth camera; and obtain the real wrist position of the real hand based on the position of the sensor and the relative distance;

[0041] A virtual coordinate corresponding unit, configured to map the real wrist position into the virtual space with reference to the virtual coordinates of the sensor to form wrist space coordinates; perform calculation and filling based on the wrist space coordinates and pre-acquired hand joint information to form a virtual hand, and obtain the space coordinates of the virtual hand in the VR system.

[0042] Preferably, it further includes a pinching operation unit,

[0043] The pinching operation unit is configured to, when the distance between the thumb ball and the index finger cuboid is less than a preset pinching threshold, the second trigger condition is established, and trigger an operation cursor at the corresponding position of the virtual hand on the display in the VR system according to the second trigger condition to perform VR interaction operations; wherein, the distance between the thumb ball and the index finger cuboid represents the distance formed between the thumb ball and the index finger cuboid when the thumb moves closer to the index finger to perform a pinching action.

[0044] As can be seen from the above technical solutions, the bare hand operation method and system in the augmented reality provided by the present invention obtain the space coordinates of the virtual hand in the VR system based on the depth information of the real hand, and then bind a tracking ball to the virtual hand based on the space coordinates; wherein, a palm ball is bound to the palm of the virtual hand, and a fingertip ball is bound to the fingertip of the virtual hand, and the palm ball is larger than the fingertip ball; then, the straight-line distance between the fingertip ball and the palm ball is obtained in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is established, and thus VR interaction operations are performed according to the first trigger condition. Since this bare hand operation method introduces a small ball that moves with the hand, the stability is higher, the accuracy requirement is smaller, thereby reducing manpower and financial resources, and because the accuracy requirement is smaller, it is convenient to perform click operations, greatly improving the user's interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By referring to the following description of the specification in conjunction with the drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more apparent and easier to understand. In the drawings:

[0046] Figure 1 It is a flowchart of the bare hand operation method in the augmented reality according to an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of binding a tracking ball in the bare - hand operation method in augmented reality according to an embodiment of the present invention;

[0048] Figure 3 Schematic diagram of a thumb ball and an index - finger cuboid in the bare - hand operation method in augmented reality according to an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of a virtual ray in the bare - hand operation method in augmented reality according to an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of a bare - hand operation system in augmented reality according to an embodiment of the present invention. Detailed implementation manners

[0051] At present, most of the existing gesture interactions in this scenario are single - finger index - finger "click" gestures, thumb - and - index - finger "pinch" gestures, fist - clenching "confirm" gestures, etc. However, for the pure single - finger index - finger "click", thumb - and - index - finger "pinch", and fist - clenching for "confirmation" interaction gestures, the accuracy requirements for gesture recognition are relatively high. Therefore, the investment in human and financial resources generated therefrom is bound to be large. And when collecting the virtual coordinates of the hand, due to the high accuracy requirements, the positioning of the important joints of the hand is usually unstable, resulting in relatively poor interaction accuracy and experience.

[0052] In view of the above problems, the present invention provides a bare - hand operation method and system in augmented reality. The following will describe the specific embodiments of the present invention in detail with reference to the accompanying drawings.

[0053] To illustrate the bare - hand operation method and system in augmented reality provided by the present invention, Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 exemplarily mark the bare - hand operation method in the augmented reality embodiment of the present invention; Figure 5 exemplarily mark the bare - hand operation system in the augmented reality embodiment of the present invention.

[0054] The following description of the exemplary embodiments is actually only illustrative and in no way limits the present invention or its application or use. Technologies and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies and devices should be regarded as part of the specification.

[0055] As Figure 1 shown, the bare - hand operation method in augmented reality provided by the embodiments of the present invention includes:

[0056] S1: Obtain the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand;

[0057] S2: Bind tracking balls to the virtual hand based on spatial coordinates; specifically, bind a palm ball to the palm of the virtual hand and fingertip balls to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls.

[0058] S3: Continuously obtain the straight-line distance between the fingertip balls and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is satisfied.

[0059] S4: Perform VR interaction operations according to the first trigger condition.

[0060] As Figure 1 shown, step S1 is the process of obtaining the spatial coordinates of the virtual hand based on the depth information of the real hand. In this embodiment, the process of obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand includes:

[0061] S11: Obtain the relative distance between the real hand and the sensor in the VR system through a depth camera.

[0062] S12: Obtain the real position of the wrist of the real hand based on the position of the sensor and the relative distance.

[0063] S13: With the virtual coordinates of the sensor as a reference, map the real position of the wrist to the virtual space to form the wrist spatial coordinates.

[0064] S14: Perform calculation and filling based on the wrist spatial coordinates and pre-obtained hand joint information to form a virtual hand, and obtain the spatial coordinates of the virtual hand in the VR system.

[0065] Specifically, first, the process of obtaining the positional relationship between the real hand and the sensor in the VR system through a depth camera is carried out. Since the real position of the sensor and the virtual coordinates of the sensor are both known, the real position of the wrist of the hand can be deduced based on the real position of the sensor. Then, based on the known virtual coordinates of the sensor, the real position of the wrist can be mapped to the virtual space, thus obtaining the wrist spatial coordinates. Moreover, the size of the hand and the positional relationship between the various joints of the hand (hand joint information) are both known. Therefore, the coordinates of the various joints of the hand in the virtual space can be calculated and deduced based on the wrist spatial coordinates, and thus a virtual hand can be filled and formed, and further the spatial coordinates of the entire virtual hand in the VR system can be obtained.

[0066] As Figure 1 、 Figure 2 collectively shown, step S2 is the process of binding tracking balls to the virtual hand. Among them, the process of binding a palm ball (large ball) to the palm of the virtual hand as shown in Figure 2 and binding fingertip balls (small balls) to the fingertips of the virtual hand includes:

[0067] S21: Obtain the virtual position coordinates of the palm center and the virtual position coordinates of the fingertip of the virtual hand;

[0068] S22: Set a palm ball at the virtual position coordinates of the palm center, and set fingertip balls 222 at the virtual position coordinates of the fingertips; and,

[0069] This palm ball always moves as the virtual position coordinates of the palm center move;

[0070] This fingertip ball always moves as the virtual position coordinates of the fingertips move;

[0071] In this way, by judging the distance between the palm ball and the fingertip ball, the distance between the fingertip and the palm center of the finger can be judged, and then the degree of finger bending can be judged to distinguish whether the whole hand makes a fist movement. In this embodiment, the fingertip ball includes a little finger fingertip ball, a ring finger fingertip ball, and a middle finger fingertip ball, that is, the position relationship between the little finger, the ring finger, and the middle finger and the palm center is judged through the fingertip ball, so as to judge whether the user makes a fist. This method of judging the state of the hand by binding balls solves the problem of high precision requirements for traditional bare - hand operation, improves the stability of virtual hand positioning, reduces the precision requirements, and further improves the user's immersive experience.

[0072] As Figure 1 、 Figure 2 Collectively shown, step S3 is the process of obtaining the straight - line distance between the fingertip ball and the palm ball in real time. If the straight - line distance is less than the preset trigger threshold, the first trigger condition is established. This process is to monitor the straight - line distance between the fingertip ball and the palm ball in real time. If the user wants to control the VR system with bare hands, they will start to make a fist. In this embodiment, this process includes:

[0073] S31: Obtain the straight - line distance between the fingertip ball and the palm ball in real time, where this straight - line distance between the fingertip ball and the palm ball represents the straight - line distance formed between the fingertip ball and the palm ball when the fingers except the thumb move closer to the palm to make a fist;

[0074] S32: Compare the straight - line distance between the fingertip ball and the palm ball with the preset trigger threshold. If the straight - line distance between the fingertip ball and the palm ball is less than the trigger threshold, the first trigger condition is established.

[0075] It should be noted that the fingertip ball and the palm ball are virtual balls in the VR system, which can be colored or colorless. For the convenience of illustration, in the appendix Figure 2The palm ball and the fingertip ball are shown, but in this embodiment, the palm ball and the fingertip ball are colorless, transparent, and invisible to the naked eye virtual spheres, which can move as the bound part of the hand moves, so as to increase the stability of the coordinate information of the virtual hand, thereby ensuring the accuracy of the determination of the posture of the virtual hand; in this embodiment, if the first trigger condition is satisfied, any operation corresponding to the first trigger condition can be automatically and immediately executed, thus improving the response speed during bare - hand operation interaction.

[0076] As Figure 1 、 Figure 3 Collectively shown, step S4 is the process of performing VR interaction operations according to the first trigger condition, that is, if the user makes a fist once, then the first trigger condition is satisfied once. If the first trigger condition is satisfied, the VR system automatically executes the interaction operation corresponding to the first trigger condition. The corresponding relationship between the interaction operation and the first trigger condition is preset in advance, and the specific preset process will not be elaborated here; and the first trigger condition can correspond to any operation with an interactive nature, such as entering and exiting a page, clicking on any cursor or identifier on the interface of the display in the VR system, or even performing opening and closing operations, etc. And the opening and closing include, but are not limited to, turning on or off the display. That is, when the user's fist - making action occurs, the VR system can automatically perform any interactive operations such as entering and exiting a page, clicking on any cursor or identifier on the interface of the display in the VR system, or even performing opening and closing operations.

[0077] In addition, as Figure 1 、 Figure 3 Collectively shown, the bare - hand operation method in augmented reality provided by the embodiments of the present invention further includes: S5: Binding a thumb ball to the thumb of the virtual hand and binding an index - finger cuboid to the index finger of the virtual hand; where

[0078] In the process of binding a thumb ball ( Figure 3 the oval shape on the thumb in Figure 3 ) to the thumb of the virtual hand as shown in

[0079] and binding an index - finger cuboid to the index finger of the virtual hand, it includes:

[0080] S511: Obtain the virtual position coordinates of the thumb and the virtual position coordinates of the index finger;

[0081] Set a thumb ball at the virtual position coordinates of the thumb, and set an index - finger cuboid at the virtual position coordinates of the index finger; and,

[0082] The thumb ball always moves as the virtual position coordinates of the thumb move;

[0083] It should be noted that the cuboid of the index finger does not refer to the traditional cuboid shape in the actual sense, but a marker in the shape of a cuboid that wraps around the index finger as shown in Figure 3 It can be slightly flatter and more irregular than a cuboid.

[0084] In the embodiments commonly shown in Figure 1 and Figure 3 include:

[0085] S521: Obtain the distance between the thumb ball and the cuboid of the index finger in real time, where the distance between the thumb ball and the cuboid of the index finger represents the distance formed between the thumb ball and the cuboid of the index finger when the thumb moves closer to the index finger to perform a pinching action;

[0086] S522: When the distance between the thumb ball and the cuboid of the index finger is less than a preset pinching threshold, the second trigger condition is established;

[0087] S523: Trigger the operation cursor at the corresponding position of the virtual hand on the display of the VR system according to the second trigger condition to perform VR interaction operations.

[0088] In the embodiments commonly shown in Figure 1 and Figure 3 and Figure 4 By binding the thumb ball to the position of the thumb of the virtual hand and binding the cuboid of the index finger to the position of the index finger, it is convenient to obtain the distance between the thumb ball and the cuboid of the index finger. When there is a certain need, the second trigger condition is triggered by pinching the index finger with the thumb, that is, the user makes a pinching action of the thumb and the index finger to make the thumb move closer to the index finger to form the distance between the thumb ball and the cuboid of the index finger. When the distance between the thumb ball and the cuboid of the index finger is less than the preset pinching threshold, the second trigger condition is established. After the second trigger condition is established, the interaction operation corresponding to the second trigger condition is directly and automatically started. In this embodiment, the interaction operation corresponding to the second trigger condition can be any operation with properties such as "click" and "press", which is not specifically limited here.

[0089] In the embodiments commonly shown in Figure 1 and Figure 3 and Figure 4 a virtual ray is determined through the joints and specific positions of the user's hand. The virtual ray can be colored or colorless in actual applications. In Figure 4For the sake of illustration, the virtual ray is shown as a shaped and colored line starting from the hand and ending at the display. However, in this embodiment, the virtual ray intersects with the display in the VR system, and the intersection point is where the virtual cursor is in the display. When the user moves the hand, the virtual ray also moves accordingly. Thus, the user can move the virtual cursor in the display by moving the hand, that is, select which position on the display to click by moving the hand. When the virtual cursor is moved to the desired click position, the second trigger condition is triggered by pinching the index finger with the thumb. If the second trigger condition is satisfied, the virtual cursor clicks any clickable page button on the display, such as clicking on an APP icon on the page, clicking OK, or clicking Cancel. After the virtual cursor clicks, it will not be elaborated here. In this embodiment, if the second trigger condition is satisfied and a click is performed on the content in the display interface, when the user releases the thumb, the click action takes effect immediately. Therefore, the second trigger condition can also correspond to the interactive operation of dragging or swiping the content in the display interface, that is, when the user performs a pinching action and the virtual cursor clicks on the content in the display, the user can then move the entire hand, causing the virtual cursor to drive the clicked content to move along with the movement of the hand. When the clicked content is moved to the target position, the user raises the thumb to terminate the pinching action, and the drag or swipe operation will take effect.

[0090] It should be noted that the interactive operations corresponding to the first trigger condition and the second trigger condition can both be set in advance. The objects of the interactive operations can be any target buttons or interfaces in any VR system that can support bare - hand operations. More detailed operation details will not be elaborated here.

[0091] As described above, the bare - hand operation method in augmented reality provided by the present invention obtains the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand, and then binds a tracking ball to the virtual hand based on the spatial coordinates. Specifically, a palm ball is bound to the palm of the virtual hand, and fingertip balls are bound to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls. Then, the straight - line distance between the fingertip ball and the palm ball is obtained in real time. If the straight - line distance is less than a preset trigger threshold, the first trigger condition is satisfied, and thus VR interaction operations are performed according to the first trigger condition. This bare - hand operation method has higher stability and lower precision requirements because of the introduction of the small balls that move with the hand, thus reducing manpower and financial resources. And due to the lower precision requirements, it is convenient to perform click operations, greatly improving the user's interaction experience.

[0092] As Figure 5 shown, the present invention also provides a bare - hand operation system 100 in augmented reality to implement the above - mentioned bare - hand operation method in augmented reality, including:

[0093] A coordinate corresponding module 101, configured to obtain the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand;

[0094] The tracking and binding module 102 is used to bind tracking balls on the virtual hand based on spatial coordinates. Among them, a palm ball is bound to the palm of the virtual hand, and fingertip balls are bound to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls.

[0095] The condition trigger module 103 is used to obtain the straight-line distance between the fingertip ball and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is established.

[0096] The interaction execution module 104 is used to perform VR interaction operations according to the first trigger condition.

[0097] In Figure 5 In the illustrated embodiment, the coordinate corresponding module 101 includes:

[0098] The real position calculation unit 101-1 is used to obtain the relative distance between the real hand and the sensor in the VR system through a depth camera; and obtain the real position of the wrist of the real hand based on the position of the sensor and the relative distance.

[0099] The virtual coordinate corresponding unit 101-2 is used to map the real position of the wrist into the virtual space with reference to the virtual coordinates of the sensor to form wrist space coordinates; perform calculation and filling based on the wrist space coordinates and pre-acquired hand joint information to form a virtual hand, and obtain the space coordinates of the virtual hand in the VR system.

[0100] In addition, the bare hand operating system 100 in the augmented reality provided by the present invention further includes a pinching operation unit 105 (not shown in the figure),

[0101] The pinching operation unit 105 is used to trigger the second trigger condition when the distance between the thumb ball and the index finger cuboid is less than a preset pinching threshold, and trigger an operation cursor at the corresponding position of the virtual hand on the display in the VR system according to the second trigger condition to perform VR interaction operations. Among them, the distance between the thumb ball and the index finger cuboid represents the distance formed between the thumb ball and the index finger cuboid when the thumb moves closer to the index finger to make a pinching action.

[0102] In this way, different functions correspond to the first trigger condition and the second trigger condition respectively, so that the user can realize two types of interaction methods with one hand. The specific interaction operation methods are not specifically corresponded here and can be determined according to the user's needs to improve the interest and stability of the user's bare hand operation.

[0103] As can be seen from the above embodiments, in the bare - hand operation system in augmented reality provided by the present invention, first, the coordinate correspondence module 101 obtains the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand, and then the tracking and binding module 102 binds tracking balls to the virtual hand based on the spatial coordinates; among them, a palm ball is bound to the palm of the virtual hand, and fingertip balls are bound to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls; then the condition trigger module 103 obtains the straight - line distance between the fingertip balls and the palm ball in real time. If the straight - line distance is less than a preset trigger threshold, the first trigger condition is established, and thus the interaction execution module 104 performs VR interaction operations according to the first trigger condition. This bare - hand operation method has higher stability and lower precision requirements due to the introduction of the small balls that move with the hand, thereby reducing manpower and financial resources, and because of the lower precision requirements, it is convenient to perform click operations, greatly improving the user's interaction experience.

[0104] As described above by way of example with reference to the drawings, the method and system of the bare - hand operation system in augmented reality proposed according to the present invention are described. However, those skilled in the art should understand that various improvements can be made to the above - mentioned method and system of the bare - hand operation system in augmented reality without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the appended claims.

Claims

1. A bare - hand operation method in augmented reality, characterized in that, Comprising: Obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand; Binding a tracking ball to the virtual hand based on the spatial coordinates; wherein, a palm ball is bound to the palm of the virtual hand, and fingertip balls are bound to the fingertips of the virtual hand, and the palm ball is larger than the fingertip balls; Obtaining in real time the straight-line distance between the fingertip ball and the palm ball, and if the straight-line distance is less than a preset trigger threshold, the first trigger condition is established; Performing VR interaction operations according to the first trigger condition; Wherein, the process of binding a palm ball to the palm of the virtual hand and binding fingertip balls to the fingertips of the virtual hand includes: Obtaining the virtual position coordinates of the palm and the virtual position coordinates of the fingertips of the virtual hand; Setting a palm ball at the virtual position coordinates of the palm and setting fingertip balls at the virtual position coordinates of the fingertips; and, The palm ball always moves along with the movement of the virtual position coordinates of the palm; The fingertip balls always move along with the movement of the virtual position coordinates of the fingertips.

2. The bare - hand operation method in augmented reality according to claim 1, wherein, The process of obtaining the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand includes: Obtaining the relative distance between the real hand and the sensor in the VR system through a depth camera; Obtaining the real position of the wrist of the real hand based on the position of the sensor and the relative distance; Taking the virtual coordinates of the sensor as a reference, mapping the real position of the wrist into the virtual space to form wrist spatial coordinates; Performing calculation and filling based on the wrist spatial coordinates and pre-obtained hand joint information to form a virtual hand, and obtaining the spatial coordinates of the virtual hand in the VR system.

3. The bare hand operation method in augmented reality according to claim 1, wherein The fingertip balls include a little finger fingertip ball, a ring finger fingertip ball, and a middle finger fingertip ball.

4. The barehanded operation method in augmented reality according to claim 3, wherein The process of obtaining in real time the straight-line distance between the fingertip ball and the palm ball, and if the straight-line distance is less than a preset trigger threshold, the first trigger condition is established, includes: Obtaining in real time the fingertip ball - palm ball straight-line distance, wherein the fingertip ball - palm ball straight-line distance represents the straight-line distance formed between the fingertip ball and the palm ball when the fingers except the thumb move towards the palm to make a fist; Comparing the fingertip ball - palm ball straight-line distance with the preset trigger threshold, and if the fingertip ball - palm ball straight-line distance is less than the trigger threshold, the first trigger condition is established.

5. The bare - hand operation method in augmented reality according to claim 1, characterized in that, Further comprising: Binding a thumb ball to the thumb of the virtual hand and binding an index finger cuboid to the index finger of the virtual hand; wherein, The process of binding a thumb ball to the thumb of the virtual hand and binding an index finger cuboid to the index finger of the virtual hand includes: Obtaining the virtual position coordinates of the thumb and the virtual position coordinates of the index finger; Setting a thumb ball at the virtual position coordinates of the thumb and setting an index finger cuboid at the virtual position coordinates of the index finger; and, The thumb ball always moves along with the movement of the virtual position coordinates of the thumb; The index finger cuboid always moves along with the movement of the virtual position coordinates of the index finger.

6. The bare hand operation method in augmented reality according to claim 5, wherein, Further comprising: Obtain the thumb ball-index finger cuboid distance in real time, where the thumb ball-index finger cuboid distance represents the distance formed between the thumb ball and the index finger cuboid when the thumb moves closer to the index finger to perform a pinching action; When the thumb ball-index finger cuboid distance is less than a preset pinching threshold, the second trigger condition is established; Trigger the operation cursor at the corresponding position of the virtual hand on the display of the VR system according to the second trigger condition to perform VR interaction operations.

7. A bare hand operating system in augmented reality, implementing the bare hand operation method in augmented reality as described in any one of claims 1-6, including: A coordinate corresponding module, configured to obtain the spatial coordinates of the virtual hand in the VR system based on the depth information of the real hand; A tracking and binding module, configured to bind a tracking ball to the virtual hand based on the spatial coordinates; wherein, a palm ball is bound to the palm of the virtual hand, a fingertip ball is bound to the fingertip of the virtual hand, and the palm ball is larger than the fingertip ball; A condition trigger module, configured to obtain the straight-line distance between the fingertip ball and the palm ball in real time. If the straight-line distance is less than a preset trigger threshold, the first trigger condition is established; An interaction execution module, configured to perform VR interaction operations according to the first trigger condition; Wherein, the process of binding a palm ball to the palm of the virtual hand and a fingertip ball to the fingertip of the virtual hand includes: Obtain the virtual position coordinates of the palm and the virtual position coordinates of the fingertip of the virtual hand; Set a palm ball at the virtual position coordinates of the palm and set a fingertip ball at the virtual position coordinates of the fingertip; and, The palm ball always moves with the movement of the virtual position coordinates of the palm; The fingertip ball always moves with the movement of the virtual position coordinates of the fingertip.

8. The barehand operating system in augmented reality according to claim 7, wherein The coordinate corresponding module includes: A real position calculation unit, configured to obtain the relative distance between the real hand and the sensor in the VR system through a depth camera; obtain the real position of the wrist of the real hand based on the position of the sensor and the relative distance; A virtual coordinate corresponding unit, configured to map the real position of the wrist into the virtual space with reference to the virtual coordinates of the sensor to form wrist spatial coordinates; calculate and fill based on the wrist spatial coordinates and pre-obtained hand joint information to form a virtual hand, and obtain the spatial coordinates of the virtual hand in the VR system.

9. The bare - hand operation system in augmented reality according to claim 7, characterized in that, It further includes a pinching operation unit, The pinching operation unit is configured to when the thumb ball-index finger cuboid distance is less than a preset pinching threshold, the second trigger condition is established, and trigger the operation cursor at the corresponding position of the virtual hand on the display of the VR system according to the second trigger condition to perform VR interaction operations; wherein, the thumb ball-index finger cuboid distance represents the distance formed between the thumb ball and the index finger cuboid when the thumb moves closer to the index finger to perform a pinching action.

Images