Gesture interaction method and device in XR environment, electronic equipment and product

Abstract

The invention discloses a gesture interaction method and device in an XR environment, electronic equipment and a product, and relates to the technical field of gesture recognition. Determining a first distance between a fingertip joint of a thumb and a fingertip joint of an index finger and a moving direction of the index finger based on the spatial position information of the finger joints; based on the spatial position information of the palm center and the finger joints of all the fingers, second distances between the fingertip joints of the other fingers except the thumb and the palm center are determined; judging whether a finger is in a kneading state or not based on the first distance, the palm center orientation and the palm center constraint direction during kneading, judging whether the finger is in a holding state or not based on the second distance and the posture information of each finger, and judging whether the finger is in a poked state or not based on the index finger moving direction and the index finger orientation in the posture information; and if the virtual operation object is in one of the states at present and the virtual operation object conforms to the view limitation, generating a corresponding interaction instruction. According to the invention, the detection precision of kneading, holding and poking gestures can be improved, and the use experience and immersion are improved.

Description

Technical Field

[0001] The present invention belongs to the field of gesture recognition technology, and specifically relates to a gesture interaction method, device, electronic device and product in an XR environment. Background Art

[0002] Extended Reality (XR) gesture recognition technology is one of the core technologies in the fields of Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It enables natural interaction between people and virtual or augmented environments by capturing user hand movements and converting them into digital commands.

[0003] Currently, existing XR gesture recognition technologies mostly detect pinching, holding, and tapping based on the distance between fingers, the distance between fingertips and palms, and the distance between fingertips and virtual objects. Interaction with virtual objects is then based on the detection results. However, this simple distance-based approach is not precise enough for gesture detection and is prone to false triggering, resulting in an unnatural or inaccurate interaction experience, reducing user immersion and operational efficiency.

[0004] Therefore, how to provide an effective solution to improve the accuracy of gesture interaction has become a difficult problem to be solved in the existing technology. Summary of the Invention

[0005] The purpose of the present invention is to provide a gesture interaction method, device, electronic device and product in an XR environment to solve the above-mentioned problems existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a gesture interaction method in an XR environment, comprising: Obtain the palm orientation of the user's hand, the spatial position information of the palm, the spatial position information of the knuckles of each finger, and the posture information of each finger; Based on the spatial position information of the knuckles of each finger, determining a first distance between the fingertip joint of the thumb and the fingertip joint of the index finger and a moving direction of the index finger; Determining a second distance between the palm and the fingertips of the fingers other than the thumb based on the spatial position information of the palm and the spatial position information of the knuckles of the fingers; Determining whether the current state is a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determining whether the current state is a hold state based on the second distance and the posture information of each finger; and determining whether the current state is a poke state based on the index finger movement direction and the index finger orientation in the posture information; If the judgment result is that the current state is one of the finger pinching state, holding state and finger poking state, and the virtual operation object currently meets the field of view limitation, then an interaction instruction corresponding to one of the states is generated to perform the interaction operation corresponding to the interaction instruction on the virtual operation object.

[0007] Based on the above disclosure, the present invention obtains the palm orientation of the user's hand, spatial position information of the palm, spatial position information of the knuckles of each finger, and posture information of each finger; determines a first distance between the fingertip joint of the thumb and the fingertip joint of the index finger and a movement direction of the index finger based on the spatial position information of the knuckles of each finger; determines a second distance between the fingertip joints of the remaining fingers (except the thumb) and the palm based on the spatial position information of the palm and the spatial position information of the knuckles of each finger; determines whether the user is currently in a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determines whether the user is currently in a grip state based on the second distance and the posture information of each finger; and determines whether the user is currently in a stab state based on the movement direction of the index finger and the index finger orientation in the posture information; if the user is currently in one of the pinch state, grip state, and finger stab state, and the virtual operation object currently meets the field of view restriction, generates an interaction instruction corresponding to the one of the states, so as to perform an interaction operation corresponding to the interaction instruction on the virtual operation object. In this way, when performing gesture interaction in the XR environment, multi-dimensional constraints such as the distance between fingertips, the distance between fingertips and palms, finger posture, finger movement direction and orientation, palm constraint direction when pinching virtual operation objects, and field of view limitations of virtual operation objects are comprehensively considered. Through multi-dimensional constraints, the detection accuracy of pinching, holding, and poking gestures is improved, the false touch rate is reduced, the user experience and immersion are enhanced, and the operational efficiency of gesture interaction is improved.

[0008] In a possible design, obtaining the spatial position information of the palm and the spatial position information of the knuckles of each finger includes: Obtaining the spatial coordinates of the palm detected by the detection device in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system; The spatial coordinates of the palm in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system are converted into coordinates in the local coordinate system where the virtual operation object is located, and the spatial position information of the palm in the local coordinate system and the spatial position information of the knuckles of each finger in the local coordinate system are obtained.

[0009] In one possible design, determining whether the finger is currently in a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched includes: determining whether the first distance is less than a first preset distance, and whether the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched; If the first distance is less than a first preset distance, and the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched, it is determined that the finger is currently in the pinch state; otherwise, it is determined that the finger is not currently in the pinch state.

[0010] In one possible design, determining whether the user is currently in a gripping state based on the second distance and the posture information of each finger includes: Determining whether the second distance is less than a second preset distance and whether the posture information of each finger is consistent with the preset posture information; If the second distance is smaller than a second preset distance, and the posture information of each finger is consistent with the preset posture information, it is determined that the current state is in a gripping state; otherwise, it is determined that the current state is not in a gripping state.

[0011] In one possible design, the posture information includes finger orientation and / or finger curvature.

[0012] In one possible design, determining whether the finger is currently in a poking state based on the index finger movement direction and the index finger orientation in the gesture information includes: Determining whether the moving direction of the index finger is toward the virtual operation object, and whether the direction of the index finger in the posture information is toward the virtual operation object; If the index finger moves toward the virtual operation object and the index finger in the gesture information points toward the virtual operation object, it is determined that the current state is the finger poking state; otherwise, it is determined that the current state is not the finger poking state.

[0013] In one possible design, if the judgment result is that the user is currently in one of the states of a pinching state, a holding state, and a poking state, and the virtual operation object currently meets the field of view restriction, then generating an interaction instruction corresponding to the one of the states includes: If the judgment result is that the current state is one of the finger pinching state, holding state and finger poking state, the virtual operation object currently meets the field of view limitation, and the interactive operations allowed by the virtual operation object match the interactive operations corresponding to one of the states, then an interactive instruction corresponding to one of the states is generated.

[0014] In a second aspect, the present invention provides a gesture interaction device in an XR environment, comprising: an acquisition unit, configured to acquire the palm orientation of the user's hand, spatial position information of the palm, spatial position information of the knuckles of each finger, and posture information of each finger; a first determining unit, configured to determine a first distance between a fingertip joint of the thumb and a fingertip joint of the index finger and a moving direction of the index finger based on spatial position information of the finger joints of the fingers; a second determining unit, configured to determine a second distance between the palm and the fingertips of the fingers except the thumb based on the spatial position information of the palm and the spatial position information of the knuckles of the fingers; a determination unit, configured to determine whether the current state is a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determine whether the current state is a hold state based on the second distance and the posture information of each finger; and determine whether the current state is a poke state based on the index finger movement direction and the index finger orientation in the posture information; A generation unit is configured to generate an interaction instruction corresponding to one of the states if a judgment result shows that the virtual operation object is currently in one of the states of finger pinching, holding, and poking, and the virtual operation object currently meets the field of view limitation, so as to perform an interaction operation corresponding to the interaction instruction on the virtual operation object.

[0015] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a transceiver that are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the gesture interaction method in the XR environment as described in the first aspect or any possible design of the first aspect.

[0016] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the gesture interaction method in the XR environment described in the first aspect or any possible design of the first aspect is executed.

[0017] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the gesture interaction method in an XR environment as described in the first aspect or any possible design of the first aspect.

[0018] Beneficial effects: The gesture interaction method, device, electronic device and product in the XR environment provided by the present invention can comprehensively consider multi-dimensional constraints such as fingertip distance, fingertip-to-palm distance, finger posture, finger movement direction and orientation, palm constraint direction when pinching virtual operation objects, and field of view limitations of virtual operation objects when performing gesture interaction in the XR environment. This can improve the detection accuracy of pinching, holding, and poking gestures through multi-dimensional constraints, reduce the false touch rate, enhance user experience and immersion, improve the operational efficiency of gesture interaction, and facilitate practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a gesture interaction method in an XR environment provided in an embodiment of the present application; Figure 2 A schematic block diagram of a gesture interaction device in an XR environment provided in an embodiment of the present application; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0021] It should be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments of the present invention.

[0022] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may indicate three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this document describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B may indicate two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0023] In order to improve the accuracy of gesture interaction, the embodiments of the present application provide a gesture interaction method, device, electronic device and product in an XR environment. The gesture interaction method, device, electronic device and product in the XR environment can improve the detection accuracy of pinching, holding and poking gestures, reduce the false touch rate, and enhance the user experience and immersion.

[0024] The gesture interaction method in the XR environment provided in the embodiment of the present application can be applied to VR devices, AR devices, or MR devices, etc. It is understood that the execution entity does not constitute a limitation on the embodiment of the present application.

[0025] like Figure 1 , which is a flowchart of the gesture interaction method in the XR environment provided by the first aspect of this embodiment. The gesture interaction method in the XR environment may include, but is not limited to, the following steps S101 to S105.

[0026] Step S101: Obtain the palm direction of the user's hand, the spatial position information of the palm, the spatial position information of the knuckles of each finger, and the posture information of each finger.

[0027] In an embodiment of the present application, the palm orientation of the user's hand, the spatial position information of the palm, the spatial position information of the knuckles of each finger, and the posture information of each finger can be obtained based on visual recognition or sensor recognition technology. For example, the palm orientation of the user's hand can be obtained by using an image acquisition device to obtain an image of the user's hand, and the palm orientation of the user's hand can be identified by image recognition technology, or the spatial position information of the palm and back of the user's hand can be detected by a sensor, and the palm orientation of the user's hand can be determined based on the spatial position information of the palm and back of the hand. For another example, the position of the finger joints can be directly detected by the inertial measurement unit IMU, bending sensor or electromagnetic / optical tracker built into the data glove, and the posture information of each finger can be determined based on the position of the finger joints.

[0028] The posture information may include, but is not limited to, finger direction and / or finger curvature.

[0029] In the embodiment of the present application, the spatial position information of the palm and the spatial position information of the knuckles of each finger may refer to the spatial position information of the palm in the local coordinate system where the virtual operation object is located and the spatial position information of the knuckles of each finger in the local coordinate system where the virtual operation object is located. When obtaining the spatial position information of the palm and the spatial position information of the knuckles of each finger, the spatial coordinates of the palm in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system detected by the detection device may be first obtained, and then the spatial coordinates of the palm in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system may be converted into coordinates in the local coordinate system where the virtual operation object is located, thereby obtaining the spatial position information of the palm in the local coordinate system and the spatial position information of the knuckles of each finger in the local coordinate system.

[0030] Step S102: Based on the spatial position information of the knuckles of each finger, determine a first distance between the fingertip joint of the thumb and the fingertip joint of the index finger and a moving direction of the index finger.

[0031] The first distance between the fingertip joint of the thumb and the fingertip joint of the index finger can be calculated based on the spatial position information of the fingertip joint of the thumb and the spatial position information of the fingertip joint of the index finger. For example, the coordinates of the fingertip joint of the thumb in the local coordinate system are (x1, y1, z1), and the coordinates of the fingertip joint of the index finger in the local coordinate system are (x2, y2, z2), then the first distance between the fingertip joint of the thumb and the fingertip joint of the index finger can be expressed as .

[0032] The movement direction of the index finger can be calculated based on the spatial position information of the finger joint detected at multiple consecutive time points.

[0033] Step S103: Based on the spatial position information of the palm and the spatial position information of the knuckles of each finger, determine the second distance between the fingertip joints of the remaining fingers except the thumb and the palm.

[0034] Specifically, the second distance between the fingertip joints of the fingers other than the thumb and the palm can be calculated based on the spatial position information of the fingers other than the thumb (i.e., the index finger, middle finger, ring finger and little finger) and the spatial position information of the palm. The calculation process of the second distance is consistent with the calculation process of the first distance in the aforementioned step S102, and will not be repeated here.

[0035] Step S104. Determine whether the current state is a pinching state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determine whether the current state is a holding state based on the second distance and the posture information of each finger; and determine whether the current state is a poking state based on the index finger movement direction and the index finger orientation in the posture information.

[0036] Specifically, judging whether the finger is currently in the pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched may include, but is not limited to, the following steps S1041-S1042.

[0037] S1041. Determine whether the first distance is less than a first preset distance, and whether the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched.

[0038] The first preset distance can be set according to actual conditions. The palm constraint direction when pinching the virtual operation object refers to the direction of the palm when pinching the virtual operation object with the thumb and index finger.

[0039] S1042. If the first distance is less than a first preset distance, and the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched, determine that the finger is currently in the pinch state; otherwise, determine that the finger is not currently in the pinch state.

[0040] If the first distance is less than a first preset distance and the palm orientation is the same as the palm constraint direction when pinching the virtual operation object, it is determined that the current state is a pinch-in state. If the first distance is greater than or equal to the first preset distance and / or the palm orientation is different from the palm constraint direction when pinching the virtual operation object, it is determined that the current state is not a pinch-in state.

[0041] Determining whether the current state is a grip state based on the second distance and the posture information of each finger may include, but is not limited to, the following steps S1043-S1044.

[0042] Step S1043: Determine whether the second distance is less than a second preset distance, and whether the posture information of each finger is consistent with the preset posture information.

[0043] The second preset distance may be set according to actual conditions. The preset posture information may refer to the posture information of each finger when the hand is held normally.

[0044] Step S1044: If the second distance is less than the second preset distance, and the posture information of each finger is consistent with the preset posture information, it is determined that the current state is in a holding state; otherwise, it is determined that the current state is not in a holding state.

[0045] If the second distance is less than a second preset distance and the posture information of each finger is consistent with the preset posture information, it is determined that the current state is a hold state. If the second distance is greater than or equal to the second preset distance and / or the posture information of each finger is consistent with the preset posture information, it is determined that the current state is not a hold state.

[0046] Based on the index finger movement direction and the index finger orientation in the gesture information, determining whether the finger is currently in a poking state may include, but is not limited to, the following steps S1045-S1046.

[0047] Step S1045: Determine whether the moving direction of the index finger is toward the virtual operation object, and whether the direction of the index finger in the gesture information is toward the virtual operation object.

[0048] Step S1046: If the index finger moves toward the virtual operation object, and the index finger in the gesture information points toward the virtual operation object, it is determined that the current state is the finger poking state; otherwise, it is determined that the current state is not the finger poking state.

[0049] If the index finger moves toward the virtual operation object and the index finger direction in the gesture information points to the virtual operation object, it is determined that the current state is a finger-poking state. If the index finger moves not toward the virtual operation object and / or the index finger direction in the gesture information does not point to the virtual operation object, it is determined that the current state is not a finger-poking state.

[0050] Step S105. If the judgment result is that the current state is one of the finger pinching state, the gripping state and the finger poking state, and the virtual operation object currently meets the field of view limitation, then generate an interaction instruction corresponding to one of the states to perform the interaction operation corresponding to the interaction instruction on the virtual operation object.

[0051] In XR gesture interaction, Field of View Constraint (FOV) ensures the effectiveness of finger taps through mechanisms such as frustum culling, occlusion testing, or dynamic FOV adaptation. This requires that virtual objects must be within the user's current visual range to avoid erroneous operations or invalid interactions caused by virtual objects being out of view or obscured. This is prior art and will not be further explained in detail in this application.

[0052] In actual situations, the interactive operations allowed by a virtual operation object may only include one or two of the following: pinching, holding, and poking. For example, for some large virtual operation objects, pinching is not possible. Therefore, in one or more embodiments, when generating an interaction instruction, if the result of the judgment is that the virtual operation object is currently in one of the states of pinching, holding, and poking, the virtual operation object currently meets the field of view restriction, and the interactive operations allowed by the virtual operation object match the interactive operations corresponding to one of the states, then an interaction instruction corresponding to one of the states will be generated.

[0053] The present invention provides a gesture interaction method in an XR environment, which obtains the palm orientation of a user's hand, spatial position information of the palm, spatial position information of the knuckles of each finger, and posture information of each finger; determines a first distance between the fingertip joint of the thumb and the fingertip joint of the index finger and a movement direction of the index finger based on the spatial position information of the knuckles of each finger; determines a second distance between the fingertip joints of the remaining fingers (except the thumb) and the palm based on the spatial position information of the palm and the spatial position information of the knuckles of each finger; determines whether the user is currently in a pinch state based on the first distance, the palm orientation, and the palm constraint direction when pinching a virtual operation object; determines whether the user is currently in a grip state based on the second distance and the posture information of each finger; and determines whether the user is currently in a stab state based on the movement direction of the index finger and the index finger orientation in the posture information; and if the user is currently in one of the pinch state, grip state, and finger stab state, and the virtual operation object currently meets the field of view restriction, generates an interaction instruction corresponding to the one of the states, so as to perform an interaction operation corresponding to the interaction instruction on the virtual operation object. In this way, when performing gesture interaction in the XR environment, multi-dimensional constraints such as fingertip distance, distance between fingertip and palm, finger posture, finger movement direction and orientation, palm constraint direction when pinching virtual operation objects, and field of view limitations of virtual operation objects are comprehensively considered. Therefore, multi-dimensional constraints are used to improve the detection accuracy of pinching, holding, and poking gestures, reduce the false touch rate, enhance user experience and immersion, improve the operational efficiency of gesture interaction, and facilitate practical application and promotion.

[0054] See also Figure 2 In a second aspect, an embodiment of the present application provides a gesture interaction device in an XR environment. The gesture interaction device in the XR environment includes: an acquisition unit, configured to acquire the palm orientation of the user's hand, spatial position information of the palm, spatial position information of the knuckles of each finger, and posture information of each finger; a first determining unit, configured to determine a first distance between a fingertip joint of the thumb and a fingertip joint of the index finger and a moving direction of the index finger based on spatial position information of the finger joints of the fingers; a second determining unit, configured to determine a second distance between the palm and the fingertips of the fingers except the thumb based on the spatial position information of the palm and the spatial position information of the knuckles of the fingers; a determination unit, configured to determine whether the current state is a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determine whether the current state is a hold state based on the second distance and the posture information of each finger; and determine whether the current state is a poke state based on the index finger movement direction and the index finger orientation in the posture information; A generation unit is configured to generate an interaction instruction corresponding to one of the states if a judgment result shows that the virtual operation object is currently in one of the states of finger pinching, holding, and poking, and the virtual operation object currently meets the field of view limitation, so as to perform an interaction operation corresponding to the interaction instruction on the virtual operation object.

[0055] The working process, working details and technical effects of the gesture interaction device in the XR environment provided in the second aspect of this embodiment can be found in the first aspect of the embodiment and will not be repeated here.

[0056] like Figure 3 As shown, the third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the gesture interaction method in the XR environment as described in the first aspect of the embodiment.

[0057] For example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series, an ARM (Advanced RISC Machines), an X86 or other architecture processor, or a processor with an integrated NPU (neural-network processing units); the transceiver may include, but is not limited to, a WiFi (Wireless Fidelity) wireless transceiver, a Bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc.

[0058] A fourth aspect of this embodiment provides a computer-readable storage medium storing instructions including the gesture interaction method in an XR environment described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, execute the gesture interaction method in an XR environment described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable device.

[0059] A fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the gesture interaction method in an XR environment as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0060] It should be understood that certain details are provided in the following description to facilitate a thorough understanding of the example embodiments. However, one of ordinary skill in the art will appreciate that the example embodiments can be practiced without these specific details. For example, a system may be shown in block diagrams to avoid obscuring the example with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the example embodiments.

[0061] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A gesture interaction method in an XR environment, characterized in that: include: Obtain the palm orientation of the user's hand, the spatial position information of the palm, the spatial position information of the knuckles of each finger, and the posture information of each finger; Based on the spatial position information of the knuckles of each finger, determining a first distance between the fingertip joint of the thumb and the fingertip joint of the index finger and a moving direction of the index finger; Determining a second distance between the palm and the fingertips of the fingers other than the thumb based on the spatial position information of the palm and the spatial position information of the knuckles of the fingers; Determining whether the current state is a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determining whether the current state is a hold state based on the second distance and the posture information of each finger; and determining whether the current state is a poke state based on the index finger movement direction and the index finger orientation in the posture information; If the judgment result is that the current state is one of the finger pinching state, holding state and finger poking state, and the virtual operation object currently meets the field of view limitation, then an interaction instruction corresponding to one of the states is generated to perform the interaction operation corresponding to the interaction instruction on the virtual operation object.

2. The gesture interaction method in an XR environment according to claim 1, characterized in that: Obtain the spatial position information of the palm and the spatial position information of each finger joint, including: Obtaining the spatial coordinates of the palm detected by the detection device in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system; The spatial coordinates of the palm in the world coordinate system and the spatial coordinates of the knuckles of each finger in the world coordinate system are converted into coordinates in the local coordinate system where the virtual operation object is located, and the spatial position information of the palm in the local coordinate system and the spatial position information of the knuckles of each finger in the local coordinate system are obtained.

3. The gesture interaction method in an XR environment according to claim 1, characterized in that: The determining whether the finger is currently in a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched includes: determining whether the first distance is less than a first preset distance, and whether the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched; If the first distance is less than a first preset distance, and the palm orientation is the same as the palm constraint direction when the virtual operation object is pinched, it is determined that the finger is currently in the pinch state; otherwise, it is determined that the finger is not currently in the pinch state.

4. The gesture interaction method in an XR environment according to claim 1, characterized in that: The determining whether the current state is in a gripping state based on the second distance and the posture information of each finger includes: Determining whether the second distance is less than a second preset distance and whether the posture information of each finger is consistent with the preset posture information; If the second distance is smaller than the second preset distance, and the posture information of each finger is consistent with the preset posture information, it is determined that the current state is in a gripping state; otherwise, it is determined that the current state is not in a gripping state.

5. The gesture interaction method in an XR environment according to claim 4, characterized in that: The posture information includes finger orientation and / or finger curvature.

6. The gesture interaction method in an XR environment according to claim 5, characterized in that: The determining whether the finger is currently in a poking state based on the index finger movement direction and the index finger orientation in the gesture information includes: Determining whether the moving direction of the index finger is toward the virtual operation object, and whether the direction of the index finger in the posture information is toward the virtual operation object; If the index finger moves toward the virtual operation object and the index finger in the gesture information points toward the virtual operation object, it is determined that the current state is the finger poking state; otherwise, it is determined that the current state is not the finger poking state.

7. The gesture interaction method in an XR environment according to claim 1, characterized in that: If the judgment result is that the current state is one of the finger pinching state, the gripping state, and the finger poking state, and the virtual operation object currently meets the field of view restriction, then generating an interaction instruction corresponding to the one of the states includes: If the judgment result is that the current state is one of the finger pinching state, holding state and finger poking state, the virtual operation object currently meets the field of view limitation, and the interactive operations allowed by the virtual operation object match the interactive operations corresponding to one of the states, then an interactive instruction corresponding to one of the states is generated.

8. A gesture interaction device in an XR environment, characterized in that: include: an acquisition unit, configured to acquire the palm orientation of the user's hand, spatial position information of the palm, spatial position information of the knuckles of each finger, and posture information of each finger; a first determining unit, configured to determine a first distance between a fingertip joint of the thumb and a fingertip joint of the index finger and a moving direction of the index finger based on spatial position information of the finger joints of the fingers; a second determining unit, configured to determine a second distance between the palm and the fingertips of the fingers except the thumb based on the spatial position information of the palm and the spatial position information of the knuckles of the fingers; a determination unit, configured to determine whether the current state is a pinch state based on the first distance, the palm orientation, and the palm constraint direction when the virtual operation object is pinched; determine whether the current state is a hold state based on the second distance and the posture information of each finger; and determine whether the current state is a poke state based on the index finger movement direction and the index finger orientation in the posture information; A generation unit is configured to generate an interaction instruction corresponding to one of the states if a judgment result shows that the virtual operation object is currently in one of the states of finger pinching, holding, and poking, and the virtual operation object currently meets the field of view limitation, so as to perform an interaction operation corresponding to the interaction instruction on the virtual operation object.

9. An electronic device, characterized in that: The device comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the gesture interaction method in the XR environment as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that When executed by a computer, the computer program or the instruction implements the gesture interaction method in an XR environment as described in any one of claims 1 to 7.

Images