A head-mounted gesture interaction device design method for 3D large screens

Abstract

The application discloses a kind of head-mounted gesture interaction device design methods for 3D large screen.The interaction device includes positioning module, gesture recognition module, 3D glasses and large screen terminal display module.Positioning module and gesture recognition module are fixed on 3D glasses, and positioning module is used to determine the view point coordinates of user in interactive space.Large screen terminal display module receives the view point coordinates and hand joint data provided by positioning module and gesture recognition module, and the 3D virtual reality scene content required by user view point is shown through screen.The 3D glasses are used to watch the content shown on screen.The interaction device binds positioning module, gesture recognition module and user's view point, and realizes synchronous hand tracking under dynamic view point.The pose of gesture recognition module is determined by the design method, and under the premise of not affecting effective interactive space, the factors affecting recognition accuracy, such as gesture occlusion, are weakened to the greatest extent, the flexibility and recognition accuracy are improved, and the cost is reduced.

Description

Technical Field

[0001] This application belongs to the field of virtual reality technology and relates to the design of gesture interaction devices in large-screen terminal scenarios, specifically to a design method for a head-mounted gesture interaction device for 3D large screens. Background Technology

[0002] In large-screen terminal scenarios, gesture recognition-based human-computer interaction technology has become a research hotspot. Its typical workflow includes two processes: user localization and hand information acquisition. User localization requires a positioning device to acquire the user's position coordinates and head orientation information in real space, mapping this information onto the virtual scene to determine the position and orientation of the virtual viewpoint, thus providing a perspective experience matched to the user's position. Hand information acquisition includes hand tracking and gesture recognition. Traditional solutions typically achieve these two functions through an externally installed hand tracking system composed of multiple hand tracking module arrays. The hand tracking system captures the user's hand movements and positions, analyzes and recognizes the gestures using algorithms, and converts the gesture signals into interactive commands to control the virtual reality scene on the large-screen display wall. The cooperation between the positioning device and the hand tracking system provides users with a natural way to interact.

[0003] However, existing VR interaction solutions for large-screen terminals typically install hand tracking systems in fixed locations, such as on a desktop or around the perimeter of the large-screen terminal. This leads to two main problems. First, the hand tracking range is limited; the user's hand movements may exceed the effective range of the tracking system, resulting in untrackable movements or inaccurate recognition due to hand occlusion. Second, because finger joints are small and lack prominent feature markers, gesture recognition accuracy is affected, and the accuracy and reliability of finger positioning decrease significantly. This necessitates hand tracking systems with a larger and more flexible detection range and better recognition accuracy; however, this also complicates device installation and system configuration, limiting the flexible deployment of the system.

[0004] Some solutions propose integrating the positioning device and hand tracking module into the user's viewpoint terminal device. This allows for flexible capture of the user's hand information as the body moves, overcoming the limitations of fixed-position hand tracking systems. This ensures that hand movements in virtual scenes are dynamically consistent with those in real-world scenarios, freeing hand tracking from the constraints of body movement. Compared to traditional multi-tracking module systems, this approach is more lightweight and better suited for gesture recognition needs in dynamic scenarios.

[0005] However, the pose of the hand tracking module when installed on the viewpoint device directly affects the final hand tracking results, thus impacting the user experience. Without optimizing the hand tracking module's pose based on the range of hand gestures, common gestures may not fall within the module's effective recognition range, leading to low-confidence tracking. In actual interaction, this can cause some common gestures to fail to achieve stable and accurate recognition, thereby affecting the smoothness of the interaction and the user experience. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a design method for a head-mounted gesture interaction device for 3D large screens. It binds the positioning module and gesture recognition module to the user's viewpoint and proposes a specific installation method for the gesture recognition module. By optimizing the optimal installation posture of the gesture recognition module, the overlap between the gesture range that reaches the confidence threshold and the range of commonly used gestures is maximized, minimizing factors affecting recognition accuracy such as gesture occlusion. Based on the dual objectives of effective interaction space and recognition effect, this invention improves the flexibility of the interaction system and the accuracy of gesture recognition without compromising the effective interaction space, while reducing costs.

[0007] A design method for a head-mounted gesture interaction device for 3D large screens determines the installation position of the gesture recognition module through the following steps:

[0008] Step 1: Calculate the coverage area of ​​the effective interaction space S.

[0009] The effective interaction space S is defined by the recognizable range S of the gesture recognition module. S The reachable range S of gesture interaction H Visible range S E Joint decision:

[0010] S = S S ∩S E ∩S H

[0011]

[0012] in, These represent the recognizable range S of the gesture recognition module. S Horizontal field of view H FOV Left and right end values, For the vertical field of view V FOV The upper and lower values, r S This represents the maximum detection radius of the gesture recognition module. These represent the visible range S of the gesture interaction. E Horizontal field of view H FOV Left and right end values, For the vertical field of view VFOV The upper and lower values. S represents the reachable range of the gesture interaction. H Horizontal field of view H FOV Left and right end values, For the vertical field of view V FOV The upper and lower values, r H This refers to the length of the upper limb.

[0013] Consider adjusting the pose of the gesture recognition module in the vertical direction, simplifying the human body structure model in the vertical direction, unifying the viewpoint and the gesture recognition module to the same position, establishing a two-dimensional Cartesian coordinate system with the viewpoint position as the origin, and analyzing the vertical rotation angle. The optimal vertical rotation pose for the gesture recognition module.

[0014] Step 2: Design the confidence function

[0015] Choosing the cosine function as the basis function for the confidence function, the confidence function is designed as follows:

[0016] confidence = 0.35 * cos(2θ) c +0.65

[0017] Where, θ c θ represents the camera normal direction and the palm normal direction of the gesture recognition module. h The included angle.

[0018] Step 3: Design constraints based on confidence thresholds

[0019] Set the desired minimum confidence level C, and design the confidence level to satisfy the following constraints:

[0020] confidence = 0.35 * cos(2θ) c )+0.65≥C

[0021] Step 4: Design constraints based on gesture range

[0022] Consider the actual range of motion of commonly used gestures in the vertical field of view. The palm normal is in the vertical direction θ h The following constraints are proposed:

[0023]

[0024] Step 5: Joint Constraints

[0025] The constraints based on confidence thresholds are expanded and solved:

[0026]

[0027] θ c =|-θ-θ h |=|θ+θ h |

[0028]

[0029] The constraints on the range of the joint gestures yield the following relationship:

[0030]

[0031] Step 6: Constructing the objective function and optimizing the solution

[0032] The optimization objective is to maximize the gesture range width that achieves confidence level C. The objective function is as follows:

[0033]

[0034] Solve the above objective function, and set the angle between the camera normal of the gesture recognition module and the horizontal plane to θ. * .

[0035] A head-mounted gesture interaction device for 3D large screens includes a positioning module, a gesture recognition module, 3D glasses, and a large-screen terminal display module. The positioning module and gesture recognition module are fixed to the 3D glasses. The positioning module acquires the user's position and head orientation information to determine the user's viewpoint coordinates in the interaction space. The large-screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning and gesture recognition modules and displays the 3D virtual reality scene content required by the user's viewpoint on the screen. The 3D glasses are used to view the content displayed on the screen.

[0036] Using the aforementioned head-mounted gesture interaction device for 3D large screens, synchronous hand tracking under the user's dynamic viewpoint is achieved, specifically including the following steps:

[0037] Step 1: Obtain hand joint data.

[0038] Collect joint and pose information of the user's hand to accurately represent the real-time state of the hand. Use the gesture recognition module to acquire the joint data of the user's hand, representing the relative position and pose of the hand under the gesture recognition module. Use the camera intrinsic parameter matrix K1 of the gesture recognition module and the rigid body transformation matrix [R1|T1] between the real-world coordinate system and the coordinate system of the hand recognition module to transform the hand tracking information H in the real-world coordinate system into the hand tracking information H in the coordinate system of the gesture recognition module. s :

[0039] H s=K1[R1|T1]H

[0040] Step 2: Obtain hand tracking information H from the viewpoint e .

[0041] Hand data is adjusted based on the user's perspective to ensure that the relationship between virtual gestures and the viewpoint in the virtual scene is consistent with that in the real scene. The relationship between the gesture recognition module and the viewpoint is determined by the installation method of the gesture recognition module on the viewpoint device. The rigid body transformation matrix [R2|T2] between the real gesture recognition module coordinate system and the viewpoint coordinate system is used to transform the hand tracking information H in the gesture recognition module coordinate system. s Hand tracking information H converted to viewpoint coordinates e :

[0042] H e =[R2|T2]H s

[0043] Step 3: Implement dynamic viewpoint-synchronized hand tracking in real-world scenarios.

[0044] Due to the unrestricted free movement of users in interactive scenarios and the dynamic changes in viewpoint position, a positioning module is used to acquire the user's position and head orientation information to determine the user's viewpoint coordinates within the interactive scenario, ensuring consistent viewpoint mapping between the virtual and real worlds during interaction. The camera intrinsic parameter matrix K2 of the positioning module and the rigid body transformation matrix [R3|T3] between the viewpoint coordinate system and the interactive scene coordinate system are used to transfer the hand tracking information H in the viewpoint coordinate system. e Transform into hand tracking information H in scene coordinate system p :

[0045] H p =K2[R3|T3]H e

[0046] Therefore, when users move freely without restrictions in interactive scenarios, the synchronized hand tracking information H under dynamic viewpoints... p for:

[0047] H p =K2[R3|T3][R2|T2]K1[R1|T1]H

[0048] Step 4: Real-world mapping of hand data.

[0049] A virtual space corresponding one-to-one with the real scene is constructed, and a virtual viewpoint that moves synchronously with the user's viewpoint is set. Hand tracking data in the coordinate system of the real interaction scene is applied to the virtual scene, and the synchronous hand tracking information H under the dynamic viewpoint is used by the function f. p Hand tracking information H mapped to the virtual scene coordinate systemv :

[0050] H v =f(H p )

[0051] Step 5: Drawing virtual gestures from a virtual viewpoint.

[0052] Due to the mutual mapping between real and virtual spaces, the position and posture of the virtual hand under the virtual viewpoint can be reconstructed by utilizing the relationship between the real viewpoint coordinate system and the real interactive scene coordinate system [R3|T3], i.e., the virtual gesture drawing information H under the virtual viewpoint coordinate system. c This ensures that the gestures accurately reflect the user's actual actions.

[0053] H c =[R3|T3] -1 H v

[0054] Step 6: Implement human-computer interaction.

[0055] On large-screen terminals or other virtual display devices, the user's hand movements are displayed in real time, and feedback is provided to the user by recognizing gestures to control virtual objects or trigger interactive operations, thus achieving human-computer interaction under dynamic viewpoints.

[0056] The present invention has the following beneficial effects:

[0057] 1. The interactive device overcomes the limitations of the detection range caused by the fixed position of traditional hand tracking systems, so that hand tracking is no longer restricted by human movement, realizing synchronous hand tracking under dynamic viewpoints. It can flexibly capture user hand information, reduce the deviation of hand position, avoid tracking errors, and ensure that hand movements in virtual scenes are dynamically consistent with hand movements in real scenes.

[0058] 2. The installation method of the gesture recognition module is based on the effective interaction space and recognition effect. Under the premise of ensuring that the effective interaction space is not affected, the pose of the gesture recognition module is optimized to maximize the overlap between the gesture range that meets the confidence threshold and the range of commonly used gestures, thereby determining the optimal installation pose of the gesture recognition module, significantly improving the robustness of the overall interaction and user experience, and thus meeting the usage needs in diverse scenarios. Attached Figure Description

[0059] Figure 1 A head-mounted gesture interaction device designed for 3D large screens;

[0060] Figure 2 This is a schematic diagram of the integrated connection module;

[0061] Figure 3This is a schematic diagram illustrating the relationship between sensor normals and recognition performance.

[0062] Figure 4 This is a schematic diagram of the camera normal direction for the gesture recognition module;

[0063] Figure 5 A simplified diagram of a human body model in the vertical direction;

[0064] Figure 6 This is a schematic diagram of a two-dimensional rectangular coordinate system in the embodiment. Detailed Implementation

[0065] The present invention will be further explained below with reference to the accompanying drawings;

[0066] like Figure 1 As shown, a head-mounted gesture interaction device for 3D large screens includes a positioning module 1, a gesture recognition module 2, 3D glasses 3, and a large-screen terminal display module. The positioning module 1 and gesture recognition module 2 are fixed to the 3D glasses 3 via an integrated connection module 4. The positioning module 1 is used to acquire the user's position and head orientation information to determine the user's viewpoint coordinates in the interaction space. The gesture recognition module 2 is used to acquire the user's hand joint data. The large-screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning module 1 and gesture recognition module 2, and displays the 3D virtual reality scene content required by the user's viewpoint on the screen. The 3D glasses 3 are used to view the content displayed on the screen.

[0067] like Figure 2 As shown, the shape design of the integrated connection module 4 is optimized based on the best installation posture of the gesture recognition module, thereby improving the stability and accuracy of gesture recognition. It has a positioning module interface 5, a gesture recognition module interface 6 and a 3D glasses interface 7.

[0068] Using a head-mounted gesture interaction device designed for 3D large screens, synchronous hand tracking is achieved under the user's dynamic viewpoint, specifically including the following steps:

[0069] Step 1: Obtain hand joint data.

[0070] The gesture recognition module acquires the user's hand joint data. Using the gesture recognition module's camera intrinsic parameter matrix K1 and the rigid body transformation matrix [R1|T1] between the real-world coordinate system and the hand recognition module's coordinate system, the hand tracking information H in the real-world coordinate system is transformed into hand tracking information H in the gesture recognition module's coordinate system. s :

[0071] H s =K1[R1|T1]H

[0072] Step 2: Obtain hand tracking information H from the viewpointe .

[0073] By determining the installation method of the gesture recognition module on the viewpoint device, the relationship between the gesture recognition module and the viewpoint is established. The rigid body transformation matrix [R2|T2] between the actual gesture recognition module coordinate system and the viewpoint coordinate system is used to transform the hand tracking information H in the gesture recognition module coordinate system. s Hand tracking information H converted to viewpoint coordinates e :

[0074] H e =[R2|T2]H s

[0075] Step 3: Implement dynamic viewpoint-synchronized hand tracking in real-world scenarios.

[0076] The hand tracking information H in the viewpoint coordinate system is generated using the camera intrinsic parameter matrix K2 of the positioning module and the rigid body transformation matrix [R3|T3] between the viewpoint coordinate system and the interactive scene coordinate system. e Transform into hand tracking information H in scene coordinate system p :

[0077] H p =K2[R3|T3]H e

[0078] Therefore, when users move freely without restrictions in interactive scenarios, the synchronized hand tracking information H under dynamic viewpoints... p for:

[0079] H p =K2[R3|T3][R2|T2]K1[R1|T1]H

[0080] Step 4: Real-world mapping of hand data.

[0081] A virtual space corresponding one-to-one with the real scene is constructed, and a virtual viewpoint that moves synchronously with the user's viewpoint is set. Hand tracking data in the coordinate system of the real interaction scene is applied to the virtual scene, and the synchronous hand tracking information H under the dynamic viewpoint is used by the function f. p Hand tracking information H mapped to the virtual scene coordinate system v :

[0082] H v =f(H p )

[0083] Step 5: Drawing virtual gestures from a virtual viewpoint.

[0084] By utilizing the relationship between the real viewpoint coordinate system and the real interactive scene coordinate system [R3|T3], the virtual gesture drawing information H in the virtual viewpoint coordinate system is reconstructed in reverse. c :

[0085] H c =[R3|T3] -1 H v

[0086] Step 6: Implement human-computer interaction.

[0087] On large-screen terminals or other virtual display devices, the user's hand movements are displayed in real time, and feedback is provided to the user by recognizing gestures to control virtual objects or trigger interactive operations, thus achieving human-computer interaction under dynamic viewpoints.

[0088] Since the mounting position of the gesture recognition module on the 3D glasses directly affects the gesture tracking results, the smaller the angle between the palm normal and the principal optical axis normal of the gesture recognition module's camera (i.e., the closer the palm normal and the principal optical axis normal of the gesture recognition module are to being parallel), the more directly the palm faces the camera. This results in a larger camera observation area, less occlusion, and more information acquisition, leading to higher recognition accuracy, higher confidence, and more accurate tracking results, thus improving the interaction effect. Conversely, the closer the normal of the gesture recognition module is to being perpendicular to the normal of the palm, the lower the confidence and the less accurate the tracking results. Figure 3 As shown. Therefore, the installation of the gesture recognition module is determined by the following steps:

[0089] Step 1: Calculate the coverage area of ​​the effective interaction space S.

[0090] When changing the installation position of the gesture recognition module, the gesture recognition effect should be optimized without affecting the effective interaction space.

[0091] When interacting with a large-screen terminal using gestures, the user's gaze should be able to capture the interactive target in the virtual scene and the real hand. Hand movements outside the user's field of vision are considered invalid hand actions. Therefore, the effective interaction space S is defined by the trackable range S of the gesture recognition module. S The reachable range S of gesture interaction H Visible range S E Jointly decided:

[0092] S = S S ∩S E ∩S H

[0093] Each spatial range is defined by the horizontal field of view H. FOV Vertical field of view V FOV Determined by the spatial radius R:

[0094]

[0095] in, These represent the trackable range S of the gesture recognition module. S The left and right values ​​of the horizontal field of view. r represents the upper and lower values ​​of the vertical field of view. S This represents the maximum detection radius of the gesture recognition module. These represent the visible range S of the gesture interaction. E The left and right values ​​of the horizontal field of view. These are the upper and lower values ​​of the vertical field of view. S represents the reachable range of the gesture interaction. H The left and right values ​​of the horizontal field of view. r represents the upper and lower values ​​of the vertical field of view. H This refers to the length of the upper limb.

[0096] Adjusting the pose of the gesture recognition module horizontally can lead to a disharmony between visual and hand movements in left-right perception. For example, if the gesture recognition module rotates to the left, the eye's view remains straight ahead, but a leftward grasping motion is required, causing the hand movement to deviate from the visually forward position. This discrepancy between vision and hand movement can interfere with the intuitiveness and smoothness of the interaction. In contrast, adjusting the pose of the gesture recognition module vertically has less impact on this left-right conflict because it primarily affects the vertical adjustment of the gaze, without causing a left-right disharmony between vision and hand movements. Therefore, adjusting the pose of the gesture recognition module only in the vertical direction, assuming the head remains stationary, is preferable. Figure 4 As shown.

[0097] Based on ergonomics, the vertical human body structure model is simplified, and the viewpoint position is directly used as the location of the gesture recognition module to establish the reachable and visible range of gesture interaction, such as... Figure 5 As shown, l1, l2, and l3 represent the vertical distance between the shoulder joint and the viewpoint, the horizontal distance between the shoulder joint and the viewpoint, and the length of the upper limb, respectively. l1, l2, and l3 are determined by the human height H. θ U θ D Let represent the upper and lower field of view, respectively. A two-dimensional Cartesian coordinate system is established with the viewpoint position as the origin. Assuming the field of view of the gesture recognition module is greater than the field of view of the visible area of ​​the gesture interaction, to satisfy the condition of "not affecting the coverage of the effective interaction space," the appropriate... As the pose θ of the gesture recognition module increases, the overlap area between the detection range of the gesture recognition module and the reachable visual range of the gesture interaction in the vertical direction remains unchanged, maintaining its maximum. Therefore, the model can be further simplified, and analysis can be directly performed at the vertical rotation angle. Next, find the optimal vertical rotation pose θ of the gesture recognition module, such as... Figure 6 As shown.

[0098] Step 2: Design the confidence function

[0099] Tomas Novacek et al. proposed a confidence calculation method. First, they calculated the angle (angle) between the camera normal and the palm normal of the gesture recognition module. The confidence score was defined as the highest value of 1 when the angle was 0° or 180°, and the lowest value of 0.3 when the angle was 90°. The following confidence function was constructed to fit the relationship between the angle and the confidence score:

[0100] confidence = (0.2837 * angle) 2 )-(0.89127*angle)+1

[0101] During the rotation adjustment of the hand posture, the confidence function should have a period of 90°, which better reflects the actual movement patterns. Simultaneously, the confidence function should reach its maximum at 0° and 180°, and its minimum at 90°, requiring the derivative of the confidence function to be 0 at these key points.

[0102] Based on this, this method selects the cosine function as the basis function of the confidence function, establishes a two-dimensional rectangular coordinate system with the viewpoint position as the origin, and transforms the camera normal direction and palm normal direction of the gesture recognition module into this two-dimensional rectangular coordinate system. Wherein, θ c θ represents the angle between the camera normal and the palm normal. h This represents the angle mapped from the palm normal direction to this coordinate system. The confidence function is designed as follows:

[0103] confidence = 0.35 * cos(2θ) c +0.65

[0104] Step 3: Design constraints based on confidence thresholds

[0105] To ensure that as many interactive gestures as possible reach the desired minimum confidence level C under the installation pose θ of the gesture recognition module, the confidence level is designed to satisfy the following constraints:

[0106] confidence = 0.35 * cos(2θ) c )+0.65≥C

[0107] Step 4: Design constraints based on gesture range

[0108] Consider the actual range of motion of commonly used gestures The palm normal is in the vertical direction θ h The following constraints are proposed:

[0109]

[0110] Step 5: Joint Constraints

[0111] Without affecting the coverage of the effective interaction space

[0112] The constraints based on confidence thresholds are expanded and solved:

[0113]

[0114] The confidence threshold-based constraint reflects the range of gestures θ that can reach the confidence threshold C when the pose of the gesture recognition module is θ. h .

[0115] The constraints on the range of the joint gestures yield the following relationship:

[0116]

[0117] Step 6: Constructing the objective function and optimizing the solution

[0118] The optimization objective is to maximize the gesture range width that achieves confidence level C. The objective function is as follows:

[0119]

[0120] Solve the above objective function, and set the angle between the camera normal of the gesture recognition module and the downward rotation of the horizontal plane to θ. * .

[0121] This embodiment uses Leap Motion as the gesture recognition module, and the specific parameters are shown in Table 1:

[0122] Table 1

[0123]

[0124] Following the steps described above, substitute the parameters from Table 1 to calculate the angle θ between the camera normal of the gesture recognition module and the horizontal plane. * :

[0125]

[0126] θ * =20°

[0127] Therefore, when fixing the gesture recognition module to the 3D glasses, the downward rotation angle of the normal of the gesture recognition module relative to the horizontal plane is set to 20°.

Claims

1. A method for designing a head-mounted gesture interaction device for a 3D large screen, for determining a mounting pose of a gesture recognition module, characterized in that: Specifically comprising the following steps: Step 1, calculating the coverage of the effective interaction space S The effective interaction space S is jointly determined by the trackable range S of the hand tracking module S and the reachable range S of the gesture interaction H , the visual range S E ​ S = S S ∩ S E ∩ S H Considering adjusting the pose of the hand tracking module in the vertical direction, a two-dimensional rectangular coordinate system is established with the viewpoint position as the origin; Step 2, designing a confidence function Selecting a cosine function as the base function of the confidence function, the following confidence function confidence is designed: confidence = 0.35*cos(2θ c )+0.65 wherein θ c represents the angle between the camera normal direction θ of the hand tracking module and the palm normal direction θ h of the hand. Step 3, designing a constraint condition based on a confidence threshold Setting the desired minimum confidence level C, the confidence confidence satisfies the following constraint condition: confidence = 0.35*cos(2θ c )+0.65≥ C Step 4, designing a constraint condition based on the gesture range Consider the practical range of activity of common gestures in the vertical field of view The palm normal in the vertical direction θ h The following constraints are proposed: Step 5, joint constraint condition The constraint condition based on the confidence threshold is expanded and solved, and the constraint condition of the gesture range is combined, to obtain the following relationship: wherein Step 6, constructing the objective function and optimization solving Setting the optimization goal as maximizing the gesture range width that can reach the confidence C, the objective function is as follows: wherein a hand tracking module can track range S S lower end value of the vertical field of view angle, a visual range S for gesture interaction E lower end value of the vertical field of view angle; Solving the above objective function, set the angle between the camera normal of the hand tracking module in the gesture recognition based VR interaction device and the horizontal plane as θ * .

2. The design method of head-mounted gesture interaction device for 3D large screen according to claim 1, characterized in that: The hand tracking module's trackable range S S and the gesture interaction's reachable range S H , the visual range S E determined by the horizontal field of view H FOV , the vertical field of view V FOV and the spatial radius R: wherein, respectively represent the left and right end values of the horizontal field of view angle of the hand tracking module's trackable range S S , respectively represent the left and right end values of the horizontal field of view angle of the hand tracking module's trackable range S is the upper end value of the vertical field of view angle, r S is the maximum detection radius of the hand tracking module; respectively represent the left and right end values of the horizontal field of view angle of the hand gesture interaction's visible range S E , respectively represent the left and right end values of the horizontal field of view angle of the hand gesture interaction's visible range S is the upper end value of the vertical field of view angle; respectively represent the left and right end values of the horizontal field of view angle of the hand gesture interaction's reachable range S H , respectively represent the left and right end values of the horizontal field of view angle of the hand gesture interaction's reachable range S are the upper and lower end values of the vertical field of view angle, r H is the upper limb length.

3. The design method of head-mounted gesture interaction device for 3D large screen according to claim 1, characterized in that: The angle θ between the camera normal direction θ of the hand tracking module and the palm normal direction θ h is expressed as: c θ = arccos (cos θ · cos θ) θ c = | - θ - θ h | = | θ + θ h | 4. The design method of head-mounted gesture interaction device for 3D large screen according to claim 1, characterized in that: Setting the angle range θ of the palm method direction h ∈ [-90°, 0°], confidence threshold C = 0.

825.

5. A head-mounted gesture interaction device for 3D large screens, characterized in that: The positioning module, gesture recognition module, 3D glasses and large screen terminal display module are included; the positioning module and gesture recognition module are fixed on the 3D glasses, wherein the positioning module is used to obtain the position and head direction information of the user to determine the viewpoint coordinates of the user in the interaction space; the large screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning module and gesture recognition module, and displays the 3D virtual reality scene content required by the user's viewpoint through the screen; the 3D glasses are used to watch the content displayed on the screen; where θ is the angle between the normal of the camera of the hand tracking module relative to the horizontal plane * determined by the method of any one of claims 1 to 4.

6. The head-mounted gesture interaction device for 3D large screen according to claim 5, characterized in that: LeapMotion is used as a hand tracking module.

7. The head-mounted gesture interaction device for 3D large screen according to claim 5, wherein: The positioning module and gesture recognition module are fixed on the 3D glasses through an integrated connection module, and the integrated connection module is provided with a positioning module interface, a gesture recognition module interface and a 3D glasses interface.

8. The head-mounted gesture interaction device for 3D large screen according to claim 5, wherein: Synchronous hand tracking under dynamic viewpoint is realized by the following steps: Step 1, obtaining hand joint data; The joint data of the user's hand is acquired using the gesture recognition module, and the hand tracking information H in the real world coordinate system is converted into hand tracking information H in the gesture recognition module coordinate system using a camera intrinsic matrix K1 of the gesture recognition module and a rigid body transformation matrix [R1|T1] between the real world coordinate system and the gesture recognition module coordinate system s : H s = K1[R1|T1]H Step 2, acquire hand tracking information G under the viewpoint e ; According to the included angle θ of the normal of the hand tracking module relative to the horizontal plane * , a rigid body transformation matrix [R2|T2] between the real gesture recognition module coordinate system and the viewpoint coordinate system is determined, and the hand tracking information H s under the gesture recognition module coordinate system is converted into hand tracking information H e under the viewpoint coordinate system: H e = [R2|T2]H s Step 3, realizing dynamic viewpoint synchronous hand tracking in real scene; Using the camera intrinsic matrix K2 of the positioning module, the rigid body transformation matrix [R3|T3] between the viewpoint coordinate system and the interaction scene coordinate system, the hand tracking information H in the viewpoint coordinate system is converted into the hand tracking information H in the scene coordinate system e p :​ H p = K2[R3|T3]H e = K2[R3|T3][R2|T2]K1[R1|T1]H Step 4, virtual-real mapping of hand data; A virtual space corresponding to a real scene is constructed, and a virtual viewpoint which moves synchronously with the viewpoint of the user is set, and the synchronous hand tracking information H under the dynamic viewpoint is mapped into hand tracking information H in the virtual scene coordinate system by using a function f p v :​ H v = f(H p ) Step 5, virtual gesture drawing under virtual viewpoint; The virtual gesture drawing information H under the virtual viewpoint coordinate system is inversely reconstructed by using the relationship [R3|T3] between the real viewpoint coordinate system and the real interaction scene coordinate system c : H c = [R3|T3] -1 H v Step 6, realizing human-computer interaction; The hand action of the user is displayed in real time on the large screen terminal display module, and the virtual object is controlled or the interactive operation is triggered through gesture recognition, providing feedback for the user, achieving human-computer interaction under dynamic viewpoint.

Images