Design method of head-mounted gesture interaction device for 3D large screen

Abstract

The invention discloses a design method of a head-mounted gesture interaction device for a 3D large screen. The interaction device comprises a positioning module, a gesture recognition module, 3D glasses and a large-screen terminal display module. The positioning module and the gesture recognition module are fixed on the 3D glasses, and the positioning module is used for determining viewpoint coordinates of a user in an interaction space. The large-screen terminal display module receives viewpoint coordinates and hand joint data provided by the positioning module and the gesture recognition module, and displays 3D virtual reality scene content required by a user viewpoint through a screen. The 3D glasses are used for watching the content displayed on the screen. According to the interaction device, the positioning module and the gesture recognition module are bound with the viewpoint of the user, and synchronous hand tracking under the dynamic viewpoint is achieved. The gesture recognition module pose is determined through the design method, on the premise that an effective interaction space is not affected, factors, such as gesture shielding, influencing recognition accuracy are weakened to the maximum extent, flexibility and recognition accuracy are improved, and cost is reduced.

Description

Technical Field

[0001] The present application belongs to the field of virtual reality technology and relates to the design of a gesture interaction device in a large-screen terminal scenario, and specifically to a design method of a head-mounted gesture interaction device for a 3D large screen. Background Art

[0002] In large-screen terminal scenarios, human-computer interaction technology based on gesture recognition has become a research hotspot. Its typical workflow includes two processes: user positioning and hand information acquisition. The user positioning process requires the use of a positioning device to obtain the user's position coordinates and head direction information in the real space, and map this information to the virtual scene to determine the position and direction of the virtual viewpoint, thereby providing a viewing experience that matches the user's position. Hand information acquisition includes hand tracking and gesture recognition. Traditional solutions usually achieve these two functions through a hand tracking system composed of an array of multiple hand tracking modules installed externally. The hand tracking system captures the movement and position of the user's hand, analyzes and recognizes gestures through algorithms, and converts gesture signals into interactive commands for controlling the virtual reality scene of the large-screen display wall. The combination of the positioning device and the hand tracking system provides users with a natural way of interaction.

[0003] However, existing VR interaction solutions for large-screen terminal scenarios usually install the hand tracking system in a fixed position, such as a desktop or around a large-screen terminal, which leads to two major problems. First, the hand tracking range is limited, and the user's hand movement may exceed the effective range of the tracking system, resulting in untrackable tracking, or inaccurate recognition due to gesture occlusion. Secondly, due to the small size of the finger joints and the lack of significant feature markers, the accuracy of gesture recognition is affected, and the positioning accuracy and reliability of the fingers are significantly reduced. This requires the hand tracking system to have a larger and more flexible detection range and better recognition accuracy, but this also complicates the equipment installation and system configuration, limiting the flexible deployment of the system.

[0004] Some solutions propose to integrate the positioning device and the hand tracking module together on the user's viewpoint terminal device, which can flexibly capture the user's hand information as the body moves, getting rid of the limitations caused by the fixed position of the hand tracking system, so that hand tracking is no longer restricted by human movement, ensuring that the hand movements in the virtual scene are dynamically consistent with the hand movements in the real scene. Compared with the traditional multi-tracking module system, this solution is lighter and more suitable for gesture recognition needs in dynamic scenes.

[0005] However, the position of the hand tracking module when it is installed on the viewpoint device directly affects the final hand tracking result, which in turn affects the interactive experience. If the hand tracking module position is not optimized according to the gesture activity range, common gestures may not be within the effective recognition range of the module, resulting in low-confidence tracking results. In actual interaction, this situation will cause some common gestures to fail to achieve stable and accurate recognition results, thus affecting the smoothness of interaction and user experience. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a design method for a head-mounted gesture interaction device for a 3D large screen, which binds the positioning module, the gesture recognition module and the user's viewpoint, and proposes a specific gesture recognition module installation method. By optimizing the optimal installation position of the gesture recognition module, the overlapping area between the gesture range that can reach the confidence threshold and the commonly used gesture range is maximized, and factors that affect recognition accuracy such as gesture occlusion are weakened to the greatest extent. Based on the two goals of effective interaction space and recognition effect, the flexibility of the interaction system and the accuracy of gesture recognition are improved without affecting the effective interaction space, and the cost is reduced.

[0007] A design method for a head-mounted gesture interaction device for a 3D large screen determines the installation position of a gesture recognition module through the following steps:

[0008] Step 1: Calculate the coverage of the effective interaction space S

[0009] The effective interaction space S is composed of the recognizable range S of the gesture recognition module. S The reachable range S for gesture interaction H , Visible range S E Joint decision:

[0010] S=S S ∩S E ∩S H

[0011]

[0012] in, Respectively represent the recognizable range S of the gesture recognition module S Horizontal field of view H FOV Left and right values, is the vertical field angle V FOV The upper and lower values ​​of r S It is the maximum detection radius of the gesture recognition module. Respectively represent the visible range of gesture interaction S E Horizontal field of view H FOV Left and right values, is the vertical field angle VFOV The upper and lower values ​​of . Respectively represent the reachable range S of gesture interaction H Horizontal field of view H FOV Left and right values, is the vertical field angle V FOV The upper and lower values ​​of r H The length of upper limb.

[0013] Consider adjusting the posture of the gesture recognition module in the vertical direction, simplifying the human body structure model in the vertical direction, unifying the viewpoint and the gesture recognition module to the same position, establishing a two-dimensional rectangular coordinate system with the viewpoint position as the origin, and analyzing the vertical rotation angle. The optimal vertical rotation pose of the lower gesture recognition module.

[0014] Step 2: Design the confidence function

[0015] Select the cosine function as the basis function of the confidence function and design the following confidence function:

[0016] confidence=0.35*cos(2θ c )+0.65

[0017] Among them, θ c Represents the camera normal direction of the gesture recognition module and the palm normal direction θ h Angle.

[0018] Step 3: Design constraints based on confidence thresholds

[0019] Set the desired minimum confidence level C and design the confidence to meet the following constraints:

[0020] confidence=0.35*cos(2θ c )+0.65≥C

[0021] Step 4: Design constraints based on gesture range

[0022] Consider the actual range of common gestures in the vertical field of view The normal of the palm is in the vertical direction θ h The following constraints are proposed:

[0023]

[0024] Step 5: Combine constraints

[0025] Expand and solve the constraints based on the confidence threshold:

[0026]

[0027] θ c =|-θ-θ h |=|θ+θ h |

[0028]

[0029] Combined with the constraints of the gesture range, we get the following relationship:

[0030]

[0031] Step 6: Construct objective function and optimize solution

[0032] The optimization goal is set to maximize the width of the gesture range that can reach the confidence level C. The objective function is as follows:

[0033]

[0034] Solve the above objective function and set the angle between the camera normal of the gesture recognition module and the horizontal plane to θ * .

[0035] A head-mounted gesture interaction device for a 3D large screen includes a positioning module, a gesture recognition module, 3D glasses and a large-screen terminal display module. The positioning module and the gesture recognition module are fixed on the 3D glasses, wherein the positioning module is used to obtain the user's position and head direction information to determine the user's viewpoint coordinates in the interactive space. The large-screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning module and the gesture recognition module, and displays the 3D virtual reality scene content required by the user's viewpoint through the screen. The 3D glasses are used to view the content displayed on the screen.

[0036] Using the above-mentioned head-mounted gesture interaction device for a 3D large screen, synchronous hand tracking under a user's dynamic viewpoint is realized, which specifically includes the following steps:

[0037] Step 1: Get hand joint data.

[0038] Collect the user's hand joint and posture information to accurately represent the real-time state of the hand. Use the gesture recognition module to obtain the user's hand joint data to represent the relative position and posture of the hand under the gesture recognition module. Use the camera intrinsic parameter matrix K of the gesture recognition module 1 and the rigid body transformation matrix between the real world coordinate system and the hand recognition module coordinate system [R 1 |T 1 ], converting the hand tracking information H in the real world coordinate system into the hand tracking information H in the gesture recognition module coordinate system s :

[0039] H s =K 1 [R 1 |T 1 ]H

[0040] Step 2: Get the hand tracking information H under the viewpoint e .

[0041] Adjust the hand data according to the user's perspective to ensure that the relationship between the virtual gesture and the viewpoint in the virtual scene is consistent with the real scene. Determine the relationship between the gesture recognition module and the viewpoint by installing the gesture recognition module on the viewpoint device, and use the rigid body transformation matrix [R 2 |T 2 ], the hand tracking information H in the coordinate system of the gesture recognition module s Convert the hand tracking information H into the viewpoint coordinate system e :

[0042] H e =[R 2 |T 2 ]H s

[0043] Step 3: Realize dynamic viewpoint synchronized hand tracking in real scenes.

[0044] Due to the unlimited free movement of people in the interactive scene, the viewpoint position changes dynamically. The positioning module is used to obtain the user's position and head direction information, determine the user's viewpoint coordinates in the interactive scene, and ensure that the viewpoint mapping of the virtual and real worlds is consistent during the interaction process. The camera intrinsic parameter matrix K of the positioning module is used 2 and the rigid body transformation matrix between the viewpoint coordinate system and the interaction scene coordinate system [R 3 |T 3 ] The hand tracking information H in the viewpoint coordinate system e Transformed into the hand tracking information H in the scene coordinate system p :

[0045] H p =K 2 [R 3 |T 3 ]H e

[0046] Therefore, when the user moves freely without restrictions in the interactive scene, the synchronized hand tracking information H under the dynamic viewpoint p for:

[0047] H p =K 2 [R 3 |T3 ][R 2 |T 2 ]K 1 [R 1 |T 1 ]H

[0048] Step 4: Virtual-real mapping of hand data.

[0049] Construct a virtual space that corresponds to the real scene one by one, and set a virtual viewpoint that keeps synchronized movement with the user's viewpoint. Apply the hand tracking data in the real interactive scene coordinate system to the virtual scene, and use the function f to convert the synchronized hand tracking information H under the dynamic viewpoint p Mapped into the hand tracking information H in the virtual scene coordinate system v :

[0050] H v =f(H p )

[0051] Step 5: Drawing virtual gestures under virtual viewpoint.

[0052] Due to the mutual mapping between real space and virtual space, the relationship between the real viewpoint coordinate system and the real interactive scene coordinate system is used. 3 |T 3 ], the position and posture of the virtual hand under the virtual viewpoint can be reversely reconstructed, that is, the virtual gesture drawing information H under the virtual viewpoint coordinate system c , so that it accurately reflects the user's actual gesture actions:

[0053] H c =[R 3 |T 3 ] -1 H v

[0054] Step 6: Realize human-computer interaction.

[0055] On large-screen terminals or other virtual display devices, the user's hand movements are displayed in real time, and by recognizing gestures, virtual objects are controlled or interactive operations are triggered, providing feedback to the user to achieve human-computer interaction under a dynamic viewpoint.

[0056] The present invention has the following beneficial effects:

[0057] 1. The interactive device gets rid of the limitation of detection range caused by the fixed position of the traditional hand tracking system, so that hand tracking is no longer restricted by human body movement, and synchronous hand tracking under dynamic viewpoint is realized, which can flexibly capture the user's hand information, reduce the deviation of hand position, avoid tracking errors, and ensure that the hand movements in the virtual scene are dynamically consistent with the hand movements in the real scene.

[0058] 2. The installation method of the gesture recognition module is based on the effective interaction space and recognition effect. Under the premise of ensuring that the effective interaction space is not affected, the gesture recognition module posture is optimized to maximize the overlapping area between the gesture range that meets the confidence threshold and the commonly used gesture range, thereby determining the best gesture recognition module installation posture, significantly improving the overall interaction robustness and user experience, and thus meeting the usage needs in diverse scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a head-mounted gesture interaction device for 3D large screens;

[0060] Figure 2 It is a schematic diagram of the integrated connection module;

[0061] Figure 3 This is a schematic diagram of the relationship between the sensor normal and the recognition effect;

[0062] Figure 4 Schematic diagram of the camera normal direction of the gesture recognition module;

[0063] Figure 5 It is a simplified schematic diagram of the human body model in the vertical direction;

[0064] Figure 6 Schematic diagram of a two-dimensional rectangular coordinate system in an embodiment. DETAILED DESCRIPTION

[0065] The present invention will be further explained below with reference to the accompanying drawings;

[0066] like Figure 1 As shown, a head-mounted gesture interaction device for a 3D large screen includes a positioning module 1, a gesture recognition module 2, 3D glasses 3 and a large-screen terminal display module. The positioning module 1 and the gesture recognition module 2 are fixed on the 3D glasses 3 through an integrated connection module 4. The positioning module 1 is used to obtain the user's position and head direction information, and determine the user's viewpoint coordinates in the interactive space. The gesture recognition module 2 is used to obtain the user's hand joint data. The large-screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning module 1 and the gesture recognition module 2, and displays the 3D virtual reality scene content required by the user's viewpoint through the screen. The 3D glasses 3 are used to watch the content displayed on the screen.

[0067] like Figure 2 As shown, the shape design of the integrated connection module 4 is optimized based on the best installation posture of the gesture recognition module, so as to improve the stability and accuracy of gesture recognition, and is provided with a positioning module interface 5, a gesture recognition module interface 6 and a 3D glasses interface 7.

[0068] Using a head-mounted gesture interaction device for a 3D large screen, synchronized hand tracking under the user's dynamic viewpoint is achieved, which specifically includes the following steps:

[0069] Step 1: Get hand joint data.

[0070] Use the gesture recognition module to obtain the joint data of the user's hand. Use the camera intrinsic parameter matrix K of the gesture recognition module 1 and the rigid body transformation matrix between the real world coordinate system and the hand recognition module coordinate system [R 1 |T 1 ], converting the hand tracking information H in the real world coordinate system into the hand tracking information H in the gesture recognition module coordinate system s :

[0071] H s =K 1 [R 1 |T 1 ]H

[0072] Step 2: Get the hand tracking information H under the viewpoint e .

[0073] The relationship between the gesture recognition module and the viewpoint is determined by the installation method of the gesture recognition module on the viewpoint device, and the rigid body transformation matrix [R 2 |T 2 ], the hand tracking information H in the coordinate system of the gesture recognition module s Convert the hand tracking information H into the viewpoint coordinate system e :

[0074] H e =[R 2 |T 2 ]H s

[0075] Step 3: Realize dynamic viewpoint synchronized hand tracking in real scenes.

[0076] Use the camera intrinsic parameter matrix K of the positioning module 2 and the rigid body transformation matrix between the viewpoint coordinate system and the interaction scene coordinate system [R 3 |T 3 ] The hand tracking information H in the viewpoint coordinate system e Transformed into the hand tracking information H in the scene coordinate system p :

[0077] H p =K 2 [R 3 |T 3 ]H e

[0078] Therefore, when the user moves freely without restrictions in the interactive scene, the synchronized hand tracking information H under the dynamic viewpoint p for:

[0079] H p =K 2 [R 3 |T 3 ][R 2 |T 2 ]K 1 [R 1 |T 1 ]H

[0080] Step 4: Virtual-real mapping of hand data.

[0081] Construct a virtual space that corresponds to the real scene one by one, and set a virtual viewpoint that keeps synchronized movement with the user's viewpoint. Apply the hand tracking data in the real interactive scene coordinate system to the virtual scene, and use the function f to convert the synchronized hand tracking information H under the dynamic viewpoint p Mapped into the hand tracking information H in the virtual scene coordinate system v :

[0082] H v =f(H p )

[0083] Step 5: Drawing virtual gestures under virtual viewpoint.

[0084] Using the relationship between the real viewpoint coordinate system and the real interactive scene coordinate system [R 3 |T 3 ], reversely reconstruct the virtual gesture drawing information H in the virtual viewpoint coordinate system c :

[0085] H c =[R 3 |T 3 ] -1 H v

[0086] Step 6: Realize human-computer interaction.

[0087] On large-screen terminals or other virtual display devices, the user's hand movements are displayed in real time, and by recognizing gestures, virtual objects are controlled or interactive operations are triggered, providing feedback to the user to achieve human-computer interaction under a dynamic viewpoint.

[0088] Since the installation position of the gesture recognition module on the 3D glasses will directly affect the gesture tracking results, the smaller the angle between the palm normal and the camera principal optical axis normal of the gesture recognition module, that is, the closer the palm normal and the camera principal optical axis normal of the gesture recognition module are to parallelism, the more the palm is facing the camera, the larger the camera observation area is, the less occlusion is, and the more information can be obtained, the higher the recognition accuracy is, the higher the confidence is, the more accurate the tracking result is, and the better the interaction effect is. Conversely, when the normal of the gesture recognition module is closer to the vertical relationship with the normal of the palm, the lower the confidence is, and the less accurate the tracking result is. Figure 3 Therefore, the installation of the gesture recognition module is determined by the following steps:

[0089] Step 1: Calculate the coverage of the effective interaction space S

[0090] When changing the installation position of the gesture recognition module, the gesture recognition effect should be optimized without affecting the size of the effective interaction space.

[0091] When interacting with the large-screen terminal scene through gestures, the user's line of sight should be able to capture the interactive target in the virtual scene and the real hand. Hand activities outside the user's line of sight are considered invalid hand movements. Therefore, the effective interaction space S is composed of the trackable range S of the gesture recognition module. S The reachable range S for gesture interaction H , Visible range S E Jointly decided:

[0092] S=S S ∩S E ∩S H

[0093] Each spatial range is determined by the horizontal field of view H FOV , vertical field of view V FOV And the space radius R determines:

[0094]

[0095] in, Respectively represent the tracking range S of the gesture recognition module S The left and right end values ​​of the horizontal field of view angle, is the upper and lower values ​​of the vertical field of view, r S It is the maximum detection radius of the gesture recognition module. Respectively represent the visible range of gesture interaction S E The left and right end values ​​of the horizontal field of view angle, It is the upper and lower values ​​of the vertical field of view. Respectively represent the reachable range S of gesture interaction H The left and right end values ​​of the horizontal field of view angle, is the upper and lower values ​​of the vertical field of view, r H The length of upper limb.

[0096] When the posture of the gesture recognition module is adjusted in the horizontal direction, it may cause incoordination between vision and hand movements in left-right cognition. For example, the gesture recognition module rotates to the left, and the picture seen by the eyes remains in front, but it needs to grab to the left, and the hand movement deviates from the visual front. This deviation between vision and hand movement may interfere with the intuitiveness and smoothness of the interaction. In contrast, the posture adjustment of the gesture recognition module in the vertical direction has less impact on this left-right conflict, because it mainly affects the up and down adjustment of the line of sight, and will not cause left-right incoordination between vision and hand movements. Therefore, the posture of the gesture recognition module is adjusted only in the vertical direction, and it is assumed that the head remains still, such as Figure 4 shown.

[0097] With reference to ergonomics, the human body structure model in the vertical direction is simplified, and the viewpoint position is directly used as the position of the gesture recognition module to establish the reachable visual range of gesture interaction, such as Figure 5 As shown, where l 1 , l 2 , l 3 They represent the vertical distance between the shoulder joint and the viewpoint, the horizontal distance between the shoulder joint and the viewpoint, and the length of the upper limb. 1 , l 2 , l 3 Determined by the human body height H. θ U ,θ D Respectively represent the upper and lower field of view angles. A two-dimensional rectangular coordinate system is established with the viewpoint position as the origin. Under the premise that the field of view angle of the gesture recognition module is greater than the field of view angle of the gesture interaction visible range, in order to meet the condition of "not affecting the coverage of the effective interaction space", it can be properly When the position θ of the gesture recognition module increases, the overlap area between the detection range of the gesture recognition module in the vertical direction and the visible range of the gesture interaction remains unchanged and remains at the maximum. Therefore, the model can be further simplified and the vertical rotation angle can be directly analyzed. Next, find the optimal vertical rotation position θ of the gesture recognition module, such as Figure 6 shown.

[0098] Step 2: Design the confidence function

[0099] Tomas Novacek et al. proposed a confidence calculation method. First, the angle between the camera normal and the palm normal of the gesture recognition module is calculated. When angle is 0° or 180°, the confidence is the highest value 1; when angle is 90°, the confidence is the lowest value 0.3. The following confidence function confidence is constructed to fit the relationship between angle and confidence:

[0100] confidence=(0.2837*angle 2 )-(0.89127*angle)+1

[0101] In the process of rotation adjustment of the palm posture, the confidence function should have the characteristic of a 90° period, which is more in line with the law of change of actual movement. At the same time, the confidence function should reach a maximum at 0° and 180° and a minimum at 90°, which requires that the derivative of the confidence function at these key points is 0.

[0102] On this basis, this method selects the cosine function as the basis function of the confidence function, establishes a two-dimensional rectangular coordinate system with the viewpoint position as the origin, and converts the camera normal direction and the palm normal direction of the gesture recognition module into the two-dimensional rectangular coordinate system. c Represents the angle between the camera normal and the palm normal, θ h It represents the angle mapped from the palm normal direction to the coordinate system. The following confidence function is designed:

[0103] confidence=0.35*cos(2θ c )+0.65

[0104] Step 3: Design constraints based on confidence thresholds

[0105] In order to ensure that as many interactive gestures as possible reach the desired minimum confidence level C under the installation posture θ of the gesture recognition module, the confidence is designed to meet the following constraints:

[0106] confidence=0.35*cos(2θ c )+0.65≥C

[0107] Step 4: Design constraints based on gesture range

[0108] Consider the practical range of common gestures The normal of the palm is in the vertical direction θ h The following constraints are proposed:

[0109]

[0110] Step 5: Combine constraints

[0111] Without affecting the effective interactive space coverage

[0112] Expand and solve the constraints based on the confidence threshold:

[0113]

[0114] The constraint condition based on the confidence threshold reflects the gesture range θ that can reach the confidence threshold C when the pose of the gesture recognition module is θ. h .

[0115] Combined with the constraints of the gesture range, we get the following relationship:

[0116]

[0117] Step 6: Construct objective function and optimize solution

[0118] The optimization goal is set to maximize the width of the gesture range that can reach the confidence level C. The objective function is as follows:

[0119]

[0120] Solve the above objective function and set the angle of the camera normal of the gesture recognition module to rotate downward relative to the horizontal plane as θ * .

[0121] This embodiment uses Leap Motion as a gesture recognition module, and the specific parameters are shown in Table 1:

[0122] Table 1

[0123]

[0124] According to the above method steps, the parameters in Table 1 are substituted to solve the angle θ between the camera normal of the gesture recognition module and the horizontal plane. * :

[0125]

[0126] θ * =20°

[0127] Therefore, when the gesture recognition module is fixed to the 3D glasses, the downward rotation angle of the normal line of the gesture recognition module relative to the horizontal plane is set to 20°.

Claims

1. A design method for a head-mounted gesture interaction device for a 3D large screen, used to determine the installation position of a gesture recognition module, characterized in that: The specific steps include: Step 1: Calculate the coverage of the effective interaction space S The effective interaction space S is composed of the trackable range S of the hand tracking module S The reachable range S for gesture interaction H , Visible range S E Joint decision: S=S S ∩S E ∩S H Consider adjusting the posture of the hand tracking module in the vertical direction and establish a two-dimensional rectangular coordinate system with the viewpoint position as the origin; Step 2: Design the confidence function Select the cosine function as the basis function of the confidence function and design the following confidence function: confidence=0.35*cos(2θ c )+0.65 Among them, θ c Represents the camera normal direction θ and the palm normal direction θ of the hand tracking module h The angle of Step 3: Design constraints based on confidence thresholds Set the desired minimum confidence level C and design the confidence to meet the following constraints: confidence=0.35*cos(2θ c )+0.65≥C Step 4: Design constraints based on gesture range Consider the actual range of common gestures in the vertical field of view The normal of the palm is in the vertical direction θ h The following constraints are proposed: Step 5: Combine constraints The constraints based on the confidence threshold are expanded and solved and combined with the constraints of the gesture range to obtain the following relationship: in, Step 6: Construct objective function and optimize solution The optimization goal is set to maximize the width of the gesture range that can reach the confidence level C. The objective function is as follows: in, S is the tracking range of the hand tracking module S The lower value of the vertical field of view, S is the visible range of gesture interaction E The lower value of the vertical field of view; Solve the above objective function and set the angle between the camera normal of the hand tracking module in the VR interaction device based on gesture recognition and the horizontal plane to θ * .

2. A design method for a head-mounted gesture interaction device for a 3D large screen as claimed in claim 1, characterized in that: The tracking range S of the hand tracking module S The reachable range S for gesture interaction H , Visible range S E From the horizontal field of view H FOV , vertical field of view V FOV And the space radius R determines: in, Respectively represent the tracking range S of the hand tracking module S The left and right end values ​​of the horizontal field of view angle, is the upper value of the vertical field of view, r S The maximum detection radius of the hand tracking module; Respectively represent the visible range of gesture interaction S E The left and right end values ​​of the horizontal field of view angle, is the upper value of the vertical field of view; Respectively represent the reachable range S of gesture interaction H The left and right end values ​​of the horizontal field of view angle, is the upper and lower values ​​of the vertical field of view, r H The length of upper limb.

3. A design method for a head-mounted gesture interaction device for a 3D large screen as claimed in claim 1, characterized in that: The camera normal direction θ of the hand tracking module and the palm normal direction θ h The angle θ c It is expressed as: i c =|-θ-θ h |=|θ+θ h |。 4. A design method for a head-mounted gesture interaction device for a 3D large screen as claimed in claim 3, characterized in that: Expand the confidence threshold based constraints to:

5. The design method of a head-mounted gesture interaction device for a 3D large screen as claimed in claim 1, characterized in that: Set the angle range of the palm normal θ h ∈[-90°,0°], confidence threshold C=0.

825.

6. A head-mounted gesture interaction device for a 3D large screen, characterized in that: It includes a positioning module, a gesture recognition module, 3D glasses and a large-screen terminal display module; the positioning module and the gesture recognition module are fixed on the 3D glasses, wherein the positioning module is used to obtain the user's position and head direction information to determine the user's viewpoint coordinates in the interactive space; the large-screen terminal display module receives the viewpoint coordinates and hand joint data provided by the positioning module and the gesture recognition module, and displays the 3D virtual reality scene content required by the user's viewpoint through the screen; the 3D glasses are used to view the content displayed on the screen; The angle θ between the camera normal of the hand tracking module and the horizontal plane is * Determined by the method according to any one of claims 1 to 4.

7. The head-mounted gesture interaction device for a 3D large screen as claimed in claim 6, characterized in that: Use LeapMotion as the hand tracking module.

8. The head-mounted gesture interaction device for a 3D large screen as claimed in claim 7, characterized in that: Set the angle of the normal of the hand tracking module to 20° relative to the horizontal plane.

9. The head-mounted gesture interaction device for a 3D large screen as claimed in claim 6, characterized in that: The positioning module and the gesture recognition module are fixed on the 3D glasses through an integrated connection module, and the integrated connection module is provided with a positioning module interface, a gesture recognition module interface and a 3D glasses interface.

10. The head-mounted gesture interaction device for a 3D large screen as claimed in claim 6, characterized in that: To achieve synchronized hand tracking under dynamic viewpoint, follow these steps: Step 1: Obtain hand joint data; The gesture recognition module is used to obtain the joint data of the user's hand. The camera intrinsic parameter matrix K1 of the gesture recognition module and the rigid body transformation matrix [E1|T1] between the real world coordinate system and the hand recognition module coordinate system are used to convert the hand tracking information H in the real world coordinate system into the hand tracking information H in the gesture recognition module coordinate system. s : H s =K1[E1|T1]H Step 2: Get the hand tracking information H under the viewpoint e ; According to the angle θ between the normal line of the hand tracking module and the horizontal plane * , determine the rigid body transformation matrix [R2|T2] between the real gesture recognition module coordinate system and the viewpoint coordinate system, and transform the hand tracking information H in the gesture recognition module coordinate system s Convert the hand tracking information H into the viewpoint coordinate system e : H e =[R2|T2]H s Step 3: Realize dynamic viewpoint synchronous hand tracking in real scenes; The camera intrinsic parameter matrix K2 of the positioning module and the rigid body transformation matrix [R3|T3] between the viewpoint coordinate system and the interaction scene coordinate system are used to transform the hand tracking information H in the viewpoint coordinate system e Transformed into the hand tracking information H in the scene coordinate system p : <h2 style=";text-align:left;direction:ltr">H<h2 style=";text-align:left;direction:ltr"> p <h2 style=";text-align:left;direction:ltr"> =K2[R3|T3]H<h2 style=";text-align:left;direction:ltr"> e <h2 style=";text-align:left;direction:ltr"> =K2[R3|T3][R2|T2]K1[R1|T1]H Step 4: Virtual-real mapping of hand data; Construct a virtual space that corresponds to the real scene one by one, and set a virtual viewpoint that keeps synchronized movement with the user's viewpoint. Use the function f to convert the synchronized hand tracking information H under the dynamic viewpoint p Mapped into the hand tracking information H in the virtual scene coordinate system v : H v =f(H p ) Step 5: Drawing virtual gestures under virtual viewpoint; Using the relationship between the real viewpoint coordinate system and the real interaction scene coordinate system [R3|T3], reversely reconstruct the virtual gesture drawing information H in the virtual viewpoint coordinate system c : H c =[R3|T3] -1 H v Step 6: Realize human-computer interaction; The user's hand movements are displayed in real time on the large-screen terminal display module, and virtual objects are controlled or interactive operations are triggered by identifying gestures, providing feedback to the user to achieve human-computer interaction under a dynamic viewpoint.

Images