Gesture recognition control method for MR interactive scene

Abstract

This invention discloses a gesture recognition and control method for MR interactive scenarios, relating to the field of image recognition technology. The method includes: real-time acquisition of each frame of hand image; inputting the hand image into a hand position acquisition model to obtain the hand position; calculating the joint angle of each finger based on the hand node position; calculating the weighted angular similarity between the k-th frame hand image and the template gesture; calculating the confidence level of the k-th frame hand image; judging the confidence level of the current hand image against a hysteresis threshold to obtain the system's gesture state; and performing gesture recognition and control based on the system's gesture state. This method effectively overcomes problems such as instability, high power consumption, low recognition accuracy, and insufficient robustness, improving the accuracy and stability of dynamic and continuous gesture recognition, thereby meeting the urgent need for natural and robust human-computer interaction control in mixed reality scenarios.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a gesture recognition control method for MR interactive scenarios. Background Technology

[0002] During the interactive displays of exhibits at the science museum, visitors wearing MR glasses need to interact with objects in the scene using gestures, such as clenching a fist, grabbing, poking with a finger, opening their hands, etc. The system performs corresponding operations by recognizing the visitors' gesture intentions, so accurate gesture recognition is crucial.

[0003] Traditional methods require calculating data for all fingers and joints for each frame of gesture image, resulting in high latency and high computational resource consumption on mobile devices or XR devices. They are also prone to jitter or recognition errors when gestures are switched quickly or when there is shaky movement. Furthermore, they do not make sufficient use of temporal information, relying only on the data of the current frame and ignoring the trend of consecutive frames, which results in insufficient robustness. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies in MR interactive scenarios, such as high power consumption, low recognition accuracy, and insufficient robustness in gesture recognition, this invention proposes a gesture recognition control method for MR interactive scenarios.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a gesture recognition and control method for MR interactive scenarios, comprising:

[0006] S1: Real-time acquisition of each frame of hand image, inputting the hand image into the hand point location to obtain the model, and obtaining the hand point location; the hand point location includes the joint node position of each finger, the fingertip node position of each finger, and the wrist node position.

[0007] S2: Calculate and mark the joint angle of each finger based on the hand node position. ;

[0008] Wherein, the joint angle of each finger is the angle between the joint node of each finger and the line connecting the two adjacent nodes; k is the image frame number; i is the finger number, i∈{1,2,3,4,5}; j is the joint node number of each finger; j∈{1,2,3}.

[0009] S3: Calculate the weighted angular similarity between the hand image in frame k and the template gesture. ;

[0010] S4: Calculate the confidence level of the hand image in the k-th frame. ;

[0011] S5: Determine the confidence level of the current hand image and the hysteresis threshold to obtain the system's gesture state;

[0012] S6: Recognize and control gestures based on the system's gesture state.

[0013] Preferably, step S3 includes:

[0014] S31: Calculate the similarity between the angle of each finger and the finger angle in the template gesture; in the k-th frame of the hand image, calculate the similarity between the angle of the i-th finger and the corresponding finger angle in the template gesture. for:

[0015]

[0016] in, Let be the joint angle of the j-th joint of the i-th finger in the template gesture;

[0017] S32: Calculate the weighted angular similarity between the hand image in frame k and the template gesture based on the similarity between the angle of each finger and the finger angle in the template gesture. The calculation formula is:

[0018]

[0019] in, Let be the attention weight of the i-th finger in the current template gesture; ,and .

[0020] Preferably, step S4 includes:

[0021] S41: Calculate the continuity reward of K consecutive hand images The calculation formula is:

[0022]

[0023] in, ; For variance;

[0024] S42: Obtain the confidence score of the hand image in the k-th frame based on the weighted angular similarity between the hand image in the k-th frame and the template gesture, and the continuity reward of the hand images in the K consecutive frames. The calculation formula is:

[0025]

[0026] Where α and β are both adjustable weighting coefficients ≥ 0.

[0027] Preferably, step S5 includes:

[0028] If the system state is off and the confidence level is greater than the first hysteresis threshold for N consecutive frames, the system state switches from off to on; otherwise, the system state remains off.

[0029] If the state is either Start or Hold, and the confidence level is between the first hysteresis threshold and the second hysteresis threshold, then the corresponding system state is either Start or Hold.

[0030] If the system state is held and the confidence level is less than the second hysteresis threshold for N consecutive frames, the system state switches from held to terminated; otherwise, the system state remains held.

[0031] If the system status is "End", the system status will be directly switched to "Shutdown".

[0032] The first hysteresis threshold is greater than the second hysteresis threshold.

[0033] Preferably, N in N consecutive frames is adaptively adjusted, and the calculation method includes:

[0034] S51: For the k-th frame of the hand image, calculate the angular velocity of each joint node in each finger. ;

[0035] S52: Calculate the average angular velocity of each finger based on the angular velocity of each joint node in each finger. ;

[0036] S53: Obtain the overall angular velocity variance of the hand based on the average angular velocity of each finger. ;

[0037] S54: Based on the overall angular velocity variance of the hand, adaptively adjust the number of consecutive frames N to obtain the real-time results of the number of consecutive frames N.

[0038] Preferably, in step S51, the angular velocity of each joint node in each finger is... The calculation formula is:

[0039]

[0040]

[0041] in, Let be the joint angle of the j-th joint node of the i-th finger in the (k-1)-th frame; The angle change of the j-th joint node in the i-th finger in the k-th frame image compared to the (k-1)-th frame image; This is the time interval between two adjacent frames.

[0042] Preferred, only when Update the included angle of the joint. and angular velocity Otherwise, the result from the previous frame will be used; where, It is the threshold for angle change, which is set separately according to different system gestures.

[0043] Preferably, in step S52, the average angular velocity of each finger... The calculation formula is:

[0044]

[0045] in, denoted as the average angular velocity of the i-th finger in the k-th frame of the hand image; K is the number of neighboring frames; m is the frame number.

[0046] Preferably, in step S53, the variance of the overall angular velocity of the hand The calculation formula is:

[0047]

[0048] .

[0049] Preferably, in step S54, the formula for calculating the real-time result of the number of consecutive frames N is:

[0050]

[0051] in, This is the reference frame number; and These represent the upper and lower limits of the number of consecutive frames, respectively; λ is an adjustment parameter.

[0052] The advantages of this invention are:

[0053] (1) This invention extracts joint angles from hand images as the basis for recognition, and combines a complete technical chain of weighted similarity calculation, confidence assessment and hysteresis judgment. It effectively overcomes the problems of false triggering, jitter and instability, high power consumption, low recognition accuracy and insufficient robustness caused by occlusion, lighting changes and rapid movement in complex MR environments of traditional gesture recognition technology. It improves the accuracy and stability of dynamic and continuous gesture recognition, thereby meeting the urgent need for natural and robust human-computer interaction control in mixed reality scenarios.

[0054] (2) This invention calculates joint angles using the finger joint node positions and uses this as a basis to calculate the weighted angle similarity with the template gesture. Compared to methods that only use fingertip positions or hand contours, joint angles can more precisely describe hand posture, especially the degree of finger bending. This enables the system to accurately distinguish gestures that are similar in shape but different in meaning (e.g., a clenched fist versus a relaxed palm, an extended index finger versus an extended and slightly bent index finger), greatly improving the accuracy of gesture recognition and the richness of recognizable gestures. At the same time, angle calculation is relatively dependent on absolute position and is not sensitive to translation and slight rotation of the hand in the image, enhancing the stability of the system under different positions and viewpoints.

[0055] (3) By assigning attention weights to each finger, this invention can flexibly adjust the contribution of each finger according to the characteristics of different gestures. For example, for the "thumbs up" gesture, the thumb can be given a very high weight while ignoring the minor changes of other fingers. This makes the recognition strategy more targeted and fault-tolerant, and avoids overall recognition failure due to occlusion or false detection of non-critical fingers.

[0056] (4) The confidence calculation of this invention not only includes the static similarity of the current frame, but also introduces a continuity reward based on consecutive K frames. By examining the stability (low variance) of consecutive multiple frames, the confidence only increases when the gesture remains similar to the template for a period of time. This effectively filters out accidental, unintentional hand tremors, significantly reducing the false trigger rate. At the same time, it simulates the process by which humans judge the intent of a gesture—a stable, sustained gesture is more likely to represent the user's true intent than a fleeting gesture. Therefore, this confidence can more accurately reflect whether the user actually intends to make a certain control gesture, rather than just whether the hand accidentally moves through a certain shape.

[0057] (5) The number of consecutive frames N required for state switching in this invention is not a fixed value, but is dynamically adjusted according to the overall angular velocity variance of the hand. When the hand movement is slow and stable (small angular velocity variance), the system considers the user's intention to be clear and can shorten the number of consecutive frames N required for judgment, thereby speeding up the system response and reducing operation delay. When the hand movement is fast or unstable (large angular velocity variance), the system considers the current environment to be noisy or the user's actions to be inaccurate, and automatically increases the number of consecutive frames N required for judgment to improve the rigor and stability of the judgment and prevent misidentification.

[0058] (6) This invention employs a state machine with two thresholds (first hysteresis threshold > second hysteresis threshold) and four states (Closed, Begin, Stay, End) to determine the gesture state, ensuring that the "start" and "end" determinations of the gesture are clear and unambiguous. Once the gesture is confirmed (entering the Begin or Stay state), it will not immediately exit due to slight fluctuations in confidence; it must continuously fall below a lower threshold to trigger the end. This completely solves the persistent problem of "switch bounce" in gesture control, making the interaction process stable and reliable. The four states clearly define the complete lifecycle of the gesture (Closed -> Start -> Stay -> End -> Closed). This allows subsequent control logic to be designed based on states rather than simple binary results, laying a solid foundation for realizing complex, persistent gesture interactions (such as pinch-to-zoom and air drag), making the interaction more natural.

[0059] (7) This invention utilizes a hierarchical hand motion vector structure and only when Update the included angle of the joint. and angular velocity Otherwise, the results of the previous frame are used, and only the joint data that has changed significantly are updated, which reduces the amount of computation per frame and improves the real-time performance of the system, making it particularly suitable for mobile XR devices. Attached Figure Description

[0060] Figure 1 This is a flowchart of the steps of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] like Figure 1 As shown, this invention proposes a gesture recognition control method for MR interactive scenarios, including:

[0063] S1: For each gesture, capture each frame of hand image in real time;

[0064] Each frame of hand image is input into the mediapipe hand landmark model to obtain the hand landmark positions; the hand landmark positions include the joint node positions of each finger, the fingertip node positions of each finger, and the wrist node positions;

[0065] S2: Calculate and mark the joint angle of each finger based on the hand node position. .

[0066] Wherein, the joint angle of each finger is the angle between the joint node of each finger and the line connecting the two adjacent nodes; k is the image frame number; i is the finger number, i∈{1,2,3,4,5}; j is the joint node number of each finger; j∈{1,2,3}. Note that since the thumb only has two joint nodes, ... .

[0067] S3: Calculate the weighted angular similarity between the hand image in frame k and the template gesture, including:

[0068] Calculate the similarity between the angle of each finger and the finger angle in the template gesture; in the k-th frame of the hand image, calculate the angle similarity between the i-th finger and the corresponding finger angle in the template gesture. for:

[0069]

[0070] in, This represents the joint angle of the j-th joint of the i-th finger in the template gesture; the data values ​​in the template gesture are obtained by the developer making a standard gesture and recording the current data using the program.

[0071] The range of values ​​is [ [1,1], the closer to 1, the more similar they are.

[0072] Based on the similarity between the angle of each finger and the finger angle in the template gesture, calculate the weighted angular similarity between the hand image in the k-th frame and the template gesture. The calculation formula is:

[0073]

[0074] in, Let represent the attention weight of the i-th finger in the current template gesture. A higher attention weight indicates greater importance, highlighting the role of key fingers in each current gesture. ,and .

[0075] S4: Calculate the confidence score of the hand image in the k-th frame, including:

[0076] The continuity reward for K consecutive frames of hand images is calculated using the following formula:

[0077]

[0078] in, The smaller the variance, the higher the reward; Let Variance be the variance.

[0079] The confidence score of the hand image at frame k is obtained based on the weighted angular similarity of the template gesture in the hand image at frame k and the continuity reward of the hand images in consecutive frames K. The calculation formula is:

[0080]

[0081] Where α and β are both adjustable weighting coefficients ≥ 0.

[0082] S5: Determine the confidence level and hysteresis threshold of the current hand image to obtain the system's gesture state; including:

[0083] If the system state is Closed and the confidence level is greater than the first hysteresis threshold for N consecutive frames, the system state switches from Closed to Begin; otherwise, the system state remains Closed.

[0084] If the state is Begin or Stay, and the confidence level is between the first hysteresis threshold and the second hysteresis threshold (including equal to the first hysteresis threshold or the second hysteresis threshold), then the system state remains Begin or Stay.

[0085] If the system state is Stay and the confidence level is less than the second hysteresis threshold for N consecutive frames, the system state switches from Stay to End; otherwise, the system state remains Stay.

[0086] If the system status is End, the system status will directly switch to Closed.

[0087] The first hysteresis threshold is greater than the second hysteresis threshold.

[0088] Traditional gesture recognition often uses simple Euclidean distance metrics, which are easily affected by noise. This invention proposes a multi-level confidence calculation method that comprehensively considers angle similarity, finger weight, and temporal continuity, thereby greatly improving the accuracy of gesture recognition.

[0089] To accommodate different user actions and system device performance, the number of consecutive frames N can be adaptively adjusted. The specific calculation method for the number of consecutive frames N is as follows:

[0090] S51: For the current k-th frame of the hand image, calculate the angular velocity of each joint node in each finger using the following formula:

[0091]

[0092]

[0093] in, Let be the joint angle of the j-th joint node of the i-th finger in the (k-1)-th frame; The angle change of the j-th joint node in the i-th finger in the k-th frame image compared to the (k-1)-th frame image; This is the time interval between two adjacent frames.

[0094] Only when The angle of the joint is updated only when the time comes. and angular velocity Otherwise, the result from the previous frame will be used; where, It is the threshold for angle change, which can be 0.5° to 2° depending on the gesture.

[0095] S52: Calculate the average angular velocity of each finger based on the angular velocity of each joint node in each finger. The calculation formula is as follows:

[0096]

[0097] in, Let be the average angular velocity of the i-th finger in the k-th frame of the hand image; K is the number of neighboring frames; m is the frame number.

[0098] S53: Obtain the overall angular velocity variance of the hand based on the average angular velocity of each finger. The calculation formula is:

[0099]

[0100]

[0101] S54: Based on the overall angular velocity variance of the hand, adaptive adjustments are made to the number of consecutive frames N to obtain the real-time result of the number of consecutive frames N. The calculation formula is as follows:

[0102]

[0103] in, This is the reference frame number; and These represent the upper and lower limits of the consecutive frame count; λ is an adjustment parameter, with possible values ​​of 5, 3, 15, and 2; clip(.) is the clamping function, specifically:

[0104]

[0105] like A smaller value indicates stable operation; reducing N makes the recognition response faster.

[0106] like A large value indicates motion jitter; increasing N improves robustness and prevents false triggering.

[0107] For viewers wearing MR glasses who need to interact with objects in the scene using gestures, the system recognizes the viewer's gesture intentions and performs corresponding operations through four states (Closed, Begin, Stay, ...). The system uses a state machine (End) to determine the gesture state, ensuring that the "start" and "end" of the gesture are clear and unambiguous. For example, clenching a fist, grasping, poking with a finger, opening the hand, etc. When the system state is off, it means that the system is not executing any instruction functions. However, when the confidence level is greater than the first hysteresis threshold for N consecutive frames, the system state changes from off to start, indicating that a user has started interacting with gestures in the scene, and the system begins to recognize the user's gestures. The system will then continue to recognize the user's gestures and execute the corresponding gesture actions. At this time, the system is in the state of maintaining recognition. When the system is maintaining the execution of the corresponding gesture actions, if the current confidence level is less than the second hysteresis threshold for N consecutive frames, it proves that the system has recognized that the user has no gesture action changes, and the system switches to the end state, indicating that the user's gesture recognition has ended, and the system switches to the off state. Furthermore, once the gesture is confirmed during the system's recognition of the user's gesture (entering the Begin or Stay state), it will not immediately exit due to small fluctuations in the confidence level. It must continuously fall below a lower threshold to trigger the end. This completely solves the persistent problem of "on / off jitter" in gesture control, making the interaction process stable and reliable.

[0108] S6: Recognize and control gestures based on the system's gesture state.

[0109] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0110] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0111] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A gesture recognition control method for an MR interactive scene, characterized in that, include: S1: Real-time acquisition of each frame of hand image, inputting the hand image into the hand point location to obtain the model, and obtaining the hand point location; the hand point location includes the joint node position of each finger, the fingertip node position of each finger, and the wrist node position. S2: Calculate joint angles of each finger based on hand node positions and label as ; Wherein, the joint angle of each finger is the angle between the joint node of each finger and the line connecting the two adjacent nodes; k is the image frame number; i is the finger number, i∈{1,2,3,4,5}; j is the joint node number of each finger; j∈{1,2,3}. S3: calculating a weighted angle similarity of the kth frame of the hand image and the template gesture ; S4: Calculate the confidence of the kth frame of hand image ; S5: Determine the confidence level of the current hand image and the hysteresis threshold to obtain the system's gesture state; S6: Recognize and control gestures based on the system's gesture state; Step S3 includes: S31: Calculate the similarity of the angle of each finger with the angle of the finger in the template gesture; in the kth frame of the hand image, the similarity of the angle of the ith finger with the angle of the corresponding finger in the template gesture is: in, Let be the joint angle of the j-th joint of the i-th finger in the template gesture; S32: Calculate the weighted angular similarity between the hand image in frame k and the template gesture based on the similarity between the angle of each finger and the finger angle in the template gesture. The calculation formula is: in, Let be the attention weight of the i-th finger in the current template gesture; ,and ; Step S4 includes: S41: Calculate the continuity reward of K consecutive hand images The calculation formula is: in, ; For variance; S42: Obtain the confidence score of the hand image in the k-th frame based on the weighted angular similarity between the hand image in the k-th frame and the template gesture, and the continuity reward of the hand images in the K consecutive frames. The calculation formula is: Where α and β are both adjustable weighting coefficients ≥ 0.

2. The gesture recognition control method for MR interactive scenarios as described in claim 1, characterized in that, Step S5 includes: If the system state is off and the confidence level is greater than the first hysteresis threshold for N consecutive frames, the system state switches from off to on; otherwise, the system state remains off. If the state is either Start or Hold, and the confidence level is between the first hysteresis threshold and the second hysteresis threshold, then the corresponding system state is either Start or Hold. If the system state is held and the confidence level is less than the second hysteresis threshold for N consecutive frames, the system state switches from held to terminated; otherwise, the system state remains held. If the system status is "End", the system status will be directly switched to "Shutdown". The first hysteresis threshold is greater than the second hysteresis threshold.

3. The gesture recognition control method for MR interactive scenarios as described in claim 2, characterized in that, N in N consecutive frames is adaptively adjusted, and the calculation method includes: S51: For the k-th frame of the hand image, calculate the angular velocity of each joint node in each finger. ; S52: Calculate the average angular velocity of each finger based on the angular velocity of each joint node in each finger. ; S53: Obtain the overall angular velocity variance of the hand based on the average angular velocity of each finger. ; S54: Based on the overall angular velocity variance of the hand, adaptively adjust the number of consecutive frames N to obtain the real-time results of the number of consecutive frames N.

4. The gesture recognition control method for MR interactive scenarios as described in claim 3, characterized in that, In step S51, the angular velocity of each joint node in each finger is... The calculation formula is: in, Let be the joint angle of the j-th joint node of the i-th finger in the (k-1)-th frame; The angle change of the j-th joint node in the i-th finger in the k-th frame image compared to the (k-1)-th frame image; This represents the time interval between two adjacent frames.

5. The gesture recognition control method for MR interactive scenarios as described in claim 4, characterized in that, Only when Update the included angle of the joint. and angular velocity Otherwise, the result from the previous frame will be used; where, It is the threshold for angle change, which is set separately according to different system gestures.

6. The gesture recognition control method for MR interactive scenarios as described in claim 3, characterized in that, In step S52, the average angular velocity of each finger The calculation formula is: in, denoted as the average angular velocity of the i-th finger in the k-th frame of the hand image; K is the number of neighboring frames; m is the frame number.

7. The gesture recognition control method for MR interactive scenarios as described in claim 3, characterized in that, In step S53, the variance of the overall angular velocity of the hand The calculation formula is: 。 8. The gesture recognition control method for MR interactive scenarios as described in claim 7, characterized in that, In step S54, the formula for calculating the real-time result of the number of consecutive frames N is as follows: in, It is the reference frame number; and These represent the upper and lower limits of the number of consecutive frames, respectively; λ is an adjustment parameter.

Images