Gesture recognition method and device for user gesture, equipment and medium

Abstract

The invention discloses a gesture recognition method and device for a user gesture, equipment and a medium, and relates to the field of man-machine interaction, and the method comprises the steps: obtaining a to-be-recognized gesture of a user, determining the minimum cumulative difference when the to-be-recognized gesture and a gesture map template of each preset gesture complete the optimal time alignment, and determining the minimum cumulative difference when the to-be-recognized gesture and the gesture map template of each preset gesture complete the optimal time alignment; obtaining a full-dimensional mixed distance between the to-be-recognized gesture and each gesture map template; and determining an optimal matching template and a hit mode in the gesture map template according to the full-dimensional mixed distance, and generating an interaction instruction of the to-be-recognized gesture according to the optimal matching template and the hit mode. According to the method, the constraint of static template matching in the prior art is broken, the method not only can adapt to skeleton differences among different users and personalized body languages of the users, but also can really realize that the users adapt to gesture recognition to adapt to qualitative change of the users through gesture recognition.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and specifically to a method, apparatus, device, and medium for recognizing user gestures. Background Technology

[0002] With the rapid development of technologies such as augmented reality, virtual reality, and mixed reality, gesture recognition, as a natural and intuitive human-computer interaction method, has received widespread attention and application. Traditional gesture recognition technologies are mainly divided into mechanical types based on data gloves and vision-based machine vision types. In recent years, with the popularization of camera technology and breakthroughs in deep learning in the field of computer vision, vision-based gesture recognition has become the mainstream.

[0003] In existing technologies, in order to realize user-defined gestures, the common approach is to use a mode where the user records the gestures once and uses them permanently. That is, during the initial setup phase, the user faces the camera and records one or more standard actions according to the instructions. The system then uses feature extraction to store these actions as static templates or to train a personalized model for that user.

[0004] However, the aforementioned existing technologies have the following problems in practical applications: Existing technologies are mostly based on static template matching under a standard viewpoint. However, in actual use, the field of view and distance of the user's hand relative to the camera vary greatly. More importantly, due to differences in skeletal structure such as finger length and palm width, the 2D projection features of the same gesture vary greatly among different users, and even among the same user at different times. Therefore, simply relying on geometric normalization (such as scaling and rotation) cannot handle the complex nonlinear deformation and self-occlusion problems caused by changes in viewpoint. Users' body language habits are also time-varying. For example, when recording for the first time, users often make standard, forceful, and large-amplitude movements to ensure correct recognition. However, with long-term use and increased proficiency, their body language-based gestures may gradually become more relaxed, casual, and smaller in amplitude, forming a personalized gesture pattern. However, neither template matching nor pre-trained models can keep up with this personalized change process, resulting in a significant decrease in recognition rate over time.

[0005] It is evident that existing technologies suffer from rigid recognition patterns and are unable to adapt to skeletal differences among users and their personalized body language. Therefore, developing a gesture recognition method that can accommodate these differences is a crucial issue that the industry urgently needs to address. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a gesture recognition method, apparatus, device and medium for user gestures, thereby solving the problem that the prior art cannot learn the skeletal differences and personalized body language between different users.

[0007] According to a first aspect, embodiments of the present invention provide a gesture recognition method for user gestures, the method comprising: The user's gesture to be recognized is obtained, and the minimum cumulative difference between the gesture to be recognized and the gesture map template of each preset gesture is determined when the best time alignment is achieved. The full-dimensional mixing distance between the gesture to be recognized and each gesture map template is obtained. The gesture map template includes a reference gesture template and gesture variant templates. Each preset gesture has a reference gesture template and several gesture variant templates. The gesture variant templates are generated based on the reference gesture template and / or implicit positive feedback. The implicit positive feedback is the user's historical recognized gestures that have not been revoked. The best matching template and hit mode in the gesture map template are determined based on the full-dimensional mixed distance. The interaction command of the gesture to be recognized is generated based on the best matching template and hit mode. The hit mode includes exact hit and derivative hit. When it is determined to be an exact hit, the gesture recognition result and operation command of the gesture to be recognized are generated based on the preset gesture corresponding to the best matching template and the operation command of the preset gesture. When it is determined to be a derivative hit, the gesture confirmation instruction is generated based on the preset gesture corresponding to the best matching template to instruct the user to confirm the preset gesture.

[0008] In conjunction with the first aspect, in the first embodiment of the first aspect, the reference gesture template for each preset gesture is constructed in the following manner: Identify key points of the hand; Obtain the standard gesture recording video corresponding to each preset gesture, and extract each hand key point from each frame of the standard gesture recording video; Using the wrist joint among the key points of the hand as the reference origin, the relative displacement vector between each key point of the hand and the reference origin in each frame is determined, and the key point vector of the hand in each frame is obtained. Determine the Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger in each frame, use the Euclidean distance as the global reference scalar for each frame, and normalize all hand keypoint vectors using the global reference scalar. Based on the positional changes of the wrist joint in each frame and the previous frame adjacent to each frame, determine the relative translational velocity and relative scale change rate of the wrist joint in each frame. By fusing the normalized hand keypoint vectors, relative translation speed, and relative scale change rate of each frame, a composite feature vector for each frame is constructed. Based on the composite feature vector of each frame, linear interpolation is performed on the continuous frame sequence of historical standard gesture recording videos to generate a baseline gesture template for the preset gesture.

[0009] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the step of fusing the normalized hand keypoint vector, relative translation velocity, and relative scale change rate of each frame to construct a composite feature vector for each frame specifically includes: Determine the first weight for relative translational velocity and the second weight for relative scale change rate; The first feature vector is constructed based on all normalized hand keypoint vectors in each frame; Based on the first weight and the relative translation velocity, a second feature vector is constructed; Based on the second weight and the relative scale change rate, a third feature vector is constructed; Based on the first, second, and third feature vectors, construct a composite feature vector for each frame.

[0010] In conjunction with the first aspect, in the third embodiment of the first aspect, the variant gesture map template for each preset gesture is constructed in the following manner: Add a preset amount of sampling time random scaling perturbation to the baseline gesture template of each preset gesture to obtain the first variant template of each preset gesture; The variance of physiological hand tremor in the standard gesture video of the preset gesture is determined. A preset Gaussian distribution is determined based on the variance of physiological hand tremor. Random noise conforming to the preset Gaussian distribution is superimposed on the first feature vector of the baseline gesture template to obtain the second variant template of each preset gesture. If the historical recognized gesture is determined to be implicit positive feedback, the shortest distance of the historical recognized gesture is used to determine whether the historical recognized gesture can be recorded as a variant gesture in the video. If it is determined that it can be recorded as a variant gesture in the video, a gesture variant template is generated based on the historical recognized gesture.

[0011] In conjunction with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the step of determining whether a historically recognized gesture can be recorded as a variant gesture in video based on the shortest distance of the historically recognized gesture when it is determined that the historically recognized gesture is an implicit positive feedback, and generating a gesture variant template based on the historically recognized gesture when it is determined that the historically recognized gesture can be recorded as a variant gesture in video, specifically includes: If the historical gesture recognition results are generated and the operation command is executed, the historical gesture recognition is determined as implicit positive feedback and an observation window of preset duration is started. If it is determined that the operation command within the observation window has not been revoked, the best matching template is determined based on the shortest distance of the historically recognized gestures. If the shortest distance exceeds the third threshold but does not exceed the second threshold, the historically recognized gestures are recorded in the video as variant gestures; the third threshold is a third preset multiple of the second threshold. Based on the recorded video of the variant gesture, a gesture variant template is generated and placed into the gesture map template pool of the preset gesture corresponding to the best matching template of the historical recognized gesture.

[0012] In conjunction with the first aspect, in the fifth embodiment of the first aspect, the step of determining the optimal matching template and hit pattern in the gesture map template based on the full-dimensional mixed distance, and generating the interaction command for the gesture to be recognized based on the optimal matching template and hit pattern, specifically includes: Determine the shortest distance among all full-dimensional mixed distances, and use the gesture map template corresponding to the shortest distance as the best matching template; If the shortest distance does not exceed the first threshold, the hit mode of the gesture to be recognized is determined to be a precise hit, and the gesture recognition result and operation command of the gesture to be recognized are generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture respectively. If the shortest distance exceeds the first threshold but does not exceed the second threshold, the hit pattern of the gesture to be recognized is determined to be a derivative hit, and a gesture confirmation instruction is generated according to the preset gesture corresponding to the best matching template.

[0013] In conjunction with the first aspect, in the sixth embodiment of the first aspect, the step of acquiring the user's gesture to be recognized, determining the minimum cumulative difference when the gesture to be recognized and the gesture map template of each preset gesture achieve optimal time alignment, and obtaining the full-dimensional mixing distance between the gesture to be recognized and each gesture map template, specifically includes: Extract real-time video stream segments containing user gestures from the user's real-time input video stream, and use these real-time video stream segments as the gestures to be recognized; Determine the inter-frame difference cost between the gesture to be recognized and each gesture map template; The time lengths of the gesture to be recognized and each gesture map template are determined. Based on the time lengths, a global cost accumulation path is planned using a time dynamic warping algorithm. Inter-frame difference costs are accumulated along this global cost accumulation path to obtain the cumulative cost matrix between the gesture to be recognized and each gesture map template. The global cost accumulation path starts from the starting point... To the finish line Path, destination The coordinate information is based on the duration of the gesture to be recognized. and the duration of the gesture map template Received; The minimum cumulative difference is determined from the cumulative cost matrix, and the minimum cumulative difference is normalized to obtain the full-dimensional mixed distance.

[0014] According to a second aspect, embodiments of the present invention also provide a gesture recognition device for user gestures, the device comprising: The path matching module is used to acquire the user's gesture to be recognized, determine the minimum cumulative difference when the gesture to be recognized and the gesture map template of each preset gesture achieve the best time alignment, and obtain the full-dimensional mixing distance between the gesture to be recognized and each gesture map template. The gesture map template includes a baseline gesture template and gesture variant templates. Each preset gesture has a baseline gesture template and several gesture variant templates. The gesture variant templates are generated based on the baseline gesture template and / or implicit positive feedback. The implicit positive feedback is the user's historical recognized gestures that have not been revoked. The gesture recognition module is used to determine the best matching template and hit mode in the gesture map template based on the full-dimensional mixed distance, and to generate the interaction command of the gesture to be recognized based on the best matching template and hit mode. The hit mode includes exact hit and derivative hit. When it is determined to be an exact hit, the gesture recognition result and operation command of the gesture to be recognized are generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture. When it is determined to be a derivative hit, the gesture confirmation instruction is generated according to the preset gesture corresponding to the best matching template to instruct the user to confirm the preset gesture.

[0015] According to a third aspect, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the gesture recognition method for user gestures as described above.

[0016] According to a fourth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gesture recognition method for user gestures as described above. Attached Figure Description

[0017] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings: Figure 1 One of the flowcharts of the user gesture recognition method provided by the present invention is shown; Figure 2 The second flowchart of the gesture recognition method for user gestures provided by the present invention is shown; Figure 3 This diagram illustrates the key hand points in the gesture recognition method for user gestures provided by the present invention. Figure 4 The third flowchart of the gesture recognition method for user gestures provided by the present invention is shown; Figure 5 A schematic diagram of the structure of the gesture recognition device for user gestures provided by the present invention is shown; Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To address the aforementioned issues, this specification provides a gesture recognition method for user gestures, aiming to break free from the constraints of static template matching in existing technologies. This method not only adapts to skeletal differences among different users but also to personalized body language, thereby enhancing the user's interactive experience. Figure 1 This is a flowchart illustrating a user gesture recognition method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method may include the following steps: S101. Obtain the user's gesture to be recognized. Determine the gesture to be recognized The gesture to be recognized is obtained by minimizing the cumulative difference when achieving optimal time alignment with the gesture map template of each preset gesture. The full-dimensional mixing distance between each gesture map template, wherein the gesture map template includes a baseline gesture template and gesture variant templates. Each preset gesture has a baseline gesture template and several gesture variant templates. The gesture variant templates are generated based on the baseline gesture template and / or the user's implicit positive feedback, which is the user's unrevoked historical recognized gestures.

[0020] In this embodiment of the invention, several preset gestures are pre-set, and each preset gesture has a corresponding base gesture template and several gesture variant templates. The base gesture template and gesture variant templates constitute the gesture map template of the corresponding preset gesture.

[0021] It is understandable that the gesture map templates for different preset gestures are different, that is, the base gesture template and gesture variant templates for different preset gestures are different. At the same time, the number of gesture map templates for different preset gestures may also be different.

[0022] In this embodiment of the invention, the gesture to be recognized The full-dimensional mixture distance between the gesture map template and the target gesture is used to measure gesture similarity; the smaller the full-dimensional mixture distance, the more representative the gesture to be recognized. The more similar it is to the gesture map template, that is, the higher the degree of similarity.

[0023] Gesture to be recognized The full-dimensional blending distance between each gesture map template is determined by the gesture to be recognized. The gesture to be recognized is obtained by minimizing the cumulative difference when achieving optimal time alignment with the gesture map templates of each preset gesture. Matching with the gesture map template is transformed into a path planning problem that minimizes energy consumption. The full-dimensional mixing distance can be determined by finding the globally optimal matching path.

[0024] In this embodiment of the invention, whenever a gesture to be recognized is... When a preset gesture is successfully matched with one of the gesture map templates, the gesture to be recognized... This can be used as a historical recognition gesture. A gesture to be recognized can be generated based on the preset gestures corresponding to the gesture map template. The gesture recognition results can be pre-programmed with corresponding operation commands for each preset gesture. When a gesture recognition result is generated, the interactive device, such as an in-vehicle system, can execute the corresponding operation command based on the preset gesture's operation command. Whenever a gesture is to be recognized... After a gesture is successfully recognized, a corresponding gesture recognition result is generated, and the corresponding operation command is executed, a period of [duration] will immediately be initiated for that historically recognized gesture. The observation window, with a duration of If the user triggers an undo operation on the historical recognition gesture within the observation window, negative feedback from the user can be triggered. : Conversely, if the duration is No negative feedback was detected in the observation window. If so, the previous recognition result is determined to be implicit positive feedback. .

[0025] In existing technologies, when encountering the aforementioned cancellation, users are typically required to re-enter the gesture or calibrate whether their entered gesture is standard. This interaction method interrupts the user's interaction process, resulting in a poor user experience. Moreover, the system itself does not learn from these missed cases and cannot respond correctly to similar "non-standard" gestures in the future. In this embodiment of the invention, the recognition result of the user's uncancelled gesture is first used as implicit positive feedback. This involves labeling these unrevoked gesture recognition results as implicit positive samples in an unsupervised manner. This captures variant gestures that, while perhaps not perfectly standard compared to standard gesture recordings, are acceptable to the user during normal use. These variant gestures are then used to generate gesture variant templates. These templates, as part of the gesture atlas template, participate in the recognition of the gesture. The identification and matching process captures the user's implicit positive feedback. And based on implicit positive feedback Generate gesture variation templates to further capture the personalized drift of user gestures. For example, as the user's proficiency increases, the range of motion of the user's gestures relative to the standard gestures recorded in the video gradually decreases, while implicit positive feedback... The accumulation of data will lead to a deeper understanding of users' personalized gestures, and thus enable us to accurately understand users' operational intentions from their extremely simplified micro-movements.

[0026] It should be noted that this is based on the user's implicit positive feedback. This evolution in generating gesture variant templates has gradually improved the user's interactive experience from initial static template matching to highly tacit dynamic template matching. Users only need slight movements to perform various operation commands, avoiding the inability to recognize relaxed or smaller gestures. This gives the system a tacit advanced gesture matching capability, truly realizing a qualitative change from users adapting to gesture recognition to gesture recognition adapting to users.

[0027] S102. Determine the best matching template and hit pattern in the gesture map template based on the full-dimensional mixed distance, and generate the gesture to be recognized based on the best matching template and hit pattern. The interactive commands include two hit modes: exact hit and derivative hit.

[0028] In this embodiment of the invention, the precise matching of the user's gesture to be recognized is achieved. If the gesture is extremely similar to a certain gesture map template, the confidence level of recognition is very high. Accordingly, if it is determined to be a perfect match, the gesture to be recognized is generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture. The gesture recognition result and operation command are used; in this case, the interaction command is the operation command of the preset gesture. The derived hit indicates the user's gesture to be recognized. While the gesture map is relatively close to a certain gesture map template, it still has some discrepancies. Accordingly, if it is determined to be a derived hit, a gesture confirmation instruction is generated based on the preset gesture corresponding to the best matching template, instructing the user to confirm the preset gesture. The user then confirms whether it is the preset gesture corresponding to the best matching template based on the gesture confirmation instruction. If the user confirms "yes", the gesture to be recognized is generated based on the preset gesture corresponding to the best matching template and the operation command of the preset gesture. The system can recognize the gestures and input commands, and conversely, if the user confirms "no", the system can ask the user to re-enter the gestures.

[0029] The gesture recognition method of the present invention uses the full-dimensional mixed distance between the gesture to be recognized and each gesture map template to determine the minimum cumulative difference when the gesture to be recognized and the gesture map templates of each preset gesture achieve the best time alignment. By converting the matching of the gesture to be recognized and the gesture map template into a path planning problem that minimizes energy consumption, the full-dimensional mixed distance can be determined by finding the globally optimal matching path, and the full-dimensional mixed distance is used to measure gesture similarity. By using the user's unrevoked gesture recognition results as implicit positive feedback, this system captures the user's implicit positive feedback in an unsupervised manner and generates gesture variant templates based on this implicit positive feedback. This further captures the personalized drift of the user's gestures. As the implicit positive feedback data accumulates, the understanding of the user's personalized gestures becomes deeper, enabling the system to accurately understand the user's operational intentions from extremely simplified micro-movements. Moreover, this evolution of generating gesture variant templates based on the user's implicit positive feedback gradually improves the user's interaction experience from the initial static template matching to a highly tacit dynamic template matching. Users only need slight movements to complete various operation commands, avoiding the inability to recognize relaxed or smaller gestures. This gives the system a tacit advanced gesture matching capability, breaking the constraints of static template matching in existing technologies. It can not only adapt to the differences in skeletal structure among different users and the personalized body language of users, but also truly achieve a qualitative change from users adapting to gesture recognition to gesture recognition adapting to users.

[0030] In this embodiment of the invention, step S102 includes: S1021. Determine the shortest distance among all multidimensional mixed distances. and the shortest distance The corresponding gesture map template is used as the best matching template.

[0031] S1022, Determining the shortest distance Not exceeding the first threshold That is to say In this case, determine the gesture to be recognized. The hit mode is precise hit, and the gesture to be recognized is generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture. The gesture recognition results and operation instructions.

[0032] S1023, Determining the shortest distance Exceeding the first threshold And not exceeding the second threshold In this case, determine the gesture to be recognized. The hit mode is derived hit, and a gesture confirmation instruction is generated based on the preset gesture corresponding to the best matching template.

[0033] Specifically, in determining the shortest distance Exceeding the second threshold In such cases, prompts can be generated as needed. This is used to determine the gesture to be recognized. It does not match any preset gesture. Instead, it generates prompts that require the user to re-enter the gesture or to calibrate whether the entered gesture is standard, thus avoiding directly interrupting the user interaction process.

[0034] Each gesture map template has its corresponding first threshold. and the second threshold More specifically:

[0035] in, Indicates the first The empirical baseline value of each gesture pattern template can be set in specific values ​​according to the gesture pattern template. Different gesture pattern templates can have different empirical baseline values. Indicates the first The first threshold of a gesture map template; Indicates the first The second threshold of a gesture map template; Indicates the first The empirical coefficient of the preset gesture corresponding to each gesture map template; Indicates the first The variance of hand physiological tremor in the standard gesture recording video corresponding to the preset gesture for each gesture atlas template is used to generate the baseline gesture template for the preset gesture.

[0036] In an embodiment of the present invention, the first threshold The second threshold is used to characterize the tolerable error range of standard gesture recording videos that conform to preset gestures. The first threshold The first preset multiplier can be set to 1.15-1.25 times.

[0037] In this embodiment of the invention, each gesture map template has its corresponding preset gesture, and the preset gesture has its corresponding operation instruction. Thus, when the target is accurately hit and the user confirms "yes" according to the gesture confirmation instruction, the operation instruction can be further determined according to the best matching template.

[0038] Please see Figure 2 In this embodiment of the invention, the baseline gesture template for each preset gesture is constructed in the following manner: S201. Determine the key hand points for gesture recognition.

[0039] Please see Figure 3 In this embodiment of the invention, there are 21 key points on the hand, namely: Node 0 - wrist joint, Node 1 - thumb carpometacarpophalangeal joint, Node 2 - thumb metacarpophalangeal joint, Node 3 - thumb interphalangeal joint, Node 4 - thumb fingertip, Node 5 - index finger metacarpophalangeal joint, Node 6 - index finger proximal interphalangeal joint, Node 7 - index finger distal interphalangeal joint, Node 8 - index fingertip, Node 9 - middle finger metacarpophalangeal joint, Node 10 - middle finger proximal interphalangeal joint, Node 11... Node 11 - distal interphalangeal joint of the middle finger, Node 12 - fingertip of the middle finger, Node 13 - metacarpophalangeal joint of the ring finger, Node 14 - proximal interphalangeal joint of the ring finger, Node 15 - distal interphalangeal joint of the ring finger, Node 16 - fingertip of the ring finger, Node 17 - metacarpophalangeal joint of the little finger, Node 18 - proximal interphalangeal joint of the little finger, Node 19 - distal interphalangeal joint of the little finger, and Node 20 - fingertip of the little finger. The wrist joint is also the reference origin for constructing the reference gesture template.

[0040] S202. Obtain the standard gesture recording video corresponding to each preset gesture, and extract each hand key point from each frame of the standard gesture recording video.

[0041] In this embodiment of the invention, the standard gesture recording video is processed by frame segmentation, and the three-dimensional coordinates of each hand key point in each frame are extracted by using a trained lightweight convolutional neural network model and regression model, thereby extracting each hand key point in each frame.

[0042] Specifically, the standard gesture recording video can be set and provided by the manufacturer, set and provided by the user, or a combination of settings provided by either the manufacturer or the user to form the standard gesture recording video. Multiple preset gestures can be set, and each preset gesture has its corresponding standard gesture recording video and operation command. Different preset gestures correspond to different standard gesture recording videos and operation commands.

[0043] S203. Using the wrist joint as the reference origin, determine the relative displacement vector between each hand keypoint and the reference origin in each frame, thus obtaining the hand keypoint vector for each frame. More specifically:

[0044] in, Indicates the first The first frame The hand key point vector of the nth hand key point, that is, the th hand key point vector. The first frame The relative displacement vector between key hand points and wrist joint; Indicates the first The first frame The vector between each key hand point and the origin of the coordinate system; Indicates the first The vector between the wrist joint and the origin of the coordinate system within the frame. The origin can be any point within the frame.

[0045] In this embodiment of the invention, considering that the wrist joint is the central connection between the arm and the palm and that its position is relatively fixed, unlike key points of the hand such as the fingertip of the index finger which change frequently due to finger bending and extension, and that the wrist joint is also the transmission point for the user's forearm movement and the starting point for the user to make various hand gestures, the relative displacement vector determined by using the wrist joint as the reference origin can better characterize the relative displacement in the user's hand gestures, and the relative displacement vector has biological significance.

[0046] S204. Determine the Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger in each frame. Use the Euclidean distance as the global reference scalar for each frame and normalize all hand keypoint vectors using the global reference scalar. This will give you the normalized hand keypoint vector for each frame. The normalized hand keypoint vector is the spatial pose feature.

[0047] In this embodiment of the invention, the distance between the wrist joint and the metacarpophalangeal joint of the middle finger is actually the length of the third metacarpal bone in the hand. This distance is highly rigid and does not change significantly with different user gestures. The Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger, the most structurally stable part of the hand, is selected as the global reference scalar. All hand keypoint vectors within the frame are then normalized using this global reference scalar to obtain normalized hand keypoint vectors. This scale normalization of the hand keypoint vectors using the global reference scalar, which is the Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger within the same frame, enables the normalized hand keypoint vectors to adapt to differences in bone structure and distance variations among different users.

[0048] More specifically:

[0049] in, Indicates the first The global reference scalar of the frame, that is, the first The Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger in the frame; Indicates the first The relative displacement vector between the metacarpophalangeal joint of the middle finger and the wrist joint in the frame; Indicates the first The displacement vector of the metacarpophalangeal joint of the middle finger relative to the wrist joint in the X-axis direction in the frame; Indicates the first The displacement vector of the metacarpophalangeal joint of the middle finger relative to the wrist joint in the Y-axis direction in the frame; Indicates the first The displacement vector of the metacarpophalangeal joint of the middle finger relative to the wrist joint in the Z-axis direction within the frame.

[0050]

[0051] in, Indicates the first The first frame Normalized hand keypoint vectors.

[0052] S205. Based on the positional changes of the wrist joint in each frame and the previous frame adjacent to each frame, determine the relative translational velocity and relative scale change rate of the wrist joint in each frame. The relative translational velocity and relative scale change rate are dynamic characteristics. More specifically:

[0053]

[0054] in, Indicates the wrist joint at the 1st The relative translation speed of the frames; Indicates the wrist joint at the 1st The relative scale change rate of the frame.

[0055] In this embodiment of the invention, the relative translational velocity of the wrist joint in the current frame can be determined by dividing the vector difference between the wrist joint in the current frame and the origin in the adjacent previous frame by the global reference scalar of the current frame. The relative scale change rate of the wrist joint in the current frame can be determined by dividing the difference in Euclidean distance between the wrist joint in the current frame and the metacarpophalangeal joint of the middle finger in the previous frame by the global reference scalar of the current frame. It is understandable that the above-mentioned point... The frame represents the current frame, and correspondingly, the first frame... A frame represents the previous frame adjacent to the current frame.

[0056] S206. The normalized hand keypoint vectors, relative translation velocity, and relative scale change rate of each frame are fused, that is, spatial pose features and dynamic features are fused to construct a composite feature vector for each frame. Specifically, the first... The composite feature vector of the frame is:

[0057] in, Indicates the first The composite feature vector of a frame; Indicates the first weight; This indicates the second weight.

[0058] In this embodiment of the invention, the first weight It is associated with the wrist joint in the first The relative translation speed of the frame, with the first weight This is also a translational motion weight, achieved by setting the first weight. It can control the dynamic matching effect of the user's hand waving direction; second weight It is associated with the wrist joint in the first The relative scale change rate of the frame, the second weight This also refers to the forward and backward momentum weights, which are determined by setting a second weight. It can control the dynamic matching effect of the forward and backward pushing and pulling speed of the hand.

[0059] It should be noted that when the system detects significant movement of the user's hand in space (such as waving), it can automatically increase the first weight. The weight of the first part is increased to enhance the ability to distinguish translational movements; when the user's hand is detected to be mainly making forward and backward pushing or pulling movements, the weight of the second part is increased. The weight.

[0060] S207. Based on the composite feature vector of each frame, perform linear interpolation on the continuous frame sequence of the historical standard gesture recording video to generate a baseline gesture template for the preset gesture. Specifically, after performing linear interpolation on the continuous frame sequence of the historical standard gesture recording video, the continuous frame sequence can be unified to a fixed length, thus obtaining a standard feature tensor. Standard feature tensor That is, the reference gesture template corresponding to the aforementioned preset gesture.

[0061] More specifically, step S206 includes: S2061. Determine the first weight affecting the relative translational velocity and the second weight affecting the relative scale change rate.

[0062] S2062. Construct the first feature vector based on all normalized hand keypoint vectors in each frame.

[0063] S2063. Based on the first weight and the relative translation velocity, construct the second feature vector.

[0064] S2064. Based on the second weight and the relative scale change rate, construct the third feature vector.

[0065] S2065. Construct a composite feature vector for each frame based on the first, second, and third feature vectors.

[0066] The composite feature vector contains a second and a third feature vector. The second feature vector is obtained based on the first weight and the relative translation velocity, and the third feature vector is obtained based on the second weight and the relative scale change rate. Therefore, the composite feature vector retains a strong ability to distinguish gesture translation, hand pushing and pulling, and other actions.

[0067] In this embodiment of the invention, step S101 includes: S1011. Extract a real-time video stream segment containing the user's gesture from the user's real-time input video stream, and use the real-time video stream segment as the gesture to be recognized. .

[0068] S1012. Determine the gesture to be recognized. The inter-frame difference cost between each gesture map template is as follows:

[0069] in, Indicates the gesture to be recognized The The first frame and gesture map template Inter-frame difference cost; The preset key point weight coefficients are indicated. Except for the wrist joint, which serves as the reference origin, the other 20 hand key points all have their corresponding key point weight coefficients. Indicates the gesture to be recognized No. Frame number Normalized hand key point vectors; Gesture Atlas Template No. Frame number Normalized hand key point vectors; Indicates the gesture to be recognized No. The relative translational velocity of the wrist joint; Gesture Atlas Template No. The relative translational velocity of the wrist joint; Indicates the gesture to be recognized No. The relative scale change rate of the wrist joint in a frame; Gesture Atlas Template No. The relative scale change rate of the wrist joint in the frame. In this calculation formula... For posture difference terms, For the translation difference term, This represents the axial difference term.

[0070] , , , , as well as The calculation methods for these parameters are shown in steps S204 and S205, and will not be repeated here.

[0071] In this embodiment of the invention, the calculation process of inter-frame difference cost includes the gesture to be recognized. The The first frame and gesture map template The differences in hand key points across frames are addressed by normalizing the hand key point vectors, which are capable of handling variations in skeletal structure and distance between different users. This is then combined with the first weight of the gesture to be recognized. The The first frame and gesture map template The differences in relative translation speed of frames and the differences in relative scale change rate combined with the second weight retain the two dynamic features of relative translation speed difference and relative scale change rate, and also include the differences in the momentum of gesture translation, hand push and pull, etc. This makes the calculated inter-frame difference cost naturally capable of dealing with skeletal differences and distance changes. At the same time, it retains the strong discriminative power of gesture translation, hand push and pull, etc., and does not rely on black-box deep learning features, achieving high robustness under small sample conditions and reducing the deployment threshold on embedded devices.

[0072] S1013. Determine the gesture to be recognized. The time lengths of each gesture map template are used as the basis for planning a global cost accumulation path using the Dynamic Time Wrapping (DTW) algorithm. Inter-frame difference costs are accumulated along this global cost accumulation path to obtain the gesture to be recognized. Cumulative cost matrix between each gesture map template ,in, Indicates the gesture to be recognized The length of time; Indicates the duration of the gesture map template.

[0073] In this embodiment of the invention, the DTW algorithm is used to construct the cumulative cost matrix. The global cost accumulation path starts from the starting point. To the finish line Path, destination The coordinate information is based on the gesture to be recognized. Length of time and the duration of the gesture map template Obtained.

[0074] S1014, From the cumulative cost matrix The minimum cumulative difference is determined and normalized to obtain the full-dimensional mixed distance. Determining the minimum cumulative difference involves using the cumulative cost matrix. Find the starting point of the path from the global cost accumulation. To the finish line The minimum cumulative inter-frame difference cost, i.e., using the cumulative cost matrix. Find the globally optimal matching path.

[0075] In this embodiment of the invention, the optimal predecessor state function is specifically used to calculate the globally optimal matching path, specifically:

[0076] in, Indicates the gesture to be recognized The The first frame and gesture map template The cumulative cost of a frame. In this calculation:

[0077]

[0078]

[0079] It is derived from its predecessor, that is, the candidate element of the previous step. , as well as The minimum value in and The sum of them yields It is the minimum cost under monotonically increasing and adjacent constraints; the globally optimal matching path is obtained by acquiring the cumulative cost matrix. middle row and last column element The process is as follows: This means that the minimum cumulative difference can be solved. This process does not use complex deep learning methods, which significantly reduces the computational load of gesture recognition.

[0080] For the minimum cumulative difference Normalization is performed, specifically as follows:

[0081] in, Indicates the gesture to be recognized With the The full-dimensional mixing distance of a gesture map template.

[0082] Please see Figure 4 In this embodiment of the invention, the gesture variant template for each preset gesture is constructed in the following manner: S301. Add a preset amount of random scaling perturbation of sampling time to the baseline gesture template of each preset gesture to obtain the first variant template of each preset gesture. The first variant template is the non-uniform resampling result after adding the preset amount of random scaling perturbation of sampling time to the baseline gesture template, thereby simulating the speed variation of user execution.

[0083] In this embodiment of the invention, the preset amount is: 10-15%.

[0084] S302. Determine the variance of hand physiological tremor in the standard gesture video of the preset gesture. Based on the variance of hand physiological tremor, determine the preset Gaussian distribution. This preset Gaussian distribution is then superimposed on the first feature vector of the baseline gesture template, which is the baseline gesture template. Random noise conforming to a preset Gaussian distribution is superimposed on the keypoint coordinates to obtain a second variant template for each preset gesture. The preset Gaussian distribution is as follows: , This represents the variance of the detected physiological hand tremors in the user. By superimposing random noise conforming to a preset Gaussian distribution onto the key point coordinates, the subtle physiological tremors of the user's hand can be simulated.

[0085] In this embodiment of the invention, after constructing a baseline gesture template for each preset gesture, temporal elastic fission and spatial tolerance fission can be performed based on the baseline gesture template to obtain a first variant map template and a second variant map template, respectively. The first variant map template is also a temporal elastic synthesis variant template based on the baseline gesture template, and the second variant map template is also a spatial tolerance synthesis variant template based on the baseline gesture template. In this way, a baseline gesture template can be constructed first based on the standard gesture recording video for each preset gesture, and then the first and second variant templates can be generated based on the baseline gesture template. Gesture recognition can also be performed in the initial stage based on the baseline gesture template and the first and second variant templates, providing a high recall rate under cold start conditions, ensuring that users can complete the initial gesture recognition even without precise operation.

[0086] In this embodiment of the invention, the gesture map template pool for each preset gesture is: ,in, Indicates the first A pool of preset gesture map templates Indicates the first The first preset gesture A gesture diagram template Indicates the first The first gesture pattern template of the preset gesture, that is, the first... A baseline gesture template for a preset gesture, in addition to The remaining gesture map templates are all gesture variant templates. , The first A first variant of the preset gesture and a second variant of the gesture map template. to This refers to the adaptive reserved slot in the gesture graph template pool. The setting of the adaptive reserved slot can be used to store real user variants generated in subsequent online evolution.

[0087] S303. If it is determined that the historical gesture recognition is judged as implicit positive feedback, then based on the shortest distance of the historical gesture recognition... Determine whether historically recognized gestures can be recorded as variant gestures in video, and if so, generate a gesture variant template based on historically recognized gestures.

[0088] Whenever a gesture to be recognized After a gesture is successfully recognized, a corresponding gesture recognition result is generated, and the corresponding operation command is executed, a period of [duration] will immediately be initiated for that historically recognized gesture. The observation window, with a duration of If the operation command within the observation window is not canceled or the user performs continuous interaction within the observation window, then the historical recognition gesture can be judged as implicit positive feedback. .

[0089] More specifically, step S303 includes: S3031. If the historical gesture recognition result is generated and the operation command is executed, the historical gesture recognition is determined to be implicit positive feedback and the preset duration is started. The observation window.

[0090] S3032. If it is determined that the operation command within the observation window has not been cancelled, the shortest distance of the historically recognized gesture is used. Determine the best matching template. Shortest distance. The methods for determining the best matching template are shown in steps S101 and S102, respectively, and will not be elaborated here.

[0091] S3033, Determining the shortest distance Exceeding the third threshold And not exceeding the second threshold In this case, historically recognized gestures are recorded as variant gestures in the video. Among these, the third threshold... The second threshold The third preset multiple can be set to 0.6-0.7. When setting the first and third preset multiples, the product of the first and third preset multiples must be less than 1, so that the third threshold... Less than the first threshold .

[0092] In this embodiment of the invention, when determining the shortest distance Exceeding the third threshold And not exceeding the second threshold In this case, although the historical recognized gestures can be matched with one of the gesture map templates and thus generate gesture recognition results and operation instructions, there are still significant differences between the historical recognized gestures and their best matching templates. This also means that the effective data under normal user use can be used as new variants.

[0093] Understandably, each gesture map template also has its corresponding third threshold. , Indicates the first The third threshold of a gesture map template.

[0094] S3034. Generate a gesture variant template based on the video of the variant gesture recording, and put it into the gesture map template pool of the preset gesture corresponding to the best matching template of the historical recognized gesture. Specifically, write the new gesture variant template into the empty adaptive reserved slot of the gesture map template pool.

[0095] As shown in steps S202 to S207, a gesture variant template is generated based on the video of the variant gesture recording. The difference is that the new variant, i.e., the variant gesture template, is generated based on the video of the variant gesture recording, i.e., the historical recognized gesture.

[0096] By continuously generating new gesture variant templates using valid data from normal user usage, i.e., historical gesture recognition, the preset gesture map template pool can be continuously replenished as users become more familiar with the usage. In this way, the preset gesture map templates are also supplemented and improved by high-precision and private gesture variant templates generated by users, ensuring the accuracy of gesture recognition during the user's proficiency process.

[0097] In this embodiment of the invention, if the adaptive template slot for the preset gesture is full, but a new historically recognized gesture is recorded in the video as a variant gesture, in this case, all the gesture variant templates of the preset gesture will be eliminated according to the survival score, and the gesture variant template with the lowest survival score will be eliminated. In this way, an adaptive reserved slot can be freed up, and the gesture variant template generated according to the latest variant gesture recording video can be written into the freed adaptive reserved slot.

[0098] By eliminating poorly matched gesture variant templates through survival scores, it is ensured that gesture variant templates can dynamically forget old habits that users no longer use (such as old gesture amplitudes), thereby continuously following the user's current habits. This ensures that the most representative gesture variant templates can be stored while the number of gesture variant templates remains limited, avoiding the accumulation of invalid information and ensuring the long-term stability and efficiency of gesture recognition.

[0099] It should be noted that the first and second variant templates also participate in the elimination process. That is, if the survival score of the first or second variant template is the lowest, its elimination will also be triggered accordingly.

[0100] More specifically, the formula for calculating the survival score is:

[0101] in, Indicates the first Survival score for each gesture variant template; Indicates the first Confidence weights for each gesture variant template; Indicates the current timestamp; Indicates the first The timestamp of the most recent match of a gesture variant template is used as the timestamp of the most recent match when a gesture variant template is used as the best matching template and subsequent gesture recognition results and operation instructions are generated.

[0102] In this embodiment of the invention, when the shortest distance of historically recognized gestures... Not exceeding the third threshold In this case, the confidence weight of the best matching template corresponding to the historical recognized gesture is increased, and the timestamp of the most recent hit of the best matching template is updated to the current timestamp. In this case, the historically recognized gestures can be considered to closely match the best matching template, representing a natural repetition of the same action habit. The corresponding best matching template can then be determined for steady-state reinforcement, and the third threshold can be applied. This becomes the boundary for defining steady-state reinforcement, and the confidence weight of the best matching template corresponding to the historical recognized gesture is increased accordingly, with an increase of 1 value each time.

[0103] It is understandable that the more times a certain gesture variant template is reinforced in steady state, the higher its corresponding confidence weight will be.

[0104] To address the aforementioned issues, this specification provides a gesture recognition device for user gestures, aiming to break free from the constraints of static template matching in existing technologies. This device can adapt to skeletal differences among different users and personalized body language, thereby enhancing the user's interactive experience. Figure 5 This is a schematic diagram of the structure of a gesture recognition device for user gestures according to an embodiment of the present invention, as shown below. Figure 5 As shown, the device may include: Path matching module 10 is used to obtain the user's gesture to be recognized. Determine the gesture to be recognized The gesture to be recognized is obtained by minimizing the cumulative difference when achieving optimal time alignment with the gesture map template of each preset gesture. The full-dimensional mixing distance between each gesture map template, wherein the gesture map template includes a baseline gesture template and gesture variant templates. Each preset gesture has a baseline gesture template and several gesture variant templates. The gesture variant templates are generated based on the baseline gesture template and / or the user's implicit positive feedback, which is the gesture recognition result that the user has not revoked.

[0105] The gesture recognition module 20 is used to determine the best matching template and the hit pattern in the gesture map template based on the full-dimensional mixed distance, and to generate the gesture to be recognized based on the best matching template and the hit pattern. The interactive commands include two hit modes: exact hit and derivative hit.

[0106] The gesture recognition device of the present invention uses a full-dimensional mixed distance between the gesture to be recognized and each gesture map template to determine the minimum cumulative difference when the gesture to be recognized and the gesture map templates of each preset gesture achieve the best time alignment. By converting the matching of the gesture to be recognized and the gesture map template into a path planning problem that minimizes energy consumption, the full-dimensional mixed distance can be determined by finding the globally optimal matching path, and the full-dimensional mixed distance is used to measure gesture similarity. By using the user's unrevoked gesture recognition results as implicit positive feedback, this system captures the user's implicit positive feedback in an unsupervised manner and generates gesture variant templates based on this implicit positive feedback. This further captures the personalized drift of the user's gestures. As the implicit positive feedback data accumulates, the understanding of the user's personalized gestures becomes deeper, enabling the system to accurately understand the user's operational intentions from extremely simplified micro-movements. Moreover, this evolution of generating gesture variant templates based on the user's implicit positive feedback gradually improves the user's interaction experience from the initial static template matching to a highly tacit dynamic template matching. Users only need slight movements to complete various operation commands, avoiding the inability to recognize relaxed or smaller gestures. This gives the system a tacit advanced gesture matching capability, breaking the constraints of static template matching in existing technologies. It can not only adapt to the differences in skeletal structure among different users and the personalized body language of users, but also truly achieve a qualitative change from users adapting to gesture recognition to gesture recognition adapting to users.

[0107] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical commands in the memory 630 to execute a gesture recognition method for user gestures, the method including: The user's gesture to be recognized is obtained, and the minimum cumulative difference between the gesture to be recognized and the gesture map template of each preset gesture is determined when the best time alignment is achieved. The full-dimensional mixing distance between the gesture to be recognized and each gesture map template is obtained. The gesture map template includes a reference gesture template and gesture variant templates. Each preset gesture has a reference gesture template and several gesture variant templates. The gesture variant templates are generated based on the reference gesture template and / or implicit positive feedback. The implicit positive feedback is the user's historical recognized gestures that have not been revoked. The best matching template and hit mode in the gesture map template are determined based on the full-dimensional mixed distance. The interaction command of the gesture to be recognized is generated based on the best matching template and hit mode. The hit mode includes exact hit and derivative hit. When it is determined to be an exact hit, the gesture recognition result and operation command of the gesture to be recognized are generated based on the preset gesture corresponding to the best matching template and the operation command of the preset gesture. When it is determined to be a derivative hit, the gesture confirmation instruction is generated based on the preset gesture corresponding to the best matching template to instruct the user to confirm the preset gesture.

[0108] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for recognizing user gestures, characterized in that, The method includes: The user's gesture to be recognized is obtained, and the minimum cumulative difference between the gesture to be recognized and the gesture map template of each preset gesture is determined when the best time alignment is achieved. The full-dimensional mixing distance between the gesture to be recognized and each gesture map template is obtained. The gesture map template includes a reference gesture template and gesture variant templates. Each preset gesture has a reference gesture template and several gesture variant templates. The gesture variant templates are generated based on the reference gesture template and / or implicit positive feedback. The implicit positive feedback is the user's historical recognized gestures that have not been revoked. The best matching template and hit mode in the gesture map template are determined based on the full-dimensional mixed distance. The interaction command of the gesture to be recognized is generated based on the best matching template and hit mode. The hit mode includes exact hit and derivative hit. When it is determined to be an exact hit, the gesture recognition result and operation command of the gesture to be recognized are generated based on the preset gesture corresponding to the best matching template and the operation command of the preset gesture. When it is determined to be a derivative hit, the gesture confirmation instruction is generated based on the preset gesture corresponding to the best matching template to instruct the user to confirm the preset gesture.

2. The gesture recognition method for user gestures according to claim 1, characterized in that, The baseline gesture template for each preset gesture is constructed in the following way: Identify key points of the hand; Obtain the standard gesture recording video corresponding to each preset gesture, and extract each hand key point from each frame of the standard gesture recording video; Using the wrist joint among the key points of the hand as the reference origin, the relative displacement vector between each key point of the hand and the reference origin in each frame is determined, and the key point vector of the hand in each frame is obtained. Determine the Euclidean distance between the wrist joint and the metacarpophalangeal joint of the middle finger in each frame, use the Euclidean distance as the global reference scalar for each frame, and normalize all hand keypoint vectors using the global reference scalar. Based on the positional changes of the wrist joint in each frame and the previous frame adjacent to each frame, determine the relative translational velocity and relative scale change rate of the wrist joint in each frame. By fusing the normalized hand keypoint vectors, relative translation speed, and relative scale change rate of each frame, a composite feature vector for each frame is constructed. Based on the composite feature vector of each frame, linear interpolation is performed on the continuous frame sequence of historical standard gesture recording videos to generate a baseline gesture template for the preset gesture.

3. The gesture recognition method for user gestures according to claim 2, characterized in that, The composite feature vector for each frame is constructed by fusing the normalized hand keypoint vector, relative translation velocity, and relative scale change rate of each frame, specifically including: Determine the first weight for relative translational velocity and the second weight for relative scale change rate; The first feature vector is constructed based on all normalized hand keypoint vectors in each frame; Based on the first weight and the relative translation velocity, a second feature vector is constructed; Based on the second weight and the relative scale change rate, a third feature vector is constructed; Based on the first, second, and third feature vectors, construct a composite feature vector for each frame.

4. The gesture recognition method for user gestures according to claim 1, characterized in that, Each preset gesture variant gesture map template is constructed in the following way: Add a preset amount of sampling time random scaling perturbation to the baseline gesture template of each preset gesture to obtain the first variant template of each preset gesture; The variance of physiological hand tremor in the standard gesture video of the preset gesture is determined. A preset Gaussian distribution is determined based on the variance of physiological hand tremor. Random noise conforming to the preset Gaussian distribution is superimposed on the first feature vector of the baseline gesture template to obtain the second variant template of each preset gesture. If the historical recognized gesture is determined to be implicit positive feedback, the shortest distance of the historical recognized gesture is used to determine whether the historical recognized gesture can be recorded as a variant gesture in the video. If it is determined that it can be recorded as a variant gesture in the video, a gesture variant template is generated based on the historical recognized gesture.

5. The gesture recognition method for user gestures according to claim 4, characterized in that, When it is determined that a historically recognized gesture is implicit positive feedback, the process involves determining whether the historically recognized gesture can be recorded as a variant gesture in the video based on its shortest distance, and if it is determined that it can be recorded as a variant gesture, generating a gesture variant template based on the historically recognized gesture. This specifically includes: If the historical gesture recognition results are generated and the operation command is executed, the historical gesture recognition is determined as implicit positive feedback and an observation window of preset duration is started. If it is determined that the operation command within the observation window has not been revoked, the best matching template is determined based on the shortest distance of the historically recognized gestures. If the shortest distance exceeds the third threshold but does not exceed the second threshold, the historically recognized gestures are recorded in the video as variant gestures; the third threshold is a third preset multiple of the second threshold. Based on the recorded video of the variant gesture, a gesture variant template is generated and placed into the gesture map template pool of the preset gesture corresponding to the best matching template of the historical recognized gesture.

6. The gesture recognition method for user gestures according to claim 1, characterized in that, The process of determining the optimal matching template and hit pattern in the gesture map template based on the full-dimensional mixed distance, and generating the interaction command for the gesture to be recognized based on the optimal matching template and hit pattern, specifically includes: Determine the shortest distance among all full-dimensional mixed distances, and use the gesture map template corresponding to the shortest distance as the best matching template; If the shortest distance does not exceed the first threshold, the hit mode of the gesture to be recognized is determined to be a precise hit, and the gesture recognition result and operation command of the gesture to be recognized are generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture respectively. If the shortest distance exceeds the first threshold but does not exceed the second threshold, the hit pattern of the gesture to be recognized is determined to be a derivative hit, and a gesture confirmation instruction is generated according to the preset gesture corresponding to the best matching template.

7. The gesture recognition method for user gestures according to claim 1, characterized in that, The process of acquiring the user's gesture to be recognized, determining the minimum cumulative difference when the gesture to be recognized and the gesture map template of each preset gesture achieve optimal time alignment, and obtaining the full-dimensional mixing distance between the gesture to be recognized and each gesture map template specifically includes: Extract real-time video stream segments containing user gestures from the user's real-time input video stream, and use these real-time video stream segments as the gestures to be recognized; Determine the inter-frame difference cost between the gesture to be recognized and each gesture map template; The time lengths of the gesture to be recognized and each gesture map template are determined. Based on the time lengths, a global cost accumulation path is planned using a time dynamic warping algorithm. Inter-frame difference costs are accumulated along this global cost accumulation path to obtain the cumulative cost matrix between the gesture to be recognized and each gesture map template. The global cost accumulation path starts from the starting point... To the finish line Path, destination The coordinate information is based on the duration of the gesture to be recognized. and the duration of the gesture map template Received; The minimum cumulative difference is determined from the cumulative cost matrix, and the minimum cumulative difference is normalized to obtain the full-dimensional mixed distance.

8. A gesture recognition device for user gestures, characterized in that, The device includes: The path matching module is used to acquire the user's gesture to be recognized, determine the minimum cumulative difference when the gesture to be recognized and the gesture map template of each preset gesture achieve the best time alignment, and obtain the full-dimensional mixing distance between the gesture to be recognized and each gesture map template. The gesture map template includes a baseline gesture template and gesture variant templates. Each preset gesture has a baseline gesture template and several gesture variant templates. The gesture variant templates are generated based on the baseline gesture template and / or implicit positive feedback. The implicit positive feedback is the user's historical recognized gestures that have not been revoked. The gesture recognition module is used to determine the best matching template and hit mode in the gesture map template based on the full-dimensional mixed distance, and to generate the interaction command of the gesture to be recognized based on the best matching template and hit mode. The hit mode includes exact hit and derivative hit. When it is determined to be an exact hit, the gesture recognition result and operation command of the gesture to be recognized are generated according to the preset gesture corresponding to the best matching template and the operation command of the preset gesture. When it is determined to be a derivative hit, the gesture confirmation instruction is generated according to the preset gesture corresponding to the best matching template to instruct the user to confirm the preset gesture.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the gesture recognition method for user gestures as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the gesture recognition method for user gestures as described in any one of claims 1 to 7.

Images