Dynamic gesture real-time recognition method and computer readable storage medium

Abstract

The invention provides a dynamic gesture real-time recognition method and a computer readable storage medium. The method comprises the following steps: acquiring continuous multi-frame gesture images of a current target video stream; wherein the gesture image comprises a gesture infrared image and a gesture depth image; according to continuous multi-frame gesture infrared images of the current target video stream, identifying a static gesture category and a gesture anchor box corresponding to each frame of gesture image; acquiring position information of a target position point in a gesture anchor frame of each frame of gesture image; projecting the target position point corresponding to each frame of gesture image to a three-dimensional space according to the gesture depth image corresponding to each frame of gesture image and the position information of the target position point, and obtaining three-dimensional information of the target position point corresponding to each frame of gesture image; and determining a dynamic gesture category corresponding to the current target video stream according to the static gesture category corresponding to each frame of gesture image in the continuous multiple frames of gesture images and the three-dimensional information of the target position point.

Description

Technical Field

[0001] The present application relates to the field of gesture recognition technology, and in particular to a method for real-time recognition of dynamic gestures and a computer-readable storage medium. Background Art

[0002] As a core technology in human-computer interaction, dynamic gesture recognition enables natural interaction with devices by capturing and analyzing human hand movements. It has wide applications in areas such as intelligent driving and virtual reality. For example, in in-vehicle systems, gesture control can reduce driver distraction and improve driving safety.

[0003] In existing technologies, there are two common approaches to deep learning-based dynamic gesture recognition: methods based on hand skeletal keypoint detection and methods that directly classify video sequences. The latter requires inputting the video sequence into a 3D CNN, Long Short-Term Memory (LSTM), or Transformer model, extracting spatiotemporal features for a fixed sequence length, and directly outputting the gesture category. While this approach can improve the robustness and accuracy of gesture recognition, the computational complexity and high cost of using 3D CNN, Long Short-Term Memory (LSTM), or Transformer models make it difficult to deploy in embedded devices for real-time operation. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a real-time dynamic gesture recognition method and computer-readable storage medium to solve the problems of current video sequence classification methods for dynamic gesture recognition using 3D CNN, long short-term memory network (LSTM) or Transformer models, which are computationally complex, have high computational costs, and are difficult to deploy in embedded devices for real-time operation.

[0005] In a first aspect, the present application provides a method for real-time recognition of dynamic gestures, the method comprising: obtaining a continuous multi-frame gesture image of a current target video stream; wherein the gesture image comprises a gesture infrared image and a gesture depth image; identifying the static gesture category and gesture anchor frame corresponding to each frame gesture image based on the continuous multi-frame gesture infrared image of the current target video stream; obtaining the position information of the target position point in the gesture anchor frame of each frame gesture image; projecting the target position point corresponding to each frame gesture image into three-dimensional space based on the gesture depth image and the position information of the target position point corresponding to each frame gesture image, and obtaining the three-dimensional information of the target position point corresponding to each frame gesture image; determining the dynamic gesture category corresponding to the current target video stream based on the static gesture category and the three-dimensional information of the target position point corresponding to each frame gesture image in the continuous multi-frame gesture image.

[0006] This scheme designs a real-time dynamic gesture recognition method. This scheme uses the infrared image of each frame in the video stream data to identify the static gesture category and gesture anchor frame position of each frame in the video stream data, and then converts the target position point in the gesture into three-dimensional space based on the depth map of each frame in the video stream data and the gesture anchor frame position. The dynamic gesture category is determined based on the static gesture results of consecutive frames and the position changes of the target position points in consecutive frames after conversion to three-dimensional space. In this way, the real-time recognition of dynamic gestures in multiple frames can be achieved by cleverly combining single-frame static gesture recognition with the three-dimensional transformation of the target position point. Compared with the traditional method of using 3D CNN, long short-term memory network (LSTM) or Transformer model, the memory and computing resources occupied are significantly reduced, which enables this scheme to run in real time in embedded devices.

[0007] In an optional implementation of the first aspect, the dynamic gesture category corresponding to the current target video stream is determined based on the static gesture category corresponding to each frame of the gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point, including: judging whether only gestures of the same attribute category exist in the multi-frame gesture images based on the static gesture category corresponding to each frame of the gesture image in the continuous multi-frame gesture image; if it is judged that only gestures of the same attribute category exist in the multi-frame gesture image, calculating the attribute offset corresponding to the attribute category gesture based on the three-dimensional information of the target position point corresponding to the multi-frame gesture image; judging whether the attribute offset exceeds the offset threshold corresponding to the attribute offset; if it is judged that the attribute offset exceeds the offset threshold corresponding to the attribute offset, determining the attribute category as the dynamic gesture category corresponding to the current target video stream.

[0008] In the above-mentioned implementation mode, this scheme determines the gesture attribute category of continuous multi-frame images by the static gesture category of continuous multi-frame images, and then calculates the attribute offset based on the three-dimensional information change of the target position point of the continuous multi-frame images. Then, when it is determined that the attribute offset exceeds the corresponding threshold, the attribute category is determined to be the dynamic gesture category corresponding to the current target video stream, thereby combining the static gestures in the continuous multi-frame images with the three-dimensional motion information to identify the dynamic gestures, thereby improving the accuracy and robustness of real-time recognition of dynamic gestures.

[0009] In an optional implementation of the first aspect, the dynamic gesture category corresponding to the current target video stream is determined based on the static gesture category corresponding to each frame of the gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point, including: inputting the static gesture category corresponding to each frame of the gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point into the sequence sliding window in sequence; wherein the sequence sliding window size is N frames; based on the static gesture category corresponding to each frame of the gesture image in the N frames of the gesture image in the sequence sliding window, determining whether only gestures of the same attribute category exist in the N frames of the gesture image in the sequence sliding window; if it is determined that only gestures of the same attribute category exist in the N frames of the gesture image in the sequence sliding window For a gesture of an attribute category, the attribute offset corresponding to the attribute category gesture is calculated based on the three-dimensional information of the target position point corresponding to the N frames of gesture images in the sequence sliding window; it is determined whether the attribute offset exceeds the offset threshold corresponding to the attribute offset; if it is determined that the attribute offset exceeds the offset threshold corresponding to the attribute offset, the dynamic gesture category of the consecutive N frames of gesture images in the current sequence sliding window is identified as the attribute category, and the recognition result is recorded; after multiple consecutive frames of gesture images have passed through the sequence sliding window, each recognition result recorded by the sequence sliding window is obtained; based on each recognition result recorded by the sequence sliding window, the dynamic gesture category corresponding to the current target video stream is determined.

[0010] In an optional implementation of the first aspect, the dynamic gesture category corresponding to the current target video stream is determined based on each recognition result recorded by the sequence sliding window, including: determining the attribute category with the highest frequency of occurrence in each recognition result recorded by the sequence sliding window as the dynamic gesture category corresponding to the current target video stream.

[0011] In an optional implementation of the first aspect, calculating an attribute offset corresponding to an attribute category gesture based on three-dimensional information of target position points corresponding to N frames of gesture images in a sequential sliding window includes: calculating an offset between the target position points of a first frame and a last frame of the N frames of gesture images in the sequential sliding window based on the three-dimensional information of the target position points corresponding to the N frames of gesture images in the sequential sliding window to obtain an overall attribute offset; calculating an offset of each frame of the N frames of gesture images in the sequential sliding window relative to the target position point of the first frame based on the three-dimensional information of the target position points corresponding to the N frames of gesture images in the sequential sliding window to obtain a maximum attribute offset; and determining whether the attribute offset exceeds an offset threshold corresponding to the attribute offset includes: determining whether the overall attribute offset exceeds a first offset threshold corresponding to the overall attribute offset; and determining whether the maximum attribute offset exceeds a second offset threshold corresponding to the maximum attribute offset; if it is determined that the overall attribute offset exceeds the first offset threshold corresponding to the overall attribute offset and the maximum attribute offset exceeds the second offset threshold corresponding to the maximum attribute offset, then determining that the attribute offset exceeds the offset threshold corresponding to the attribute offset.

[0012] Among the various implementation methods mentioned above, this solution adopts a sequential sliding window method to determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each gesture image in multiple consecutive frames of gesture images and the three-dimensional information of the target position point. The sequential sliding window is used to limit the size of the gesture image data processed each time, thereby avoiding the problem of a surge in computing power caused by ultra-long video stream data, which occupies huge computing resources. This reduces the complexity of the calculation and reduces the consumption of computing resources, so that when this solution is set in an embedded device, its computing efficiency can be significantly improved, ensuring the real-time operation of the embedded device.

[0013] In an optional implementation of the first aspect, based on the gesture depth image corresponding to each frame of gesture image and the position information of the target position point, the target position point corresponding to each frame of gesture image is projected into three-dimensional space to obtain three-dimensional information of the target position point corresponding to each frame of gesture image, including: determining the projection depth value corresponding to each frame of gesture image based on the gesture depth image corresponding to each frame of gesture image; based on the camera model projection process, based on the projection depth value corresponding to each frame of gesture image and the position information of the target position point, the target position point corresponding to each frame of gesture image is projected into three-dimensional space to obtain three-dimensional information of the target position point corresponding to each frame of gesture image.

[0014] In the above implementation mode, this scheme projects the target position point corresponding to each frame of gesture image into three-dimensional space based on the projection depth value and position information of the target position point corresponding to each frame of gesture image, based on the camera model projection process, and obtains the three-dimensional information of the target position point corresponding to each frame of gesture image, and then determines the three-dimensional change process of the gesture based on the three-dimensional information, thereby eliminating the scale ambiguity in the space, and thus improving the robustness of the real-time recognition of dynamic gestures of this scheme.

[0015] In an optional implementation of the first aspect, determining the projection depth value corresponding to each frame of gesture image based on the gesture depth image corresponding to each frame of gesture image includes: determining the depth value of the target position point in each frame of gesture depth image as the projection depth value corresponding to each frame of gesture image.

[0016] In the above implementation manner, this solution directly determines the depth value of the target position point in each frame of the gesture depth image as the projection depth value corresponding to each frame of the gesture image, thereby improving the efficiency of the three-dimensional projection.

[0017] In an optional implementation of the first aspect, the projection depth value corresponding to each frame of the gesture image is determined based on the gesture depth image corresponding to each frame of the gesture image, including: obtaining the depth value of each position point in the target area of ​​each frame of the gesture image; wherein the target area is an area range with the target position point as the center and a preset number of position points as the radius; calculating the average value of the depth values ​​of all pixel points in the target area of ​​each frame of the gesture image to obtain the projection depth value corresponding to each frame of the gesture image; or, taking the median of the depth values ​​of all pixel points in the target area of ​​each frame of the gesture image as the projection depth value corresponding to each frame of the gesture image.

[0018] In the above implementation mode, this solution uses the average value or median of the depth values ​​of the target area around the target position point of each frame of the gesture image as the corresponding projected depth value of each frame of the gesture image, thereby avoiding the problem of abnormal depth value corresponding to the center point of the gesture anchor frame caused by errors in the TOF camera itself or mirror reflection, thereby improving the accuracy of the projected depth value and the accuracy of the final real-time recognition of dynamic gestures.

[0019] In an optional implementation of the first aspect, based on the continuous multiple-frame gesture infrared images of the current target video stream, the static recognition result corresponding to each frame of the gesture image is identified, including: inputting the continuous multiple-frame gesture infrared images of the current target video stream into a pre-trained static gesture recognition model, and obtaining the static gesture category and gesture anchor frame corresponding to each frame of the gesture image output by the static gesture recognition model.

[0020] In the above implementation mode, this solution uses a pre-trained static gesture recognition model to recognize multiple consecutive frames of gesture infrared images in the current target video stream, thereby improving the accuracy and efficiency of static gesture category and gesture anchor frame recognition.

[0021] In a second aspect, the present application provides a real-time dynamic gesture recognition device, which includes: an acquisition module, an identification module, a projection module and a determination module, wherein the acquisition module is used to acquire a continuous multi-frame gesture image of the current target video stream; wherein the gesture image includes a gesture infrared image and a gesture depth image; the identification module is used to identify the static gesture category and gesture anchor frame corresponding to each frame gesture image based on the continuous multi-frame gesture infrared image of the current target video stream; the acquisition module is also used to obtain the position information of the target position point in the gesture anchor frame of each frame gesture image; the projection module is used to project the target position point corresponding to each frame gesture image into three-dimensional space based on the gesture depth image corresponding to each frame gesture image and the position information of the target position point, so as to obtain the three-dimensional information of the target position point corresponding to each frame gesture image; the determination module is used to determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each frame gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point.

[0022] The above-designed real-time dynamic gesture recognition device uses the infrared image of each frame in the video stream data to identify the static gesture category and gesture anchor frame position of each frame in the video stream data, and then converts the target position point in the gesture into three-dimensional space based on the depth map of each frame in the video stream data and the gesture anchor frame position. The dynamic gesture category is determined based on the static gesture results of consecutive frames and the position changes of the target position points in consecutive frames after conversion to three-dimensional space. In this way, the real-time recognition of dynamic gestures in multiple frames can be achieved by cleverly combining single-frame static gesture recognition with the three-dimensional transformation of the target position point. The memory and computing resource usage is significantly reduced compared to the traditional methods using 3D CNN, long short-term memory network (LSTM) or Transformer models, which enables this solution to run in real time in embedded devices.

[0023] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes the method described in the first aspect, any optional embodiment of the first aspect, or the second aspect.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the method described in the first aspect, any optional embodiment of the first aspect, or the second aspect.

[0025] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs the method described in the first aspect, any optional embodiment of the first aspect, or the second aspect.

[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of the first process flow of the method for real-time dynamic gesture recognition provided by an embodiment of the present application; Figure 2Schematic diagram of some static gesture images in the static gesture image dataset; Figure 3 A second flow chart of the method for real-time dynamic gesture recognition provided by an embodiment of the present application; Figure 4 A third flow chart of the method for real-time dynamic gesture recognition provided by an embodiment of the present application; Figure 5 A schematic diagram of the structure of a device for real-time dynamic gesture recognition provided by an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0029] Icons: 500 - acquisition module; 510 - identification module; 520 - projection module; 530 - determination module; 6 - electronic device; 601 - processor; 602 - memory; 603 - communication bus. DETAILED DESCRIPTION

[0030] The following embodiments of the technical solution of the present application will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application and are therefore only examples and are not intended to limit the scope of protection of the present application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned figure descriptions are intended to cover non-exclusive inclusions.

[0032] In the description of the embodiments of this application, the technical terms "first" and "second" are used only to distinguish different objects and should not be understood to indicate or imply relative importance or implicitly specify the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, the meaning of "plurality" is more than two, unless otherwise clearly and specifically defined.

[0033] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0035] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0036] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise frame", "counterclockwise frame", "axial", "radial", "circumferential", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0037] In the description of the embodiments of the present application, unless otherwise expressly specified or limited, technical terms such as "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components or interactions between two components. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present application based on specific circumstances.

[0038] As a core technology in human-computer interaction, dynamic gesture recognition enables natural interaction with devices by capturing and analyzing human hand movements. It has wide applications in areas such as intelligent driving and virtual reality. For example, in in-vehicle systems, gesture control can reduce driver distraction and improve driving safety.

[0039] In existing technologies, there are two common approaches to deep learning-based dynamic gesture recognition: methods based on hand skeletal keypoint detection and methods that directly classify video sequences. The latter requires inputting the video sequence into a 3D CNN, Long Short-Term Memory (LSTM), or Transformer model, extracting spatiotemporal features for a fixed sequence length, and directly outputting the gesture category. While this approach can improve the robustness and accuracy of gesture recognition, the computational complexity and high cost of using 3D CNN, Long Short-Term Memory (LSTM), or Transformer models make it difficult to run in real time on embedded devices.

[0040] Based on the above problems, the present application designs a method for real-time recognition of dynamic gestures and a computer-readable storage medium. First, the static gesture category and gesture anchor frame position of each frame image in the video stream data are identified through the infrared image of each frame image in the video stream data. Then, based on the depth map of each frame image in the video stream data and the gesture anchor frame position, the target position point in the gesture is converted to three-dimensional space. The dynamic gesture category is determined based on the static gesture results of consecutive frames and the position changes of the target position points in consecutive frames after conversion to three-dimensional space. In this way, the real-time recognition of dynamic gestures in multiple frames can be achieved by cleverly combining single-frame static gesture recognition with the three-dimensional transformation of the target position point. The memory and computing resources occupied are significantly reduced compared with the traditional method of using 3D CNN, long short-term memory network (LSTM) or Transformer model, so that this solution can run in real time in embedded devices.

[0041] Based on the above ideas, the present application first provides a method for real-time dynamic gesture recognition, which is applied to computing devices, including but not limited to computers, servers, embedded devices and embedded chips, etc. Figure 1 As shown, the real-time dynamic gesture recognition method can be implemented by the following methods, including: Step S100: Acquire a continuous multi-frame gesture image of the current target video stream, wherein the gesture image includes a gesture infrared image and a gesture depth image.

[0042] Step S110: identifying the static gesture category and gesture anchor frame corresponding to each frame of gesture image according to the continuous multiple frames of gesture infrared images of the current target video stream.

[0043] Step S120: Obtain the position information of the target position point in the gesture anchor frame of each frame of the gesture image.

[0044] Step S130: projecting the target position point corresponding to each frame of gesture image into three-dimensional space according to the gesture depth image corresponding to each frame of gesture image and the position information of the target position point to obtain three-dimensional information of the target position point corresponding to each frame of gesture image.

[0045] Step S140: determining the dynamic gesture category corresponding to the current target video stream according to the static gesture category corresponding to each gesture image in the continuous multiple gesture image frames and the three-dimensional information of the target position point.

[0046] In the above embodiment, the current target video stream obtained by this application is a TOF (Time-of-Flight Video Stream) video stream. The TOF video stream is a continuous dynamic image sequence collected based on the time-of-flight (TOF) technology. Its core feature is that each frame of the image not only contains traditional grayscale or color information, but also can synchronously output the distance (depth) information of each pixel in the scene and the camera, forming "color + depth" dual-channel dynamic data; among them, the user makes gestures around the set TOF camera, and this solution can collect the user's waving gesture image through the TOF camera to form a TOF video stream.

[0047] Based on the above, it can be seen that the current target video stream obtained by this solution is based on the TOF video stream. Each frame of the continuous multiple frames of gesture images in the current target video stream can include a gesture infrared image and a gesture depth image.

[0048] When continuous multi-frame gesture infrared images and gesture depth images are obtained by the above method, this solution first identifies the static gesture category and gesture anchor frame corresponding to each frame of gesture image based on the continuous multi-frame gesture infrared images of the current target video stream.

[0049] As a possible implementation method, this solution can pre-train a static gesture recognition model, and input multiple consecutive frames of gesture infrared images of the current target video stream into the pre-trained static gesture recognition model, so as to obtain the static gesture category and gesture anchor frame corresponding to each frame of gesture image output by the static gesture recognition model.

[0050] Specifically, if Figure 2 As shown, Hagridv2 is a large RGB static gesture dataset that defines 33 static gestures and contains 1,086,158 images. However, the key to this solution is recognizing dynamic gestures. Therefore, rather than requiring a large number of static gestures, only the subset of dynamic gestures can be selected. This improves recall and reduces false positives. Specifically, this solution selects the fist, palm, stop, stop inv, and point gestures from the static gesture dataset as training data.

[0051] Considering that both left-hand and right-hand gesture recognition need to be effective in practical applications, the image needs to be enhanced by random rotation and mirror flipping. Secondly, considering the noise in the ToF module imaging process and the overexposure problem caused by light, the training data is enhanced by random photometric enhancement, random noise, and random erasure. In addition, the model also needs to adapt to the infrared image of the ToF module instead of RGB input, and color information is not that important in the gesture recognition task, so the data is highly grayscale enhanced. In order to ensure the robustness of recognition, mixup data enhancement is introduced. Finally, the processed data can be annotated with gesture anchor boxes to obtain a training sample dataset.

[0052] In the training sample data set, each training sample includes a gesture grayscale image, a static gesture category corresponding to the image, and a gesture anchor frame corresponding to the gesture grayscale image.

[0053] Specifically, this solution can adopt the neural network model yolov5-s. In order to improve its recognition accuracy, the present invention replaces the backbone network of yolov5-s with Mobileone, which enhances the feature extraction capability while only increasing the amount of calculation. The neural network model yolov5-s is trained with the above-mentioned training sample data set until the model parameters converge or the number of iterations is reached, and a static gesture recognition model can be obtained. The process of training the neural network model yolov5-s with the static gesture training sample data set is similar to the training process of any existing static gesture recognition model, and will not be repeated here.

[0054] By obtaining the static gesture category and gesture anchor frame corresponding to each gesture image frame through the above method, this solution can obtain the position information of the target position point in the gesture anchor frame of each gesture image frame. Specifically, as a possible implementation method, this solution can use the center point of the gesture anchor frame as the target position point to obtain the position information of the center point of the gesture anchor frame.

[0055] In the case of obtaining the position information of the target position point in the gesture anchor frame of each frame of gesture image as mentioned above, this solution can project the target position point corresponding to each frame of gesture image into three-dimensional space based on the gesture depth image corresponding to each frame of gesture image and the position information of the target position point, and obtain the three-dimensional information of the target position point corresponding to each frame of gesture image.

[0056] Specifically, this solution can first determine the projection depth value corresponding to each frame of gesture image based on the gesture depth image corresponding to each frame of gesture image, and then based on the camera model projection process, according to the projection depth value corresponding to each frame of gesture image and the position information of the target position point, project the target position point corresponding to each frame of gesture image into three-dimensional space to obtain the three-dimensional information of the target position point corresponding to each frame of gesture image.

[0057] As previously mentioned, each frame in a TOF video stream corresponds to a depth map. Specifically, each pixel in each depth image captured by TOF corresponds to a depth value. As a possible implementation, this solution can determine the depth value of the target location in each gesture depth image as the projected depth value corresponding to each gesture image frame. For example, if the target location is the center point of a gesture anchor frame, this solution can determine the depth value corresponding to the center point of each gesture anchor frame as the projected depth value for the corresponding frame.

[0058] As another possible implementation, in order to avoid the possibility that the depth value corresponding to the center point of the gesture anchor frame may be abnormal due to errors in the TOF camera itself or mirror reflection, in this case, this solution can use neighborhood depth sampling to eliminate such problems.

[0059] Specifically, this solution can obtain the depth value of each location point in the target area of ​​each frame of the gesture image. The target area is the area with the target location point as the center and a preset location point as the radius. Each location point can be a pixel point. For example, this solution can use the center point of the gesture anchor frame as the center and the area with a radius of 6 locations (pixels) as the target area.

[0060] When the depth value of each position point in the target area is obtained, this solution can calculate the average depth value of all position points in the target area of ​​each frame of gesture image to obtain the projection depth value corresponding to each frame of gesture image; or this solution can use the median of the depth values ​​of all position points in the target area of ​​each frame of gesture image as the projection depth value corresponding to each frame of gesture image.

[0061] When this scheme obtains the projection depth value corresponding to each frame of gesture image through the above method, this scheme can use the camera model projection equation to project the target position point corresponding to each frame of gesture image into three-dimensional space based on the projection depth value of each frame of gesture image and the position information of the target position point of the corresponding frame image, and obtain the three-dimensional information of the target position point corresponding to each frame of gesture image. Then, according to the static gesture category corresponding to each frame of gesture image in multiple consecutive frames of gesture images and the three-dimensional information of the target position point, the dynamic gesture category corresponding to the current target video stream is determined.

[0062] The above-mentioned design of the real-time dynamic gesture recognition method uses the infrared image of each frame in the video stream data to identify the static gesture category and gesture anchor frame position of each frame in the video stream data, and then converts the target position point in the gesture into three-dimensional space based on the depth map of each frame in the video stream data and the gesture anchor frame position. The dynamic gesture category is determined based on the static gesture results of consecutive frames and the position changes of the target position points in consecutive frames after conversion to three-dimensional space. In this way, the real-time recognition of dynamic gestures in multiple frames can be achieved by cleverly combining single-frame static gesture recognition with the three-dimensional transformation of the target position point. The memory and computing resource usage are significantly reduced compared to the traditional methods using 3D CNN, long short-term memory network (LSTM) or Transformer models, which enables this solution to run in real time in embedded devices.

[0063] In an optional implementation manner of this embodiment, for step S140, as a possible implementation manner, as follows Figure 3 As shown, this solution can determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each frame of gesture image and the three-dimensional information of the target position point in the following manner, including: Step S300: judging whether there are only gestures of the same attribute category in the multiple frames of gesture images according to the static gesture category corresponding to each frame of gesture image in the multiple frames of gesture images.

[0064] Step S310: If it is determined that only gestures of the same attribute category exist in the multiple gesture image frames, the attribute offset corresponding to the attribute category gesture is calculated based on the three-dimensional information of the target position points corresponding to the multiple gesture image frames.

[0065] Step S320: Determine whether the attribute offset exceeds the offset threshold corresponding to the attribute offset. If so, go to step S330.

[0066] Step S330: Determine the attribute category as the dynamic gesture category corresponding to the current target video stream.

[0067] In the above embodiment, the same attribute category represents gestures belonging to the same attribute category, for example, gestures of the sliding attribute category, gestures of the rotation attribute category, and gestures of the grabbing attribute category. Specifically, according to the static gesture category described above, this solution can classify palm category gestures, stop category gestures, and stop inv category gestures as gestures of the sliding attribute category, this solution can classify point category gestures separately as gestures of the rotation attribute category, and this solution can classify palm category gestures and fist category gestures as gestures of the grabbing attribute category.

[0068] Based on the above classification, this solution can determine whether only gestures of the same attribute category exist in multiple frames of gesture images based on the static gesture category corresponding to each frame of gesture images in the continuous multiple frames of gesture images. For example, based on the static gesture category corresponding to each frame of gesture images in the continuous multiple frames of gesture images, it can be determined whether only palm category gestures, stop category gestures, and stop inv category gestures exist in the multiple frames of gesture images. If so, it is determined that only gestures of the same attribute category (sliding attribute category) exist in the multiple frames of gesture images. For another example, based on the static gesture category corresponding to each frame of gesture images in the continuous multiple frames of gesture images, it can be determined whether only point category gestures exist in the multiple frames of gesture images. If so, it is determined that only gestures of the same attribute category (rotation attribute category) exist in the multiple frames of gesture images. For another example, based on the static gesture category corresponding to each frame of gesture images in the continuous multiple frames of gesture images, it can be determined whether only palm category gestures and fist category gestures exist in the multiple frames of gesture images. If so, it is determined that only gestures of the same attribute category (grasping attribute category) exist in the multiple frames of gesture images.

[0069] Among them, each attribute category gesture has a corresponding attribute offset. For example, the sliding attribute category corresponds to the sliding attribute offset. The sliding attribute offset can be a horizontal offset or a depth offset. The horizontal offset is used to measure the offset degree of the gesture sliding left and right, and the depth offset is used to measure the offset degree of the gesture sliding forward and backward (pushing, pulling); the rotation attribute category corresponds to the rotation attribute offset, which is used to measure the offset degree of the gesture rotation, etc.

[0070] In the above situation, when the present solution determines that only gestures of the same attribute category exist in the multi-frame gesture images, the present solution can calculate the attribute offset corresponding to the attribute category gesture based on the three-dimensional information of the target position point corresponding to the multi-frame gesture images. When the attribute offset corresponding to the attribute category gesture is determined to exceed the offset threshold corresponding to the attribute offset, the present solution determines the attribute category as the dynamic gesture category corresponding to the current target video stream. For example, assuming that the present solution determines that only gestures of the sliding attribute category exist in the multi-frame gesture images, and the present solution calculates that the sliding attribute offset corresponding to the sliding attribute category exceeds the corresponding offset threshold based on the three-dimensional information of the target position point corresponding to the multi-frame gesture images, then the present solution determines that the dynamic gesture category corresponding to the current target video stream is a sliding gesture.

[0071] Furthermore, in order to distinguish between left and right sliding and front and back sliding, this solution can respectively calculate the horizontal offset and depth offset of multiple consecutive frames of images, determine whether the horizontal offset exceeds the corresponding horizontal offset threshold, and determine whether the depth offset exceeds the corresponding depth offset threshold; if it is determined that the horizontal offset exceeds the corresponding horizontal offset threshold, then the dynamic gesture category corresponding to the current target video stream is determined to be left and right sliding; if it is determined that the depth offset exceeds the corresponding depth offset threshold, then the dynamic gesture category corresponding to the current target video stream is determined to be front and back sliding.

[0072] Furthermore, the attribute offset set in this scheme can also be divided into an overall attribute offset and a maximum attribute offset. The overall attribute offset represents the offset of the target position point between the first frame and the last frame, and the maximum attribute offset represents the sum of the offsets of the target position point between each frame and the first frame.

[0073] Specifically, this solution can calculate the offset of the target position point between the first frame and the last frame in a continuous multi-frame gesture image to obtain an overall attribute offset; calculate the sum of the offsets of the target position point of each frame relative to the first frame in the continuous multi-frame gesture image to obtain a maximum attribute offset, and then determine whether the overall attribute offset exceeds the first offset threshold corresponding to the overall attribute offset; and determine whether the maximum attribute offset exceeds the second offset threshold corresponding to the maximum attribute offset; if it is determined that the overall attribute offset exceeds the first offset threshold corresponding to the overall attribute offset, and the maximum attribute offset exceeds the second offset threshold corresponding to the maximum attribute offset, then it is determined that the attribute offset exceeds the offset threshold corresponding to the attribute offset. The calculated offset can be calculated based on the three-dimensional coordinates of the target position point between the two frames of images. For example, the horizontal offset of the target position point between the two frames of images can be calculated by extracting the coordinate difference between the x-coordinate and the y-coordinate of the target position point between the two frames of images, thereby calculating the distance to obtain the horizontal offset; the depth offset is similar, and can be obtained by calculating the z-coordinate difference between the target position point between the two frames of images.

[0074] The above implementation can be summarized by the following examples: If only palm / stop / stop inv. gestures exist in multiple consecutive gesture image frames, and their overall horizontal offset and maximum horizontal offset both exceed the corresponding thresholds, then the dynamic gesture category corresponding to the current target video stream is determined to be a left / right slide gesture; If only palm / stop / stop inv. gestures exist in multiple consecutive gesture image frames, and their total depth offset and maximum depth offset both exceed the corresponding thresholds, then the dynamic gesture category corresponding to the current target video stream is determined to be a left / right slide gesture; If there are only point gestures in multiple consecutive frames of gesture images, and their rotation change exceeds the corresponding threshold, the dynamic gesture category corresponding to the current target video stream is determined to be a rotation gesture.

[0075] In addition, the grabbing attribute category of the gesture can be determined by the order of the gestures. Specifically, if there are only fist / palm category gestures in multiple consecutive frames of gesture images, and they have a continuous order relationship, such as the first N / 2 frames are fist category gestures and the rest are palm category gestures, then the dynamic gesture category corresponding to the current target video stream is determined to be release; otherwise, the dynamic gesture category corresponding to the current target video stream is determined to be grabbing.

[0076] In the above-mentioned implementation mode, this scheme determines the gesture attribute category of continuous multi-frame images by the static gesture category of continuous multi-frame images, and then calculates the attribute offset based on the three-dimensional information change of the target position point of the continuous multi-frame images. Then, when it is determined that the attribute offset exceeds the corresponding threshold, the attribute category is determined to be the dynamic gesture category corresponding to the current target video stream, thereby combining the static gestures in the continuous multi-frame images with the three-dimensional motion information to identify the dynamic gestures, thereby improving the accuracy and robustness of real-time recognition of dynamic gestures.

[0077] In an optional implementation of this embodiment, when the current target video stream generates a large number of gesture images, in order to reduce the complexity of calculation and improve the recognition accuracy, this solution can adopt a sequential sliding window method to determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each gesture image in multiple consecutive frames of gesture images and the three-dimensional information of the target position point, such as Figure 4 Shown, including: Step S400: inputting the static gesture category and the three-dimensional information of the target position point corresponding to each frame of the gesture image in the continuous multiple frames of gesture images into the sequence sliding window in sequence.

[0078] Step S410: determining whether only gestures of the same attribute category exist in the N frames of gesture images in the sequence sliding window according to the static gesture category corresponding to each frame of gesture image in the sequence sliding window.

[0079] Step S420: If it is determined that only gestures of the same attribute category exist in the N gesture images in the sequence sliding window, the attribute offset corresponding to the attribute category gesture is calculated based on the 3D information of the target position point corresponding to the N gesture images in the sequence sliding window.

[0080] Step S430: Determine whether the attribute offset exceeds the offset threshold corresponding to the attribute offset. If so, go to step S440.

[0081] Step S440: identifying the dynamic gesture categories of the N consecutive frames of gesture images within the current sequence sliding window as attribute categories, and recording the recognition results.

[0082] Step S450: After the plurality of consecutive frames of gesture images have passed through the sequence sliding window, each recognition result recorded by the sequence sliding window is obtained.

[0083] Step S460: Determine the dynamic gesture category corresponding to the current target video stream according to each recognition result recorded by the sequence sliding window.

[0084] In the above embodiment, this solution can set a sequential sliding window with a sliding window size of N frames, and the sliding step size of the sliding window is K frames each time. The static gesture category corresponding to each gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point can be input into the sequential sliding window in sequence. The N frames in the sequential sliding window can be used to perform attribute category gesture judgment and offset calculation judgment, and output the dynamic gesture category recognition results corresponding to the N frames of gesture images in the sequential sliding window. After all the continuous multi-frame gesture images have passed through the sequential sliding window, each recognition result recorded by the sequential sliding window is obtained, and then according to each recognition result recorded by the sequential sliding window, the dynamic gesture category corresponding to the current target video stream is determined.

[0085] Specifically, as a possible implementation method, this solution may determine the attribute category with the highest frequency of occurrence in each recognition result recorded by the sequence sliding window as the dynamic gesture category corresponding to the current target video stream.

[0086] It should be noted here that the process of attribute category gesture determination and offset calculation determination for N frames in the sequence sliding window in this embodiment is consistent with the process described above, that is: if there are only palm / stop / stop inv. category gestures in the consecutive N frames of gesture images in the sequence sliding window, and their overall horizontal offset and maximum horizontal offset both exceed the corresponding thresholds, then the dynamic gesture category corresponding to the current target video stream is determined to be a left / right sliding gesture; if there are only palm / stop / stop inv. category gestures in the consecutive N frames of gesture images in the sequence sliding window, Inv. category gesture, and its overall depth offset and maximum depth offset both exceed the corresponding threshold, then the dynamic gesture category corresponding to the current target video stream is determined to be a left / right sliding gesture; if there are only point category gestures in the N consecutive gesture images in the sequence sliding window, and their rotation change exceeds the corresponding threshold, then the dynamic gesture category corresponding to the current target video stream is determined to be a rotation gesture; if there are only fist / palm category gestures in the N consecutive gesture images in the sequence sliding window, and they have a continuous order relationship, such as the first N / 2 frames are fist category gestures and the rest are palm category gestures, then the dynamic gesture category corresponding to the current target video stream is determined to be a release; otherwise, the dynamic gesture category corresponding to the current target video stream is determined to be a grab. The specific and detailed judgment process is the same as the previous article and will not be repeated here.

[0087] The above implementation can be illustrated by the following examples: For example, when the multi-frame gesture image is 30 frames, the sliding window size is 5 frames and the sliding step of the sliding window is 1 frame, the static gesture category and the three-dimensional information of the target position point of the 1st to 5th frames in the multi-frame gesture image can be first input into the sequence sliding window for attribute category gesture judgment and offset calculation judgment, and the dynamic gesture category recognition results corresponding to the 1st to 5th frames are recorded. Then the sequence sliding window slides 1 frame, the 1st frame image in the multi-frame gesture image exits the sliding window, and the 6th frame enters the sliding window. The sequence sliding window performs attribute category gesture judgment and offset calculation judgment on the 2nd to 6th frames therein, and records the 2nd to 5th frames. The dynamic gesture category recognition results corresponding to the 6 frames are obtained, and so on, until the sequence sliding window performs attribute category gesture judgment and offset calculation judgment on the 26th to 30th frames within it, and records the dynamic gesture category recognition results corresponding to the 26th to 30th frames; this scheme can obtain each recognition result recorded by the sequence sliding window, and then determine the dynamic gesture category corresponding to the current target video stream based on each recognition result recorded by the sequence sliding window; assuming that the dynamic gesture category of left and right sliding appears most frequently in the recognition results recorded by the sequence sliding window, then this scheme determines that the dynamic gesture category corresponding to the current target video stream is left and right sliding.

[0088] In the above implementation mode, this solution adopts a sequential sliding window method to determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each gesture image in multiple consecutive frames of gesture images and the three-dimensional information of the target position point. The sequential sliding window is used to limit the size of the gesture image data processed each time, thereby avoiding the problem of a surge in computing power caused by ultra-long video stream data, which occupies huge computing resources. This reduces the complexity of the calculation and reduces the consumption of computing resources, so that when this solution is set in an embedded device, its computing efficiency can be significantly improved, ensuring the real-time operation of the embedded device.

[0089] Figure 5 The present application provides a schematic structural block diagram of a dynamic gesture real-time recognition device. It should be understood that the device is applied to the computing device described above. Figures 1 to 4 The device corresponds to the method embodiment executed in the embodiment and is capable of executing the steps involved in the aforementioned method. The specific functions of the device can be found in the description above, and a detailed description is omitted here to avoid repetition. The device includes at least one software functional module that can be stored in a memory in the form of software or firmware or embedded in the device's operating system (OS). Specifically, the device includes: an acquisition module 500, an identification module 510, a projection module 520 and a determination module 530, wherein the acquisition module 500 is used to acquire a continuous multi-frame gesture image of the current target video stream; wherein the gesture image includes a gesture infrared image and a gesture depth image; the identification module 510 is used to identify the static gesture category and the gesture anchor frame corresponding to each frame gesture image based on the continuous multi-frame gesture infrared image of the current target video stream; the acquisition module 500 is also used to acquire the position information of the target position point in the gesture anchor frame of each frame gesture image; the projection module 520 is used to project the target position point corresponding to each frame gesture image into three-dimensional space based on the gesture depth image corresponding to each frame gesture image and the position information of the target position point, so as to obtain the three-dimensional information of the target position point corresponding to each frame gesture image; the determination module 530 is used to determine the dynamic gesture category corresponding to the current target video stream based on the static gesture category corresponding to each frame gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point.

[0090] The above-designed real-time dynamic gesture recognition device uses the infrared image of each frame in the video stream data to identify the static gesture category and gesture anchor frame position of each frame in the video stream data, and then converts the target position point in the gesture into three-dimensional space based on the depth map of each frame in the video stream data and the gesture anchor frame position. The dynamic gesture category is determined based on the static gesture results of consecutive frames and the position changes of the target position points in consecutive frames after conversion to three-dimensional space. In this way, the real-time recognition of dynamic gestures in multiple frames can be achieved by cleverly combining single-frame static gesture recognition with the three-dimensional transformation of the target position point. The memory and computing resource usage is significantly reduced compared to the traditional methods using 3D CNN, long short-term memory network (LSTM) or Transformer models, which enables this solution to run in real time in embedded devices.

[0091] According to some embodiments of the present application, Figure 6 As shown, the present application provides an electronic device 6, including: a processor 601 and a memory 602, the processor 601 and the memory 602 are interconnected and communicate with each other through a communication bus 603 and / or other forms of connection mechanisms (not shown), the memory 602 stores a computer program executable by the processor 601, and when the computing device is running, the processor 601 executes the computer program to execute any optional implementation method, such as steps S100 to S140: obtaining a continuous multi-frame gesture image of the current target video stream, wherein the gesture image includes a gesture infrared image and a gesture depth image. image; based on the continuous multi-frame gesture infrared images of the current target video stream, identify the static gesture category and gesture anchor frame corresponding to each frame gesture image; obtain the position information of the target position point in the gesture anchor frame of each frame gesture image; based on the gesture depth image corresponding to each frame gesture image and the position information of the target position point, project the target position point corresponding to each frame gesture image into three-dimensional space to obtain the three-dimensional information of the target position point corresponding to each frame gesture image; based on the static gesture category corresponding to each frame gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point, determine the dynamic gesture category corresponding to the current target video stream.

[0092] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method in any of the aforementioned optional implementations is executed.

[0093] The storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0094] The present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the method in any optional implementation manner.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for real-time recognition of dynamic gestures, characterized in that: The method comprises: Acquire a continuous multi-frame gesture image of the current target video stream; wherein the gesture image includes a gesture infrared image and a gesture depth image; According to the continuous multiple frames of gesture infrared images of the current target video stream, identifying the static gesture category and gesture anchor frame corresponding to each frame of gesture image; Get the position information of the target position point in the gesture anchor frame of each frame of gesture image; According to the gesture depth image corresponding to each frame of the gesture image and the position information of the target position point, the target position point corresponding to each frame of the gesture image is projected into three-dimensional space to obtain the three-dimensional information of the target position point corresponding to each frame of the gesture image; The dynamic gesture category corresponding to the current target video stream is determined according to the static gesture category corresponding to each gesture image in the continuous multi-frame gesture image and the three-dimensional information of the target position point.

2. The method according to claim 1, characterized in that The determining of the dynamic gesture category corresponding to the current target video stream according to the static gesture category corresponding to each gesture image in the continuous multiple frames of gesture images and the three-dimensional information of the target position point includes: According to the static gesture category corresponding to each frame of gesture image in the continuous multi-frame gesture image, it is determined whether only gestures of the same attribute category exist in the multi-frame gesture image; If it is determined that only gestures of the same attribute category exist in the multiple frames of gesture images, then calculating the attribute offset corresponding to the attribute category gesture based on the three-dimensional information of the target position point corresponding to the multiple frames of gesture images; Determining whether the attribute offset exceeds an offset threshold corresponding to the attribute offset; If it is determined that the attribute offset exceeds the offset threshold corresponding to the attribute offset, the attribute category is determined as the dynamic gesture category corresponding to the current target video stream.

3. The method according to claim 1, characterized in that The determining of the dynamic gesture category corresponding to the current target video stream according to the static gesture category corresponding to each gesture image in the continuous multiple frames of gesture images and the three-dimensional information of the target position point includes: Inputting the static gesture category and the three-dimensional information of the target position point corresponding to each frame of the gesture image in the continuous multi-frame gesture image into the sequence sliding window in sequence; wherein the size of the sequence sliding window is N frames; According to the static gesture category corresponding to each gesture image in the N frames of gesture images in the sequence sliding window, it is determined whether there are only gestures of the same attribute category in the N frames of gesture images in the sequence sliding window; If it is determined that only gestures of the same attribute category exist in the N frames of gesture images in the sequence sliding window, then calculating the attribute offset corresponding to the attribute category gesture according to the three-dimensional information of the target position point corresponding to the N frames of gesture images in the sequence sliding window; Determining whether the attribute offset exceeds an offset threshold corresponding to the attribute offset; If it is determined that the attribute offset exceeds the offset threshold corresponding to the attribute offset, identifying the dynamic gesture category of the N consecutive frames of gesture images in the current sequence sliding window as the attribute category, and recording the identification result; After multiple consecutive frames of gesture images are passed through the sequence sliding window, each recognition result recorded by the sequence sliding window is obtained; According to each recognition result recorded by the sequence sliding window, the dynamic gesture category corresponding to the current target video stream is determined.

4. The method according to claim 3, characterized in that Determining the dynamic gesture category corresponding to the current target video stream according to each recognition result recorded by the sequence sliding window includes: The attribute category with the highest occurrence frequency in each recognition result recorded by the sequence sliding window is determined as the dynamic gesture category corresponding to the current target video stream.

5. The method according to claim 3, characterized in that The step of calculating the attribute offset corresponding to the attribute category gesture based on the three-dimensional information of the target position point corresponding to the N frames of gesture image in the sequence sliding window includes: According to the three-dimensional information of the target position point corresponding to the N-frame gesture image in the sequence sliding window, the offset of the target position point between the first frame and the last frame in the N-frame gesture image in the sequence sliding window is calculated to obtain the overall attribute offset; Based on the three-dimensional information of the target position point corresponding to the N frames of gesture images in the sequence sliding window, the sum of the offsets of the target position point of each frame in the N frames of gesture images in the sequence sliding window relative to the first frame is calculated to obtain the maximum attribute offset; The determining whether the attribute offset exceeds an offset threshold corresponding to the attribute offset includes: Determining whether the overall attribute offset exceeds a first offset threshold corresponding to the overall attribute offset; and determining whether the maximum attribute offset exceeds a second offset threshold corresponding to the maximum attribute offset; If it is determined that the overall attribute offset exceeds the first offset threshold corresponding to the overall attribute offset, and the maximum attribute offset exceeds the second offset threshold corresponding to the maximum attribute offset, then it is determined that the attribute offset exceeds the offset threshold corresponding to the attribute offset.

6. The method according to claim 1, characterized in that The method of projecting the target position point corresponding to each frame of the gesture image into a three-dimensional space based on the gesture depth image corresponding to each frame of the gesture image and the position information of the target position point to obtain the three-dimensional information of the target position point corresponding to each frame of the gesture image includes: Determine the projection depth value corresponding to each frame of the gesture image according to the gesture depth image corresponding to each frame of the gesture image; Based on the camera model projection process, the target position point corresponding to each frame of gesture image is projected into three-dimensional space according to the projection depth value corresponding to each frame of gesture image and the position information of the target position point, so as to obtain the three-dimensional information of the target position point corresponding to each frame of gesture image.

7. The method according to claim 6, characterized in that The determining, based on the gesture depth image corresponding to each frame of the gesture image, a projection depth value corresponding to each frame of the gesture image, includes: The depth value of the target position point in each frame of the gesture depth image is determined as the projection depth value corresponding to each frame of the gesture image.

8. The method according to claim 6, characterized in that The determining, based on the gesture depth image corresponding to each frame of the gesture image, a projection depth value corresponding to each frame of the gesture image, includes: Obtaining the depth value of each position point in the target area of ​​each frame of the gesture image; wherein the target area is an area range with the target position point as the center and a preset number of position points as the radius; Calculate the average depth value of all positions in the target area of ​​each frame of gesture image to obtain the corresponding projection depth value of each frame of gesture image; Alternatively, the median of the depth values ​​of all the position points in the target area of ​​each frame of the gesture image is used as the corresponding projection depth value of each frame of the gesture image.

9. The method according to claim 1, characterized in that The step of identifying the static gesture category and the gesture anchor frame corresponding to each frame of the gesture image based on the continuous multiple frames of gesture infrared images of the current target video stream includes: The continuous multi-frame gesture infrared images of the current target video stream are input into a pre-trained static gesture recognition model to obtain the static gesture category and gesture anchor frame corresponding to each frame of gesture image output by the static gesture recognition model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images