Gesture recognition method and electronic equipment

Abstract

The embodiment of the invention discloses a gesture recognition method and electronic equipment. The method comprises the following steps: performing image acquisition on a user gesture through an image acquisition device to obtain a continuous video sequence; performing gesture region extraction on each image frame in the continuous video sequence to obtain a gesture image, and performing feature extraction on the gesture image to obtain spatial stream features; performing optical flow calculation on two continuous frames of images to obtain optical flow images, and performing feature extraction on the optical flow images to obtain time flow features; and fusing the spatial flow features and the time flow features to obtain fused features, and performing gesture classification on the fused features based on a classifier to determine a gesture recognition result. According to the scheme, the space features and the time features can be subjected to cross-modal correlation fusion, so that the two features are complementary in gesture recognition, the motion features can know that the apparent features pay attention to a dynamic area, the border features can correct misjudgment of the operation features, and the accuracy of gesture recognition is improved.

Description

Technical Field

[0001] This application relates to the field of gesture recognition technology, and in particular to a gesture recognition method and electronic device. Background Technology

[0002] Gesture recognition is a computer vision technology that uses mathematical algorithms to recognize human gestures, enabling natural human-computer interaction. It primarily captures hand movements or postures and converts them into digital commands, allowing users to control operations without touching the device. Specifically, vision-based gesture recognition typically uses cameras or sensors to acquire image sequences, which undergo preprocessing, hand detection, key point estimation, and trajectory analysis before finally mapping the results into control commands.

[0003] However, current solutions rely on manually designed features (such as HOG, optical flow histograms, etc.) combined with classifiers (such as SVM, HMM) for recognition. These methods are effective in simple scenes, but their feature representation capabilities are limited, making it difficult to handle complex backgrounds, lighting changes, and diverse gesture shapes. Summary of the Invention

[0004] This application provides a gesture recognition method and electronic device that can combine temporal and spatial features to achieve feature complementarity and more accurate feature expression, thereby improving the accuracy of gesture recognition.

[0005] According to one aspect of this application, a gesture recognition method is provided, the method comprising:

[0006] A continuous video sequence is obtained by capturing images of the user's gestures using an image capture device;

[0007] Gesture images are obtained by extracting gesture regions from each image frame in the continuous video sequence, and spatial flow features are obtained by extracting features from the gesture images.

[0008] For two consecutive frames in the continuous video sequence, optical flow is calculated on the two consecutive frames to obtain an optical flow image, and feature extraction is performed on the optical flow image to obtain temporal flow features;

[0009] The spatial flow features and the temporal flow features are fused to obtain fused features, and the gesture recognition result is determined by classifying the fused features based on a classifier.

[0010] According to one aspect of this application, a gesture recognition device is provided, the device comprising:

[0011] The image acquisition module is used to acquire images of user gestures through an image acquisition device to obtain a continuous video sequence;

[0012] The spatial flow feature extraction module is used to extract gesture regions from each image frame in the continuous video sequence to obtain gesture images, and to extract features from the gesture images to obtain spatial flow features.

[0013] The temporal flow feature extraction module is used to perform optical flow calculation on two consecutive frames of images in the continuous video sequence to obtain an optical flow image, and to extract features from the optical flow image to obtain temporal flow features.

[0014] The gesture recognition module is used to fuse the spatial flow features and the temporal flow features to obtain fused features, and to classify the gestures based on the fused features using a classifier to determine the gesture recognition result.

[0015] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and,

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the gesture recognition method of any embodiment of this application.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the gesture recognition method of any embodiment of this application.

[0020] According to another aspect of this application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the gesture recognition method of any embodiment of this application.

[0021] The technical solution of this application embodiment acquires a continuous video sequence by capturing user gestures using an image acquisition device; gesture regions are extracted from each image frame in the continuous video sequence to obtain gesture images, and spatial flow features are extracted from the gesture images; optical flow is calculated on two consecutive frames in the continuous video sequence to obtain optical flow images, and temporal flow features are extracted from the optical flow images; the spatial flow features and temporal flow features are fused to obtain fused features, and gesture classification is performed based on the fused features to determine the gesture recognition result. The above solution can acquire a continuous video sequence of user gestures using an image acquisition device, and by combining multi-dimensional analysis of spatial flow features and temporal flow features, it achieves efficient and robust gesture recognition, possessing significant technical advantages and application value. Spatial flow features, by extracting morphological details from a single frame of gesture image, can accurately capture the static structural information of the gesture, providing basic features for recognition; temporal flow features, by calculating motion changes between consecutive frames using optical flow, reflect the dynamic trajectory and speed characteristics of the gesture, effectively compensating for the shortcomings of static features in dynamic scenes. The fusion of these two approaches not only enhances the comprehensiveness of features but also improves the model's robustness to complex backgrounds, lighting variations, or partial occlusion, ensuring a high recognition rate even in suboptimal environments. Furthermore, the aforementioned scheme utilizes the temporal information of continuous video sequences to capture transient changes in gestures through optical flow calculations, significantly improving the recognition ability for fast movements or subtle gestures and avoiding misjudgments that may occur with traditional single-frame analysis. The feature fusion mechanism integrates spatial and temporal information through weighted or neural networks, optimizing feature representation and enabling the classifier to make decisions more efficiently, thereby improving recognition accuracy and real-time performance.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0025] Figure 2 A flowchart illustrating a gesture recognition method provided in another embodiment of this application;

[0026] Figure 3 A flowchart of a gesture recognition method provided in another embodiment of this application;

[0027] Figure 4 A flowchart illustrating a specific implementation method provided in this application embodiment;

[0028] Figure 5 This is a schematic diagram of the structure of a gesture recognition device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. The acquired data is obtained with authorization and will not be disclosed without permission, used for illegal purposes, purposes detrimental to the interests of others, or for personalized analysis or product promotion. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary and intended only to illustrate the feasibility of implementing the technical solution of this application, but do not imply that the applicant has already used or necessarily used the relevant content of such solutions.

[0033] Figure 1This is a flowchart illustrating a gesture recognition method provided in an embodiment of this application. This embodiment is applicable to situations involving the identification of malicious payloads. The method can be executed by a gesture recognition device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0034] S110. A continuous video sequence is obtained by capturing images of the user's gestures using an image acquisition device.

[0035] The image acquisition device can be a device set up in a specific application scenario to acquire images of user gestures. The specific application scenario can be a scenario that requires recognizing user gestures and executing corresponding instructions based on user gestures. For example, a robot recognizes user gestures and executes corresponding instructions and operations. In this case, the image acquisition device is a device set up on the robot that can acquire images of the gestures of surrounding users.

[0036] In this embodiment, an image acquisition device can continuously capture images of a user's hand to obtain multiple consecutive frames of user gesture images, forming a continuous video sequence. The event that triggers the image acquisition device to capture images of the user's hand can be the image acquisition device detecting the user's hand, recognizing hand movement, or recognizing external trigger events such as user voice commands. The frequency of image acquisition by the image acquisition device can be set according to actual conditions.

[0037] S120. Perform gesture region extraction on each image frame in the continuous video sequence to obtain a gesture image, and perform feature extraction on the gesture image to obtain spatial flow features.

[0038] For example, for a continuous video sequence, each image frame can be processed separately. During the processing, gesture regions can be extracted from each image frame to extract gesture images, which makes it easier to directly recognize and determine the user's gesture commands based on the gesture images.

[0039] In the process of extracting gesture regions from image frames, hand recognition technology can be used to identify hands within the image frame. This could be based on hand shape recognition or skin tone recognition to determine the pixel regions belonging to the user's hand, thus obtaining the gesture region. The image within this gesture region is then extracted as the gesture image. After extraction, the gesture region can be overlaid onto a monochrome background image to form a gesture image of a specified size.

[0040] In this embodiment of the application, in order to further recognize gesture images, spatial flow features, that is, spatially distributed visual gesture features, can be extracted from the gesture images. During the feature extraction process, the gesture image can be input into a feature extraction model for feature extraction, and the extracted features can be used as spatial flow features.

[0041] S130. For two consecutive frames in the continuous video sequence, optical flow calculation is performed on the two consecutive frames to obtain an optical flow image, and feature extraction is performed on the optical flow image to obtain temporal flow features.

[0042] In this embodiment, the user's gesture command may be a dynamic one, meaning the gesture changes over time. In this case, the change in the user's gesture can be detected. Specifically, for two consecutive frames in a continuous video sequence, optical flow calculation can be performed on the two frames to obtain the optical flow value between them, and the optical flow image can be determined based on the optical flow value.

[0043] Specifically, for consecutive video frames, the t-th frame and the... The frame uses the FlowNet optical flow computation network to generate X-direction and Y-direction optical flow maps representing pixel-level motion information. These two maps are combined into a two-channel image to obtain the optical flow image.

[0044] To facilitate subsequent recognition of optical flow images, temporal flow features can be extracted from the optical flow images. The optical flow image can be input into a feature extraction model, and the output features after feature extraction can be used as the temporal flow features.

[0045] S140. The spatial flow features and the temporal flow features are fused to obtain fused features, and the gestures are classified based on the fused features using a classifier to determine the gesture recognition result.

[0046] For example, after obtaining spatial flow features and temporal flow features, the spatial flow features and temporal flow features can be fused to obtain fused features. The fusion of spatial flow features and temporal flow features can be achieved by directly concatenating the spatial flow features and temporal flow features, by weighted concatenation, or by fusing spatial flow features and temporal flow features based on an attention mechanism.

[0047] In this embodiment, a classifier can be pre-trained to classify the user's gesture features to obtain corresponding gesture commands. The fused features can be input into the classifier, and the classifier can classify the gestures based on the fused features to determine the gesture recognition result.

[0048] The technical solution of this application embodiment acquires a continuous video sequence by capturing user gestures using an image acquisition device; gesture regions are extracted from each image frame in the continuous video sequence to obtain gesture images, and spatial flow features are extracted from the gesture images; optical flow is calculated on two consecutive frames in the continuous video sequence to obtain optical flow images, and temporal flow features are extracted from the optical flow images; the spatial flow features and temporal flow features are fused to obtain fused features, and gesture classification is performed based on the fused features to determine the gesture recognition result. The above solution can acquire a continuous video sequence of user gestures using an image acquisition device, and by combining multi-dimensional analysis of spatial flow features and temporal flow features, it achieves efficient and robust gesture recognition, possessing significant technical advantages and application value. Spatial flow features, by extracting morphological details from a single frame of gesture image, can accurately capture the static structural information of the gesture, providing basic features for recognition; temporal flow features, by calculating motion changes between consecutive frames using optical flow, reflect the dynamic trajectory and speed characteristics of the gesture, effectively compensating for the shortcomings of static features in dynamic scenes. The fusion of these two approaches not only enhances the comprehensiveness of features but also improves the model's robustness to complex backgrounds, lighting variations, or partial occlusion, ensuring a high recognition rate even in suboptimal environments. Furthermore, the aforementioned scheme utilizes the temporal information of continuous video sequences to capture transient changes in gestures through optical flow calculations, significantly improving the recognition ability for fast movements or subtle gestures and avoiding misjudgments that may occur with traditional single-frame analysis. The feature fusion mechanism integrates spatial and temporal information through weighted or neural networks, optimizing feature representation and enabling the classifier to make decisions more efficiently, thereby improving recognition accuracy and real-time performance.

[0049] Figure 2 The flowchart illustrates another embodiment of a gesture recognition method provided in this application. This application's embodiment is an optimization based on the above embodiment; solutions not described in detail in this application's embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0050] S210. A continuous video sequence is obtained by capturing images of the user's gestures using an image acquisition device.

[0051] S220. For each image frame in the continuous video sequence, generate a binary mask image of hand skin color based on the color features of the image frame.

[0052] For example, typically, image capture devices capture images of specific areas, meaning they only capture images of the user's hands and not the face. In the captured image frames, the hands are usually skin-colored, while other background areas are not. A binary mask image of the hand's skin color can be generated based on the color features of each image frame in a continuous video sequence. A binary mask image contains only two RGB values, typically represented by 0 and 255. This can be achieved by assigning 255 to pixel areas belonging to the hand based on color features, and assigning 0 to other pixel areas, or vice versa, as long as the pixel values ​​of the hand area differ from other pixel areas.

[0053] In this embodiment of the application, generating a binary mask image of hand skin color based on the color features of the image frame includes:

[0054] Convert the image frame from the RGB color space to the HSV color space;

[0055] For each pixel in the image frame, the value of the H channel is compared with a preset hue range, and the value of the S channel is compared with a preset saturation range to filter out skin color pixels whose color features belong to the skin color of the hand; wherein, the preset hue range and the preset saturation range are determined based on the clustering characteristics of the skin color values ​​in the H channel and S channel in the sample image;

[0056] By setting the skin color pixels to different values ​​from other pixels, a binary mask image of the hand's skin color is obtained.

[0057] For example, an image frame is converted from the RGB color space to the HSV color space. For each pixel in the image frame, the H channel value is compared with a preset hue range, and the S channel value is compared with a preset saturation range to filter out skin-colored pixels whose color features belong to the skin tone of the hand. In this embodiment, the preset hue range and preset saturation range can be pre-determined based on the clustering characteristics of the skin tone values ​​in the H and S channels of the sample image. Specifically, in the sample image, pixels with skin tones are pre-labeled. The skin-colored pixels are converted from the RGB color space to the HSV color space, and the H channel and S channel values ​​corresponding to each skin-colored pixel are counted. The preset saturation range is determined based on the upper and lower limits of all H channel values. Similarly, the preset hue range can be determined based on the upper and lower limits of all S channel values.

[0058] In an image frame, if the H channel value is within a preset hue range and the S channel value is within a preset saturation range, then the pixel is determined to be a skin color pixel. The region composed of all skin color pixels is the hand region. The skin color pixels are then set to different values ​​than other pixels, either 0 or 255, to obtain a binary mask image of the hand's skin color.

[0059] S230. For each image frame in the continuous video sequence, generate a hand depth binary mask image based on the depth information in the image frame; wherein the depth of the hand in the image frame is different from the depth of the background; the hand depth binary mask image and the hand skin color binary mask image assign the same value to the hand pixels.

[0060] For example, in general, the user's hand is closest to the image sensor relative to other parts of the body and the background. Therefore, in an image frame, the depth of the user's hand is different from the depth of other parts of the body and the background, and the hand can be distinguished from the background based on the depth information.

[0061] Specifically, for each image frame in a continuous video sequence, depth information in the image frame is identified. Based on the depth information, pixel regions belonging to the hand are determined. These hand regions are then assigned different pixel values ​​than other background regions, resulting in a hand depth binary mask image. The process of determining the hand depth binary mask image is the same as the process of determining the hand skin color binary mask image; the pixel values ​​of the hand region in the hand depth binary mask image are consistent with the pixel values ​​of the hand region in the hand skin color binary mask image.

[0062] Specifically, in determining the depth of the binary mask image for the hand, the principle can be that the depth of the hand in the image frame differs from the depth of the background; the hand has a smaller depth, while the background has a larger depth. Therefore, the hand region with the smaller depth is extracted. Specifically, a depth threshold can be set, identifying pixels smaller than the threshold as pixels in the hand region, and pixels greater than or equal to the threshold as pixels in other background regions.

[0063] S240. Generate a hand binary mask image based on the hand skin color binary mask image and the hand depth binary mask image, and extract the gesture image from the corresponding image frame based on the hand binary mask image.

[0064] For example, a binary mask image of the hand is generated based on a binary mask image of the hand skin color and a binary mask image of the hand depth. This is a mask image that reflects the hand region. A gesture image is then extracted from the corresponding image frame based on the binary mask image of the hand. In other words, a gesture image is extracted from the corresponding image frame based on the hand region reflected by the binary mask image of the hand.

[0065] In this embodiment of the application, generating a hand binary mask image based on the hand skin color binary mask image and the hand depth binary mask image includes:

[0066] The intersection of the hand skin color binary mask image and the hand region of the hand depth binary mask image is used to obtain the candidate gesture region;

[0067] The candidate gesture region is processed by smoothing the boundaries, filling the internal holes, and removing noise to obtain the target gesture region;

[0068] The target gesture region is assigned a different value from other regions to obtain the binary mask image of the hand.

[0069] For example, candidate gesture regions can be obtained by intersecting the hand region in both the hand skin color binary mask image and the hand depth binary mask image. During the intersection process, if the hand region in both the hand depth binary mask image and the hand skin color binary mask image is represented by 255, a logical AND operation is performed; otherwise, an OR operation is performed. After obtaining the candidate gesture regions through intersection, the candidate gesture regions are processed by smoothing boundaries, filling internal voids, and removing noise to obtain the target gesture region. The target gesture region is then assigned a different value than other regions to obtain the hand binary mask image; that is, the target gesture region is assigned a value of 0, and other regions are assigned a value of 255. Alternatively, the target gesture region can be assigned a value of 255, and other regions can be assigned a value of 0.

[0070] S250. Perform feature extraction on the gesture image to obtain spatial flow features.

[0071] S260. For two consecutive frames in the continuous video sequence, optical flow calculation is performed on the two consecutive frames to obtain an optical flow image, and feature extraction is performed on the optical flow image to obtain temporal flow features.

[0072] S270. The spatial flow features and the temporal flow features are fused to obtain fused features, and the gestures are classified based on the fused features using a classifier to determine the gesture recognition result.

[0073] The technical solution of this application embodiment generates a hand skin color binary mask image based on the color features of each image frame in the continuous video sequence; it also generates a hand depth binary mask image based on the depth information in each image frame in the continuous video sequence; wherein the depth of the hand in the image frame is different from the depth of the background; the values ​​assigned to the hand pixels in the hand depth binary mask image and the hand skin color binary mask image are the same; a hand binary mask image is generated based on the hand skin color binary mask image and the hand depth binary mask image, and a gesture image is extracted from the corresponding image frame based on the hand binary mask image. This solution significantly improves the accuracy and robustness of gesture recognition by fusing color features and depth information, providing reliable technical support for gesture interaction in complex scenarios. Generating a hand skin color binary mask image based on color features utilizes the distribution characteristics of skin color in the HSV color space, effectively capturing the hand region through threshold segmentation of the H and S channels, and simultaneously using morphological operations to denoise and fill holes, ensuring that the skin color mask can accurately locate the hand contour. A binary depth mask image for the hand is generated based on depth information. Using depth data acquired by a depth sensor, a reasonable threshold is set to distinguish the depth difference between the hand and the background, avoiding false detections caused by lighting variations or background interference, and ensuring that the mask only retains the true hand pixels. The collaborative fusion of skin color and depth masks, combined through logic and operations, significantly reduces the risk of misjudgment due to single features. For example, in complex backgrounds or uneven lighting conditions, depth information can effectively filter out non-hand areas with similar colors, while skin color features compensate for the lack of detail capture in depth data. The generated binary hand mask image can accurately locate the gesture region, and through mask application, a clean gesture image can be extracted from the original image frame, providing high-quality input for subsequent gesture recognition, interactive control, and other applications.

[0074] In the embodiments of this application, during the training process of the feature extraction model for spatial flow features and the feature extraction model for temporal flow features, the model parameters of the first standard convolutional layer and the three adjacent inverted residual bottleneck blocks are shared, while other parameters are individually trained and optimized.

[0075] To reduce the number of model parameters and improve training efficiency, the first standard convolutional layer and the first three Bottleneck blocks of the spatial flow feature processing network and the temporal flow feature processing network (specifically, sharing the first standard convolutional layer (3 channels → 16 channels) of the spatial flow and temporal flow networks and the subsequent first three inverted residual bottleneck blocks (16 → 16, 16 → 24, 24 → 24 channels respectively) share weights.

[0076] Figure 3This is a flowchart illustrating a gesture recognition method according to another embodiment of this application. This embodiment is an optimization based on the above embodiments; solutions not described in detail in this embodiment are found in the above embodiments. Figure 3 As shown, the method in this embodiment of the application specifically includes the following steps:

[0077] S310. A continuous video sequence is obtained by capturing images of the user's gestures using an image acquisition device.

[0078] S320. Perform gesture region extraction on each image frame in the continuous video sequence to obtain a gesture image, and perform feature extraction on the gesture image to obtain spatial flow features.

[0079] S330. For two consecutive frames in the continuous video sequence, optical flow calculation is performed on the two consecutive frames to obtain an optical flow image, and feature extraction is performed on the optical flow image to obtain temporal flow features.

[0080] S340. Based on the cross-attention mechanism, the spatial flow features and the temporal flow features are processed to obtain the corresponding spatial features and temporal features respectively.

[0081] The spatial flow features and temporal flow features are processed based on a cross-attention mechanism to obtain corresponding spatial and temporal features, including:

[0082] A query matrix is ​​obtained by performing a linear transformation on the spatial flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the temporal flow features. A first attention output is then calculated based on the query matrix, the key matrix, and the value matrix.

[0083] The spatial features are determined based on the first attention output and the spatial flow features;

[0084] A query matrix is ​​obtained by performing a linear transformation on the temporal flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the spatial flow features. A second attention output is then calculated based on the query matrix, the key matrix, and the value matrix.

[0085] The time feature is determined based on the second attention output and the time flow feature.

[0086] Exemplary, original spatial flow characteristics and time flow characteristics First, dimensional alignment and feature enhancement are performed using independent 1×1 convolutional layers to obtain aligned spatial features. and time characteristics Then, the user enters the feature interaction module for interactive calculations.

[0087] Spatial features as queries:

[0088] ;

[0089] Time features as keys:

[0090] ;

[0091] Time features as values:

[0092] ;

[0093] Calculate the first attention output:

[0094] ;

[0095] Spatial characteristics:

[0096] ;

[0097] Spatial features as queries:

[0098] ;

[0099] Time features as keys:

[0100] ;

[0101] Time features as values:

[0102] ;

[0103] Calculate the second attention output:

[0104] ;

[0105] Time characteristics:

[0106] .

[0107] S350. Perform pixel-level weighted fusion of the spatial features and the temporal features to obtain the fused features.

[0108] For example, pixel-level weighted fusion is performed based on the weights corresponding to spatial and temporal features respectively to obtain fused features, which are represented as follows: ;in, For spatial features, The weights corresponding to the spatial features. As a time feature, These are the weights corresponding to the time features.

[0109] In this embodiment of the application, the spatial features and the temporal features are fused using pixel-level weighted fusion to obtain the fused features, including:

[0110] The spatial features and the temporal features are spliced ​​together to obtain spliced ​​features, and channel descriptors are determined by statistical calculation based on the spatial position of each channel of the spliced ​​features.

[0111] Generate a first weight matrix corresponding to the spatial features and a second weight matrix corresponding to the temporal features based on the channel descriptor;

[0112] The spatial features and the temporal features are weighted and fused based on the first weight matrix and the second weight matrix to obtain the fused features.

[0113] For example, splicing spatial features and temporal features together yields a spliced ​​feature, represented as follows: Global average pooling: The channel descriptor z is obtained by averaging the spatial locations (H×W) of each channel in the concatenated feature: z ∈ Each element represents global statistics for one channel:

[0114] ;

[0115] Where H is the pixel height and W is the pixel width.

[0116] The channel descriptor is input into a multilayer perceptron with a bottleneck structure to obtain a feature vector. The feature vector is combined with the downsampled spatial features and input into the weight generation layer, which is a 1×1 convolutional layer, to obtain the first weight matrix corresponding to the spatial features. Based on the principle that the sum of the first weight matrix and the second weight matrix is ​​1, the second weight matrix is ​​determined. .

[0117] Specifically, within the weight generation network, spatial features are downsampled from (H, W) to (H', W') through convolution in the first or second layer, where H' = H / 2ⁿ and W' = W / 2ⁿ; deep feature extraction is performed through multiple convolutional layers, maintaining a resolution of (H', W'); and bi-branch convolution generates α, β ∈ Upsampling: Upsample α and β back to (H, W) to obtain as well as .

[0118] S360. Based on the classifier, perform gesture classification on the fused features to determine the gesture recognition result.

[0119] The scheme in this embodiment processes the spatial flow features and temporal flow features based on a cross-attention mechanism to obtain corresponding spatial and temporal features respectively; the spatial and temporal features are then fused at the pixel level to obtain the fused features. This scheme, by introducing a cross-attention mechanism, achieves deep interaction and fusion of spatial and temporal flow features, significantly improving the accuracy and robustness of video gesture recognition in complex scenes. Specifically, the spatial flow features are processed using a cross-attention mechanism, capturing the spatial correlation between pixels within an image frame through a self-attention mechanism. Simultaneously, temporal information from the temporal flow features is introduced through a cross-attention mechanism, enabling the spatial features to not only focus on the hand shape in the current frame but also dynamically correlate the motion trajectories of adjacent frames, thereby effectively solving the problem of local feature loss due to changes in viewpoint or occlusion. Applying a cross-attention mechanism to the temporal flow features, by modeling the long-term dependence of gesture actions through temporal self-attention, combined with the spatial context provided by the spatial flow features, enhances the temporal features' ability to capture dynamic gestures, avoiding the breakage of temporal information caused by rapid movement or background interference. Building upon this foundation, the processed spatial and temporal features are fused using pixel-level weighting. A learnable weighting mechanism dynamically adjusts the contribution ratio of spatial and temporal features; for example, in static gesture recognition, the weight of spatial features is enhanced, while in dynamic gesture tracking, the temporal information of the temporal features is emphasized, thus achieving adaptive complementarity at the feature level. The fused features not only retain the high-resolution details of spatial features but also incorporate the long-term temporal information of temporal features, providing richer and more discriminative input for subsequent gesture recognition models. This approach is particularly suitable for complex backgrounds, scenes with changing lighting, or fast-moving scenarios, significantly improving the accuracy and stability of gesture recognition and providing reliable technical support for applications such as intelligent interaction and virtual reality.

[0120] In this embodiment of the application, after fusing the spatial flow features and the temporal flow features to obtain fused features, the method further includes:

[0121] The fused features are processed by a gated loop unit to extract temporal dependency features;

[0122] Accordingly, gesture classification is performed on the fused features based on the classifier to determine the gesture recognition result, including:

[0123] The time-dependent features are input into the classifier to classify gestures and determine the gesture recognition result.

[0124] For example, input features Historical Status: , (This is the "hidden state" of a gated recurrent unit, a kind of "memory" or "contextual information" in a neural network.) : The hidden state at the previous moment.

[0125] Reset Door:

[0126] ;

[0127] Update Gate:

[0128] ;

[0129] Candidate state:

[0130] ;

[0131] New status:

[0132] ;

[0133] Enhanced features:

[0134] ;

[0135] Gating signal:

[0136] The gate output is reset and calculated using the sigmoid function.

[0137] Update the gate output, calculated using the sigmoid function;

[0138] Candidate hidden states, calculated using tanh;

[0139] The new hidden state is obtained through weighted sum calculation;

[0140] Reset door parameters: : Weight matrix, where It is a hidden state dimension; Bias vector. Update gate parameters: Weight matrix; Bias vector. Candidate state parameters: Weight matrix; Bias vector. Output transformation parameters: The weight matrix is ​​used to map the hidden states back to the feature space. This mechanism enables the system to capture the temporal dependencies of gesture actions, improving the stability of recognizing continuous actions (such as "open your hand" or "swipe left").

[0141] This application provides a specific implementation method, such as Figure 4 As shown, specifically:

[0142] Step 1: Video Sequence Acquisition and Input

[0143] Capture a continuous sequence of video frames containing user gestures using a camera (supporting RGB and RGB-D types).

[0144] Step 2: Adaptive Threshold Gesture Segmentation

[0145] Precise gesture foreground segmentation is performed on each input frame of the image. This incorporates HSV color space analysis and depth information.

[0146] Color segmentation: Convert the RGB image to the HSV color space. Based on the clustering characteristics of skin color in the H (hue) and S (saturation) channels, dynamically calculate the adaptive threshold for the current frame to generate an initial binary mask for skin color.

[0147] Depth segmentation: If the system is equipped with a depth camera, a depth map is acquired simultaneously. Utilizing the prior knowledge that the subject making the gesture is usually closer to the camera, a depth threshold is set, and a binary mask based on depth differences is generated.

[0148] Fusion and Post-processing: A logical AND operation is performed on the color mask and depth mask to obtain the fused candidate gesture regions. Morphological operations are then performed to smooth boundaries, fill internal holes, and remove small-area noise, ultimately outputting an accurate binary gesture mask. This mask is used to crop the gesture region from the original image, serving as input for subsequent spatial flow.

[0149] Step 3: Optical Flow Calculation

[0150] For consecutive video frames, frame t and frame t... The frame uses the FlowNet optical flow computation network to generate X- and Y-direction optical flow maps representing pixel-level motion information. These two maps are combined into a two-channel image, which serves as the input to the temporal stream.

[0151] Step 4: Spatiotemporal dual-stream feature extraction

[0152] Construct a lightweight spatiotemporal dual-stream convolutional neural network.

[0153] Spatial Flow Network: Using the RGB image of the gesture region obtained in step 2 as input, a lightweight CNN network, MobileNetV3-small, is used as the backbone to extract static appearance features such as the shape and texture of the gesture, and outputs a feature vector. .

[0154] Temporal Flow Network: Using the optical flow image obtained in step 3 as input, a network that shares weights with the spatial flow part of the convolutional layers is employed to specifically learn the temporal features such as the motion trajectory and velocity of the gesture, and outputs a feature vector. .

[0155] Weight sharing design: To reduce the number of model parameters and improve training efficiency, the first standard convolutional layer and the first three Bottleneck blocks of the spatial flow and temporal flow network (specifically, the first standard convolutional layer of the spatial flow and temporal flow network (3 channels → 16 channels) and the subsequent first three inverted residual bottleneck blocks (16 → 16, 16 → 24, 24 → 24 channels respectively) share weights.

[0156] Step 5: Attention-based feature fusion

[0157] The feature fusion module abandons the traditional simple weighting or concatenation method and adopts a multi-layered, adaptive attention fusion architecture, as implemented below:

[0158] Feature preprocessing and interaction layer: raw spatial features and time characteristics First, dimensional alignment and feature enhancement are performed using independent 1×1 convolutional layers to obtain aligned spatial features. and time characteristics Then, the user enters the feature interaction module for interactive calculations.

[0159] Spatial features as queries:

[0160] ;

[0161] Time features as keys:

[0162] ;

[0163] Time features as values:

[0164] ;

[0165] Calculate the first attention output:

[0166] ;

[0167] Spatial characteristics:

[0168] ;

[0169] Spatial features as queries:

[0170] ;

[0171] Time features as keys:

[0172] ;

[0173] Time features as values:

[0174] ;

[0175] Calculate the second attention output:

[0176] ;

[0177] Time characteristics:

[0178] .

[0179] Through this bidirectional cross-attention mechanism, the features of the two modalities establish a deep semantic association before fusion.

[0180] Dynamic spatial weight generation network:

[0181] A lightweight spatial weight generation network was designed to produce pixel-level fusion weights. Input: Spatial features obtained from cross-attention. and time characteristics The splicing characteristics:

[0182] ;

[0183] Specific steps:

[0184] 1. Take the average of all spatial locations (H×W) for each channel of the splicing feature to obtain the channel descriptor z: z ∈ Each element represents global statistics for one channel:

[0185] ;

[0186] Where H is the pixel height and W is the pixel width.

[0187] 2. Input the channel descriptor into a multilayer perceptron with a bottleneck structure to obtain the feature vector f.

[0188] 3. The feature vector and the downsampled spatial features are combined and input into the weight generation layer, which is a 1×1 convolutional layer, to obtain the first weight matrix corresponding to the spatial features. Based on the principle that the sum of the first weight matrix and the second weight matrix is ​​1, the second weight matrix is ​​determined. .

[0189] 4. Pixel-level weighted blending:

[0190] α and β are upsampled to the spatial resolution of the original feature map using bilinear interpolation. ,get and

[0191] ;

[0192] This design allows the network to employ different fusion strategies in different image regions (such as the edges of a rapidly moving hand versus the center of a relatively stationary palm).

[0193] Temporal context enhancement module (for video sequence processing):

[0194] For consecutive video frames, the output sequence of the pixel-level weighted fusion module, this invention also introduces a temporal context enhancement mechanism, input features Historical status: , but:

[0195] Reset Door:

[0196] ;

[0197] Update Gate:

[0198] ;

[0199] Candidate state:

[0200] ;

[0201] New status:

[0202] ;

[0203] Enhanced features:

[0204] ;

[0205] This mechanism enables the system to capture the temporal dependencies of gestures, improving the stability of recognizing continuous actions (such as "opening hands" or "swiping left").

[0206] Step 6: Gesture Classification and Output

[0207] The fused feature vector The input is fed into a fully connected layer classifier (followed by a Softmax activation function), and the output is the probability distribution of all predefined gesture categories. The category with the highest probability is the final dynamic gesture result recognized by the system. .

[0208] Figure 5 This is a schematic diagram of a gesture recognition device provided in an embodiment of this application. The device can execute the gesture recognition method provided in any embodiment of this application, and possesses the corresponding functional modules and beneficial effects of the method. For example... Figure 5 As shown, the device includes:

[0209] Image acquisition module 410 is used to acquire images of user gestures through an image acquisition device to obtain a continuous video sequence;

[0210] The spatial flow feature extraction module 420 is used to extract gesture regions from each image frame in the continuous video sequence to obtain gesture images, and to extract features from the gesture images to obtain spatial flow features.

[0211] The temporal flow feature extraction module 430 is used to perform optical flow calculation on two consecutive frames of images in the continuous video sequence to obtain an optical flow image, and to extract features from the optical flow image to obtain temporal flow features.

[0212] The gesture recognition module 440 is used to fuse the spatial flow features and the temporal flow features to obtain fused features, and to perform gesture classification on the fused features based on a classifier to determine the gesture recognition result.

[0213] In this embodiment, the spatial flow feature extraction module 420 extracts gesture images from each image frame in the continuous video sequence, including:

[0214] For each image frame in the continuous video sequence, a binary mask image of hand skin color is generated based on the color features of the image frame;

[0215] For each image frame in the continuous video sequence, a hand depth binary mask image is generated based on the depth information in the image frame; wherein the depth of the hand in the image frame is different from the depth of the background; the values ​​assigned to the hand pixels in the hand depth binary mask image and the hand skin color binary mask image are the same;

[0216] A hand binary mask image is generated based on the hand skin color binary mask image and the hand depth binary mask image, and a gesture image is extracted from the corresponding image frame based on the hand binary mask image.

[0217] In this embodiment, the spatial flow feature extraction module 420 generates a binary mask image of hand skin color based on the color features of the image frame, including:

[0218] Convert the image frame from the RGB color space to the HSV color space;

[0219] For each pixel in the image frame, the value of the H channel is compared with a preset hue range, and the value of the S channel is compared with a preset saturation range to filter out skin color pixels whose color features belong to the skin color of the hand; wherein, the preset hue range and the preset saturation range are determined based on the clustering characteristics of the skin color values ​​in the H channel and S channel in the sample image;

[0220] By setting the skin color pixels to different values ​​from other pixels, a binary mask image of the hand's skin color is obtained.

[0221] In this embodiment of the application, the spatial flow feature extraction module 420 generates a hand binary mask image based on the hand skin color binary mask image and the hand depth binary mask image, including:

[0222] The intersection of the hand skin color binary mask image and the hand region of the hand depth binary mask image is used to obtain the candidate gesture region;

[0223] The candidate gesture region is processed by smoothing the boundaries, filling the internal holes, and removing noise to obtain the target gesture region;

[0224] The target gesture region is assigned a different value from other regions to obtain the binary mask image of the hand.

[0225] In this embodiment, the gesture recognition module 440 fuses the spatial flow features and the temporal flow features to obtain fused features, including:

[0226] The spatial flow features and the temporal flow features are processed based on the cross-attention mechanism to obtain the corresponding spatial features and temporal features respectively;

[0227] The spatial features and the temporal features are then fused using pixel-level weighted fusion to obtain the fused features.

[0228] In this embodiment, the gesture recognition module 440 processes the spatial flow features and the temporal flow features based on a cross-attention mechanism to obtain corresponding spatial features and temporal features, including:

[0229] A query matrix is ​​obtained by performing a linear transformation on the spatial flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the temporal flow features. A first attention output is then calculated based on the query matrix, the key matrix, and the value matrix.

[0230] The spatial features are determined based on the first attention output and the spatial flow features;

[0231] A query matrix is ​​obtained by performing a linear transformation on the temporal flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the spatial flow features. A second attention output is then calculated based on the query matrix, the key matrix, and the value matrix.

[0232] The time feature is determined based on the second attention output and the time flow feature.

[0233] In this embodiment, the gesture recognition module 440 performs pixel-level weighted fusion of the spatial features and the temporal features to obtain the fused features, including:

[0234] The spatial features and the temporal features are spliced ​​together to obtain spliced ​​features, and channel descriptors are determined by statistical calculation based on the spatial position of each channel of the spliced ​​features.

[0235] Generate a first weight matrix corresponding to the spatial features and a second weight matrix corresponding to the temporal features based on the channel descriptor;

[0236] The spatial features and the temporal features are weighted and fused based on the first weight matrix and the second weight matrix to obtain the fused features.

[0237] In this embodiment of the application, after fusing the spatial flow features and the temporal flow features to obtain fused features, the apparatus further includes: a dependency feature extraction module, used for...

[0238] The fused features are processed by a gated loop unit to extract temporal dependency features;

[0239] Accordingly, gesture classification is performed on the fused features based on the classifier to determine the gesture recognition result, including:

[0240] The time-dependent features are input into the classifier to classify gestures and determine the gesture recognition result.

[0241] In the embodiments of this application, during the training process of the feature extraction model for spatial flow features and the feature extraction model for temporal flow features, the model parameters of the first standard convolutional layer and the three adjacent inverted residual bottleneck blocks are shared, while other parameters are individually trained and optimized.

[0242] The gesture recognition device provided in this application embodiment can execute a gesture recognition method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0243] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0244] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0245] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0246] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as gesture recognition methods.

[0247] In some embodiments, the gesture recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the gesture recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the gesture recognition method by any other suitable means (e.g., by means of firmware).

[0248] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0249] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable gesture recognition device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0250] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0251] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0252] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0253] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0254] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the gesture recognition method provided in any embodiment of this application.

[0255] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0256] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0257] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A gesture recognition method, characterized in that, The method includes: A continuous video sequence is obtained by capturing images of the user's gestures using an image capture device; Gesture images are obtained by extracting gesture regions from each image frame in the continuous video sequence, and spatial flow features are obtained by extracting features from the gesture images. For two consecutive frames in the continuous video sequence, optical flow is calculated on the two consecutive frames to obtain an optical flow image, and feature extraction is performed on the optical flow image to obtain temporal flow features; The spatial flow features and the temporal flow features are fused to obtain fused features, and the gesture recognition result is determined by classifying the fused features based on a classifier.

2. The method according to claim 1, characterized in that, Gesture images are obtained by extracting gesture regions from each image frame in the continuous video sequence, including: For each image frame in the continuous video sequence, a binary mask image of hand skin color is generated based on the color features of the image frame; For each image frame in the continuous video sequence, a hand depth binary mask image is generated based on the depth information in the image frame; wherein the depth of the hand in the image frame is different from the depth of the background; the values ​​assigned to the hand pixels in the hand depth binary mask image and the hand skin color binary mask image are the same; A hand binary mask image is generated based on the hand skin color binary mask image and the hand depth binary mask image, and a gesture image is extracted from the corresponding image frame based on the hand binary mask image.

3. The method according to claim 2, characterized in that, Generate a binary mask image of hand skin color based on the color features of the image frame, including: Convert the image frame from the RGB color space to the HSV color space; For each pixel in the image frame, the value of the H channel is compared with a preset hue range, and the value of the S channel is compared with a preset saturation range to filter out skin color pixels whose color features belong to the skin color of the hand; wherein, the preset hue range and the preset saturation range are determined based on the clustering characteristics of the skin color values ​​in the H channel and S channel in the sample image; By setting the skin color pixels to different values ​​from other pixels, a binary mask image of the hand's skin color is obtained.

4. The method according to claim 2, characterized in that, Generating a hand binary mask image based on the hand skin color binary mask image and the hand depth binary mask image includes: The intersection of the hand skin color binary mask image and the hand region of the hand depth binary mask image is used to obtain the candidate gesture region; The candidate gesture region is processed by smoothing the boundaries, filling the internal holes, and removing noise to obtain the target gesture region; The target gesture region is assigned a different value from other regions to obtain the binary mask image of the hand.

5. The method according to claim 1, characterized in that, The spatial flow features and the temporal flow features are fused to obtain fused features, including: The spatial flow features and the temporal flow features are processed based on the cross-attention mechanism to obtain the corresponding spatial features and temporal features respectively; The spatial features and the temporal features are then fused using pixel-level weighted fusion to obtain the fused features.

6. The method according to claim 5, characterized in that, The spatial flow features and temporal flow features are processed based on a cross-attention mechanism to obtain corresponding spatial and temporal features, including: A query matrix is ​​obtained by performing a linear transformation on the spatial flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the temporal flow features. A first attention output is then calculated based on the query matrix, the key matrix, and the value matrix. The spatial features are determined based on the first attention output and the spatial flow features; A query matrix is ​​obtained by performing a linear transformation on the temporal flow features, and a key matrix and a value matrix are obtained by performing a linear transformation on the spatial flow features. A second attention output is then calculated based on the query matrix, the key matrix, and the value matrix. The time feature is determined based on the second attention output and the time flow feature.

7. The method according to claim 5, characterized in that, The spatial features and the temporal features are fused using pixel-level weighted fusion to obtain the fused features, including: The spatial features and the temporal features are spliced ​​together to obtain spliced ​​features, and channel descriptors are determined by statistical calculation based on the spatial position of each channel of the spliced ​​features. Generate a first weight matrix corresponding to the spatial features and a second weight matrix corresponding to the temporal features based on the channel descriptor; The spatial features and the temporal features are weighted and fused based on the first weight matrix and the second weight matrix to obtain the fused features.

8. The method according to claim 1, characterized in that, After fusing the spatial flow features and the temporal flow features to obtain the fused features, the method further includes: The fused features are processed by a gated loop unit to extract temporal dependency features; Accordingly, gesture classification is performed on the fused features based on the classifier to determine the gesture recognition result, including: The time-dependent features are input into the classifier to classify gestures and determine the gesture recognition result.

9. The method according to claim 1, characterized in that, During the training of the feature extraction models for spatial flow features and temporal flow features, the model parameters for the first standard convolutional layer and the three adjacent inverted residual bottleneck blocks are shared, while other parameters are individually trained and optimized.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the gesture recognition method according to any one of claims 1-9.

Images