Gesture recognition method and device based on pulse camera, medium and product

Abstract

The invention discloses a gesture recognition method and device based on a pulse camera, a medium and a product, relates to the technical field of gesture recognition, and discloses the gesture recognition method based on the pulse camera, and the method comprises the steps: taking pulse image data collected by the pulse camera as the input of a hand detection network, and obtaining dimension rising feature data based on an expansion convolution layer; expanding the dimension raising feature data in an inverse residual block of the hand detection network based on a target expansion coefficient to obtain an expanded deep convolutional feature, the target expansion coefficient being smaller than a default expansion coefficient of the hand detection network; obtaining a weighted feature map obtained after the expanded depth convolution feature passes through an embedded SE architecture; performing multi-dimensional collaborative hand region directional extraction processing based on the weighted feature map to obtain a hand region of the pulse image data; and identifying the gesture type of the hand region. The stability and accuracy of gesture recognition under a low-computing-power platform are improved.

Description

Technical Field

[0001] This application relates to the field of gesture recognition technology, and in particular to gesture recognition methods, devices, media and products based on pulse cameras. Background Technology

[0002] Gesture recognition typically involves training a model to learn and recognize relevant gestures. When it's necessary to expand the recognition to include new gesture types, sample data of the target gestures must be collected and the model retrained, increasing the model's complexity.

[0003] However, when a highly complex model is deployed to a low-computing-power platform, it needs to be compressed, such as by pruning and quantization, to adapt it, which reduces the accuracy of gesture feature extraction.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a gesture recognition method, device, medium, and product based on a pulse camera, aiming to solve the technical problem of low gesture extraction accuracy.

[0006] To achieve the above objectives, this application proposes a gesture recognition method based on a pulse camera, the method comprising: Pulse image data acquired by a pulse camera is used as input to a hand detection network, and up-dimensional feature data is obtained based on dilated convolutional layers; In the inverted residual block of the hand detection network, the up-dimensional feature data is expanded based on a target expansion coefficient to obtain expanded depthwise convolutional features, wherein the target expansion coefficient is smaller than the default expansion coefficient of the hand detection network; Obtain the weighted feature map obtained by passing the expanded depthwise convolutional features through an embedded SE architecture; Based on the weighted feature map, a multi-dimensional collaborative hand region directional extraction process is performed to obtain the hand region of the pulse image data; Identify the gesture type of the hand area.

[0007] In one embodiment, before the step of using pulse image data acquired by a pulse camera as input to a hand detection network and obtaining up-dimensional feature data based on dilated convolutional layers, the pulse camera-based gesture recognition method further includes: Obtain a pre-labeled pulse dataset and an RGB image dataset, wherein the number of samples in the RGB image dataset is greater than that in the pre-labeled pulse dataset; Based on the imaging principle of a pulse camera, the RGB image dataset is converted into pulse simulation data; Within a preset parameter range, the contrast and brightness of the pulse simulation data are asymmetrically adjusted to simulate the pulse accumulation effect under different lighting conditions, and pulse noise is injected simultaneously. The hand detection network is trained using the pre-labeled pulse dataset and the pulse simulation data.

[0008] In one embodiment, the step of obtaining the weighted feature map obtained after the dilated depthwise convolutional features are processed by an embedded SE architecture includes: Based on the adaptive pooling layer, the dimensional feature map of each channel in the expanded depth convolution feature is globally averaged to obtain the channel-level feature vector of each channel. The channel-level feature vectors are nonlinearly mapped using a fully connected layer to obtain channel attention weights. The channel attention weights and the dilated depth convolution features are multiplied channel by channel to obtain the weighted feature map.

[0009] In one embodiment, before the step of obtaining the weighted feature map obtained by the dilated depthwise convolutional features through an embedded SE architecture, the gesture recognition method based on a pulse camera further includes: After performing pointwise convolution, batch normalization, and activation processing on the expanded depthwise convolution features, depthwise convolution, batch normalization, and activation processing are performed to obtain preprocessed expanded depthwise convolution features. Following the step of obtaining the weighted feature map obtained by the extended depthwise convolutional features through an embedded SE architecture, the gesture recognition method based on a pulse camera further includes: The weighted feature map is subjected to depthwise convolution and batch normalization to obtain a preprocessed weighted feature map.

[0010] In one embodiment, the step of performing multi-dimensional collaborative hand region directional extraction processing based on the weighted feature map to obtain the hand region of the pulse image data includes: If the stride of the preprocessed weighted feature map satisfies the judgment condition of residual connection, the weighted feature map and the preprocessed dilated depth convolution feature map are subjected to residual connection processing to obtain a residual fused feature map. Based on the backbone network of the hand detection network, feature extraction is performed on the residual fusion feature map, and the extracted features are fused based on the neck network. The fused features are then input into the fully connected layer. Based on the fully connected layer, the fused features are flattened and transformed by full connectivity, dimensionality compressed and fitted to suppress, normalized and nonlinearly enhanced, and then dimensionality reduced and output to obtain the hand region of the pulse image data.

[0011] In one embodiment, the step of identifying the gesture type of the hand region includes: Obtain the joint point coordinate sequence of the hand joints in the hand region from the pulse image data; The gesture type corresponding to the key point coordinate sequence is determined based on preset detection rules.

[0012] In one embodiment, before the step of expanding the up-dimensional feature data in the inverted residual block of the hand detection network based on a target dilation coefficient to obtain expanded depthwise convolutional features, wherein the target dilation coefficient is less than the default dilation coefficient of the hand detection network, the gesture recognition method based on a pulse camera further includes: The amount of data computation required to acquire the pulse image data to be processed, and / or the network hierarchy of the hand detection network; The target expansion coefficient is determined based on the first expansion coefficient corresponding to the data computation volume and / or the second expansion coefficient corresponding to the network layer.

[0013] In addition, to achieve the above objectives, this application also proposes a gesture recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the gesture recognition method based on a pulse camera as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the gesture recognition method based on a pulse camera as described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the gesture recognition method based on a pulse camera as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: By inputting pulse image data acquired by a pulse camera into a hand detection network and passing it through dilated convolutional layers to obtain upscaled feature data, a target dilation coefficient smaller than the default dilation coefficient is used in the inverse residual block to process the upscaled feature data to obtain dilated depth convolutional features. Subsequently, the dilated depth convolutional features are weighted and enhanced using an embedded SE architecture, allowing the network to focus more on key hand-related features and suppress irrelevant background interference, thus achieving feature focusing. In this way, the computational complexity of dilated convolution is reduced, making it adaptable to low-computing platforms. At the same time, the embedded SE architecture uses a channel attention mechanism to globally weight the shrunken features, compensating for the defects of reduced receptive field and insufficient capture of overall hand contour / contextual information that may be caused by the reduction of the dilation coefficient, ensuring that the overall semantic features of the hand region are not lost. Finally, based on multi-dimensional collaborative hand region directional extraction processing, the hand region in the pulse image is accurately separated and gesture type recognition is completed. While avoiding the loss of feature extraction accuracy due to model compression, it achieves synergistic optimization between the deployment requirements of low computing power platforms and gesture recognition performance. It not only ensures the integrity and relevance of hand feature extraction, but also adapts to low computing power operating environments through a lightweight design throughout the entire process, significantly improving the stability and accuracy of gesture recognition on low computing power platforms. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the gesture recognition method based on a pulse camera according to this application. Figure 2 This is a flowchart illustrating the second embodiment of the gesture recognition method based on a pulse camera in this application. Figure 3 This is a schematic diagram of the non-end-to-end gesture recognition architecture based on a pulse camera, which is the gesture recognition method based on a pulse camera in this application. Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the gesture recognition method based on a pulse camera in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] The main solution of this application embodiment is: to use the pulse image data acquired by the pulse camera as the input of the hand detection network, and to obtain the up-dimensional feature data based on the dilated convolutional layer; In the inverted residual block of the hand detection network, the up-dimensional feature data is expanded based on a target expansion coefficient to obtain expanded depthwise convolutional features, wherein the target expansion coefficient is smaller than the default expansion coefficient of the hand detection network; Obtain the weighted feature map obtained by passing the expanded depthwise convolutional features through an embedded SE architecture; Based on the weighted feature map, a multi-dimensional collaborative hand region directional extraction process is performed to obtain the hand region of the pulse image data; Identify the gesture type of the hand area.

[0023] In this embodiment, for ease of description, the gesture recognition device will be used as the execution subject in the following description.

[0024] In current technologies, gesture recognition typically involves training a model to learn and recognize relevant gestures. When it's necessary to expand the recognition to include new gesture types, sample data of the target gestures must be collected and the model retrained, increasing the model's complexity.

[0025] However, when a highly complex model is deployed to a low-computing-power platform, it needs to be compressed, such as by pruning and quantization, to adapt it, which reduces the accuracy of gesture feature extraction.

[0026] This application provides a solution that inputs pulse image data acquired by a pulse camera into a hand detection network and passes it through a dilated convolutional layer to obtain upscaled feature data. In the inverted residual block, a target dilation coefficient smaller than the default dilation coefficient is used to process the upscaled feature data to obtain dilated depth convolutional features. Subsequently, an embedded SE architecture is used to weight and enhance the dilated depth convolutional features, allowing the network to focus more on key hand-related features and suppress irrelevant background interference, thus achieving feature focusing. In this way, the computational complexity of dilated convolution is reduced, making it adaptable to low-computing platforms. Simultaneously, the embedded SE architecture uses a channel attention mechanism to globally weight the shrunken features, compensating for the potential reduction in receptive field and insufficient capture of overall hand contour / contextual information caused by the reduced dilation coefficient, ensuring that the overall semantic features of the hand region are not lost.

[0027] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or gesture recognition device capable of performing the above functions. The following description uses a gesture recognition device as an example to illustrate this embodiment and the subsequent embodiments.

[0028] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0029] This application provides a gesture recognition method based on a pulse camera, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the gesture recognition method based on a pulse camera according to this application.

[0030] In this embodiment, the gesture recognition method based on a pulse camera includes steps S10 to S50: Step S10: The pulse image data acquired by the pulse camera is used as the input to the hand detection network, and the upgraded feature data is obtained based on the dilated convolutional layer.

[0031] The aforementioned dilated convolutional layer is a convolutional structure that expands the receptive field by introducing holes between convolutional kernel elements. Up-dimensional feature data refers to feature data with increased channel count and enhanced semantic information after processing by this layer. The hand detection network is LiteInvSE-Net (Lite Inverted-Residual SE Network).

[0032] In this embodiment, after the pulse image data acquired by the pulse camera is input into the hand detection network, the pulse data is processed by the backbone network and then enters the initial convolution module. First, a two-dimensional convolution operation is performed through Conv2d (Convolution2d) to convert one input channel of the pulse data into 32 feature channels. The kernel size is preferably 3×3 and the stride of the convolution operation is 2, thereby completing the extraction of pulse features and channel expansion. Then, two-dimensional batch normalization is performed on the convolved pulse features through BatchNorm2d. After Hardswish (an activation function) processing, it enters the subsequent improved inverse residual block processing stage, thereby providing high-dimensional, low-noise initial features for the inverse residual block to adapt to the subsequent dynamic configuration processing requirements.

[0033] Step S20: Expand the dimensionality-upgraded feature data in the inverted residual block of the hand detection network based on the target dilation coefficient to obtain the expanded depth convolution feature.

[0034] The target dilation coefficient mentioned above is less than the default dilation coefficient of the hand detection network. The dilation coefficient is the dilation rate of the dilated convolution, used to control the receptive field size. The default dilation coefficient refers to the dilation coefficient used when training a typical hand detection network, usually 6. The target dilation coefficient is usually a pre-set parameter.

[0035] In this embodiment, to adapt to low-computing-power platforms and reduce computational load, the target dilation coefficient is set to less than 6, such as 2, 3, and 4. Specifically, a 3×3 dilated depth convolutional layer with a target dilation coefficient r=3 is used, with stride=1 and padding=3 (calculated according to the dilated convolution size preservation formula p=r×(k-1) / 2, where k is the convolution kernel size). Then, a depth convolution operation is performed on the increased-dimensional features. During this process, the dilation coefficient r=3 expands the effective receptive field of the convolution kernel to 7×7, which can accurately cover the hand region that occupies 15%-30% of the image. Finally, the dilated depth convolution features are output through the forward computation of this dilated depth convolutional layer.

[0036] Understandably, in the dilated convolutional layer configuration, using a 3×3 convolutional kernel, with dilation coefficients r=6, stride=1, and padding=6 (calculated as p=r×(k-1) / 2 to ensure the feature map size is fully preserved), the network's cumulative receptive field can cover the entire pulse image (320×240), and the computational cost of this layer is approximately 2.4 GFLOPs. However, in actual network deployment, by adjusting the dilation coefficients to r=3, stride=1, and padding=3 (while still preserving the size), and through the stride stacking of the preceding layers, the cumulative receptive field can still cover 15%-30% of the hand region of the image, reducing the computational cost to 1.2 GFLOPs.

[0037] Therefore, by reducing the expansion coefficient of the expansion layer, the number of parameters and computational cost of the model are significantly reduced.

[0038] Optionally, the target dilation coefficient can be calculated based on parameters such as the actual data computation volume and network layer. Therefore, before dilution of the dimensionality-increasing feature data, the current computation volume can be analyzed first, and then the corresponding dilation coefficient can be automatically selected. For example, the data computation volume of pulse image data can be obtained first, and then the dilation coefficient corresponding to this computation volume can be determined by looking up a table or numerical calculation. This dilation coefficient is the target dilation coefficient. In addition, the network layer of the hand detection network can be determined, and the target dilation coefficient can be determined by the network layer. For example, the basic constraints can be determined by the feature map size of the network layer, and then an appropriate target dilation coefficient can be selected based on the feature map size. Furthermore, the target dilation coefficient can be obtained by weighted summation based on the first dilation coefficient corresponding to the data computation volume and the second dilation coefficient corresponding to the network layer.

[0039] Step S30: Obtain the weighted feature map obtained by applying the expanded depthwise convolutional features to the embedded SE architecture.

[0040] In this embodiment, the embedded SE architecture is a channel attention module adapted to low-computing-power devices, preferably a Squeeze-and-Excitation attention module. The weighted feature map is a feature map obtained by assigning differentiated weights to each channel through the SE architecture, amplifying the feature responses of effective channels and suppressing redundant channels.

[0041] It should be noted that the output features of dilated depthwise convolution have drawbacks, including a lack of interaction between channels, poor adaptability of the number of channels to the SE module, and insufficient spatial features of hand details. Therefore, before inputting the dilated depthwise convolution features into the embedded SE architecture, pointwise convolution (PW-Conv2d) processing is required to achieve inter-channel information fusion, integrate scattered hand features, and enhance feature discriminativeness by combining two-dimensional batch normalization (BatchNorm2d) and the Hardswish activation function. Then, depthwise convolution (DWConv2d) processing is used to refine spatial details such as finger gaps and edges, improving the feature quality input to the SE module. This allows the SE's channel attention mechanism to more accurately select hand-related channels, thereby better leveraging the attention weighting effect.

[0042] Specifically, after obtaining the preprocessed dilated depthwise convolutional features, the input dilated depthwise convolutional features can be fed into an adaptive average pooling layer to compress the two-dimensional spatial features of each channel into 1×1 global features to capture global channel information. Then, after dimensionality reduction and expansion through two fully connected layers, channel attention weights matching the original number of feature channels are learned. Finally, these weights are multiplied back onto the original input features channel by channel to enhance effective feature channels and suppress redundant channels, ultimately obtaining a weighted feature map.

[0043] Step S40: Perform multi-dimensional collaborative hand region directional extraction processing based on weighted feature maps to obtain the hand region of the pulse image data.

[0044] In this embodiment, the multi-dimensional collaborative hand region directional extraction processing refers to a region extraction method based on the fusion of channel, spatial and semantic multi-dimensional features. It preserves feature integrity through residual connections and achieves localization by combining fully connected layers. Directional extraction means focusing only on the hand region rather than global features.

[0045] After obtaining the weighted adjustment map, the SE module only performs attention weighting on the feature channels, i.e., enhancing effective channels and suppressing redundant channels, but does not adjust the number of channels or optimize the feature distribution. Therefore, it is necessary to perform depthwise convolution and batch normalization on the weighted feature map. Through depthwise convolution PW-Conv2d, the number of channels in the weighted feature map can be flexibly adjusted. For example, the 128 channels of hand features after SE weighting can be adapted to the 32 channels required by subsequent network layers. At the same time, feature information between different weighted channels, such as the feature interaction between the hand contour channel and the knuckle channel, is fused to make up for the shortcomings of SE which only weights without fusion. Then, batch normalization BatchNorm2d is used to normalize the numerical distribution of the features after PW convolution, solving the problems of high effective channel values ​​and fluctuating feature distribution after SE weighting, avoiding gradient explosion or vanishing, thereby accelerating network convergence, improving the generalization of hand detection features, and ensuring that the final features retain the channel attention advantage of SE while also having the channel dimension and stable feature distribution to adapt to subsequent network layers.

[0046] Therefore, after obtaining the preprocessed weighted feature map, a multi-dimensional collaborative hand region-oriented extraction process is performed. Specifically, in this process, it is usually first determined whether the stride of the preprocessed weighted feature map meets the judgment condition of residual connection. If the condition is met, the weighted feature map and the preprocessed dilated depth convolutional features need to be subjected to residual connection processing to obtain a residual fused feature map. Subsequently, feature extraction is performed on the residual fused feature map based on the backbone network of the hand detection network, and the extracted features are fused based on the neck network. The fused feature data is then input into a fully connected layer. In the fully connected layer, the fused features are subjected to feature flattening and fully connected transformation, dimensionality compression and fitting suppression, normalization and nonlinear enhancement, dimensionality reduction and output processing to obtain the hand region of the pulse image data.

[0047] Without residual connections, the preprocessed weighted feature map will be completely covered by subsequent convolutional layers, which can easily lead to gradient vanishing in deep networks and cause the loss of key features of small hand targets. However, with residual connections, the preprocessed weighted feature map can be added element by element to the output features of subsequent convolutions, which not only preserves the effective information of the original weighted features but also adds the newly extracted semantic features. At the same time, the gradient can be directly backpropagated along the residual edge, avoiding gradient vanishing.

[0048] For example, when stride=1 and the number of feature channels matches, the input weighted feature map is residually added to the previously preprocessed dilated depthwise convolutional features, preserving the core information of the original weighted features; if the conditions are not met, the process proceeds directly to the next step. After residual processing, the residual-processed features are input into the backbone network. Based on this backbone network, high-level semantic features of the hand are extracted, and a 160-channel feature map is output. This 160-channel feature map is then fed into the neck network, which uses feature fusion / dimensionality reduction to convert it into a 64×5×4 feature map (64 channels, 8×4 spatial size), thus compressing computation while preserving key semantic information of the hand. The 64×5×4 feature map is then fed into a fully connected layer. In this layer, the 64×5×4 feature map is first flattened into a 1280-dimensional one-dimensional vector using a Flatten operation. This flattened 1280-dimensional vector is then reduced to 64 dimensions using a Linear (1280→64) layer. Next, Dropout (0.2) is used to randomly discard 20% of the neurons to prevent overfitting and improve the model's generalization ability across different hand poses. In the feature normalization and nonlinear enhancement stages, the 64-dimensional features are batch normalized using BatchNorm1d, and then nonlinearity is introduced through the ReLU activation function to further refine hand-related features. Finally, the ReLU-processed features are reduced to 8 dimensions through a Linear (64→8) layer, and then fed into the output branch for splitting and output: the first 2 dimensions are activated by Sigmoid and output exist, which is the probability of the hand being present or not present; the middle 4 dimensions output center_x and center_y, representing the x-coordinate and y-coordinate of the hand center; and the last 2 dimensions output size, representing the width and height of the hand, thus finally determining the hand region.

[0049] Step S50: Identify the gesture type of the hand area.

[0050] In this embodiment, when a hand region is present, a judgment is made through hand joint detection and based on hand joint detection rules. Specifically, the hand keypoint detection module can accurately detect the two-dimensional keypoints of the hand corresponding to the hand region from the pulse image data, obtaining a joint point coordinate sequence, that is, converting the difficult-to-understand pulse stream into a structured, semantically rich joint point coordinate sequence. Subsequently, based on preset detection rules, the gesture type corresponding to the joint point coordinate sequence is determined. Using the extracted spatial relative position and angle of the joint points, the gesture type is determined through predefined, interpretable logical rules such as finger bending state and relative distance between fingertips, thereby obtaining the gesture type recognition result.

[0051] For example, the experimental results data corresponding to this embodiment are shown in the table below:

[0052] As can be seen, in the original DetNet network (Detection Network, a convolutional neural network designed specifically for object detection tasks) data (number 0), when processed with the default dilation factor of 6, the corresponding parameter count is 1.51M and the accuracy is 0.767. In the improved scheme of this embodiment (number 1), the target dilation factor is adjusted to 3 (less than the default value of 6), reducing the parameter count to 1.32M, thus reducing the data processing volume and adapting to the computing power constraints of low-performance platforms. Based on this, this embodiment introduces an embedded SE architecture. This architecture uses adaptive pooling to aggregate global channel information and fully connected layers to generate attention weights in the 0-1 range to weight and enhance the dilated deep convolutional features. This allows for targeted amplification of key feature channel responses related to the hand, while suppressing redundant background channels, effectively compensating for the shortcomings of insufficient capture of hand contours and contextual information that may occur with low dilation factors. The final experimental results show that, under the premise of reducing the amount of data processing, the accuracy of the parameters in this embodiment has been improved from 0.767 in the original DetNet to 0.845, which fully demonstrates that this design not only meets the operating requirements of low computing power platforms, but also improves the stability and accuracy of gesture recognition.

[0053] This embodiment provides a gesture recognition method based on a pulse camera. After preprocessing and dimensionality upscaling the pulse image data, coefficient expansion is performed based on a target expansion coefficient smaller than the default expansion coefficient, reducing the amount of data processing. Furthermore, an embedded SE architecture is used to weight and enhance the expanded deep convolutional features, allowing the network to focus more on key hand-related features and reduce interference from irrelevant features. This reduces computational complexity while compensating for insufficient capture of hand contours or contextual information, significantly improving the stability and accuracy of gesture recognition on low-computing-power platforms. Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S10, the gesture recognition method based on a pulse camera further includes steps S01 to S04: Step S01: Obtain the pre-labeled pulse dataset and RGB image dataset.

[0054] In this embodiment, the pre-labeled pulse dataset serves as the true benchmark for model training, directly reflecting the abrupt changes in light intensity during hand movements, such as pulse clusters generated by fingertip movement. Models on low-computing-power platforms cannot rely on online learning / adaptive adjustment after deployment; they need strong generalization capabilities from the training phase. The RGB dataset offers advantages such as easy accessibility, comprehensive scene coverage, and a significantly larger sample size than real pulse data, providing cost-effective ways to supplement training data diversity. Furthermore, converting it to simulated pulse data avoids the model learning high-dimensional RGB features while leveraging the larger sample size of RGB data to improve generalization. This allows the final trained model to maintain stable accuracy on low-computing-power platforms when facing real-world lighting and pose variations without additional computation, preventing frequent retraining due to poor generalization. Therefore, in the acquired dataset, the RGB image dataset has a larger sample size than the pre-labeled pulse dataset.

[0055] Step S02: Based on the imaging principle of the pulse camera, the RGB image dataset is converted into pulse simulation data.

[0056] In this embodiment, the imaging principle of the pulse camera is to record pixel-level light intensity changes / temporal contrast, rather than the spatial brightness distribution of the RGB image. By simulating the imaging principle of the pulse camera, the RGB image is efficiently converted into realistic pulse simulation data as a pre-training basis, allowing the RGB data to possess the core characteristics of pulse data. Compared with training directly using RGB data, the converted pulse simulation data retains the sample diversity of the RGB dataset and conforms to the output logic of the pulse camera, significantly expanding the scale of pulse training data and solving the problem of poor model generalization ability caused by insufficient real pulse data. The conversion process of the pulse simulation data is existing technology and will not be described in detail in this application.

[0057] Step S03: Within the preset parameter range, the contrast and brightness of the pulse simulation data are asymmetrically adjusted to simulate the pulse accumulation effect under different lighting conditions, and pulse noise is injected simultaneously. In this embodiment, the pulse accumulation effect of the pulse camera exhibits asymmetric characteristics under different lighting conditions. For example, the pulse in the hand area is saturated under strong light, the pulse is sparse under weak light, and the pulse contrast between the hand and the background is reversed under backlight. Asymmetric adjustment can accurately simulate these real-world scenarios. Therefore, the contrast and brightness of the pulse simulation data are asymmetrically adjusted within a preset range to simulate the differences in pulse accumulation effects under different lighting conditions. The preset parameter range can be set during the training phase based on actual needs.

[0058] Furthermore, the pulse data output by a real pulse camera contains inherent noise. By injecting pulse noise that conforms to the hardware characteristics, such as injecting white noise events in the spatiotemporal domain, the training data can be made closer to the real hardware output, thus avoiding model overfitting.

[0059] Step S04: Train the hand detection network using the pre-labeled pulse dataset and pulse simulation data.

[0060] After obtaining the asymmetrically enhanced data, the hand detection network is trained directly based on this data to obtain a hand detection network that meets the requirements.

[0061] It is understandable that the training process of the mobile phone detection network in this embodiment is not just a simple training of model parameters, but also a targeted design deeply tied to the subsequent scaling down of the expansion coefficient (reducing it from the default value of 6 to 3 or other parameters) to adapt to low-computing platforms. The two work together to build a reverse mechanism of "feature capacity shrinkage + data diversity enhancement". That is, asymmetric data augmentation enhances data diversity by simulating real-world scene variations such as target rotation, partial occlusion, and non-uniform lighting changes, thus building a multi-variable feature space for model training. The subsequent scaling down of the expansion coefficient compresses the model's feature capacity to reduce computation. Under this contradictory constraint, the model cannot rely on redundant features to memorize specific samples, but is forced to learn the core essential features of the hand that are not affected by scene variations. Ultimately, this achieves a synergistic optimization between the deployment requirements of low-computing platforms and the accuracy of gesture recognition, effectively solving the problem of decreased feature extraction accuracy caused by simple model compression, i.e., reduction of the expansion coefficient.

[0062] For example, to help understand the implementation flow of the pulse camera-based gesture recognition method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 3 , Figure 3 A schematic diagram of a non-end-to-end gesture recognition architecture based on a pulse camera is provided for a gesture recognition method based on a pulse camera. Specifically: During the training phase, a small amount of labeled pulse data and a large amount of transformed public datasets are used for training. The transformed public datasets undergo asymmetric data augmentation based on the characteristics of pulse data, resulting in the LiteInvSE-Net hand detection module, i.e., the hand detection network. Simultaneously, a keypoint detection module and a keypoint-based logical judgment module are combined and converted into C++ code, enabling it to run as an executable program on low-computing-power platforms. During program execution, real-time data is acquired through a pulse camera, and the executable program performs real-time gesture recognition processing based on these three modules.

[0063] Furthermore, the experimental results data corresponding to this embodiment are shown in the table below:

[0064] Based on experimental parameters, under the high dilation coefficient 6 + data augmentation + SE attention mechanism (number 2), the hand detection accuracy is 0.781 and the parameter count is 1.75M. Under the low dilation coefficient 3 + no data augmentation + SE attention mechanism (number 1), even without augmenting the training data, by reducing the dilation coefficient from the default high value of 6 to 3, the model parameter count is reduced by 24.6% (from 1.75M to 1.32M), and the computational complexity is significantly reduced, while the hand detection accuracy is improved to 0.845, an improvement of approximately 8.2% compared to scheme number 1. In this embodiment, the low dilation coefficient 3 + data augmentation + SE attention mechanism (number 3) further maintains this low dilation coefficient configuration, achieving a detection accuracy of 0.894 while keeping the parameter count at 1.32M, an improvement of approximately 14.5% compared to scheme number 1.

[0065] Understandably, this embodiment achieves a reverse optimization of parameter simplification and detection accuracy improvement through the collaborative design of low expansion coefficient configuration, targeted data augmentation, and SE attention mechanism. It effectively reduces the computational complexity of the model by reducing the expansion coefficient and the number of parameters to meet the deployment requirements of low computing power platforms, and breaks through the conventional limitation that model compression will inevitably lead to a decrease in accuracy. The traditional solution of higher expansion coefficient + data augmentation significantly improves the accuracy of hand detection, and solves the core technical pain point of difficulty in balancing model compression and feature extraction accuracy in low computing power platform deployment.

[0066] This embodiment discloses a gesture recognition method based on a pulse camera. By precisely shrinking the feature capacity and directionally strengthening the core features, it reduces the computational complexity of the model to adapt to low-computing-power platforms, while achieving reverse optimization of parameter simplification and detection accuracy improvement. This effectively solves the technical pain point of difficulty in balancing model compression and feature extraction accuracy in low-computing-power platform deployment.

[0067] Furthermore, based on any of the above embodiments, in the third embodiment of this application, the specific process of obtaining the weighted feature map obtained after the expanded depthwise convolutional features are processed by the embedded SE architecture is as follows: Specifically, after inputting the dilatational depth features into the embedded SE architecture, the SE module first performs global averaging on the dimensional feature maps of each channel in the dilatational depth convolution based on an adaptive pooling layer, obtaining a channel-level feature vector for each channel. Assuming the input is a feature map of (B, C, H, W), each channel corresponds to a two-dimensional feature matrix, where B is the batch size, C is the number of channels, H is the height, and W is the width. A global averaging operation is performed on the (H, W) two-dimensional feature matrix for each channel, compressing it into (1, 1) feature values, resulting in a global feature description tensor of shape (B, C, 1, 1). Subsequently, the global features are passed to an activation branch consisting of two fully connected layers for non-linear mapping to obtain channel attention weights. Finally, the channel attention weights are multiplied by the preprocessed dilatational depth convolution features to obtain a weighted feature map. This weighted feature map has the same spatial dimensions as the dilatational depth convolution features and the channel dimensions. Among them, the pooled (B, C) features directly reflect the global information of each channel, but there is no interaction between channels. The FC layer uses a weight matrix to make the channel features related to each other and learn the dependencies between channels, such as "when channel A is important, channel B should also be strengthened". At the same time, with the activation function, the FC layer can learn complex channel weight allocation rules, rather than simple linear weighting, making the attention mechanism more in line with task requirements.

[0068] The fully connected layer includes dimensionality reduction and dimensionality expansion layers. After obtaining the channel-level feature vectors, the pooled (B, C, 1, 1) tensor is flattened into (B, C) and input into the fully connected layer (FC). In the FC layer, the number of channels is first compressed from C to C / r (where r is the compression coefficient) through a dimensionality reduction FC layer. Non-linearity is added based on an activation function such as ReLU. Then, the dimensionality expansion FC layer restores the number of channels to C. An activation function such as Sigmoid maps the output value to between 0 and 1, obtaining the attention weight (B, C) for each channel. Finally, this weight is multiplied back into the preprocessed dilated depthwise convolutional features channel by channel, adaptively enhancing the feature strength of channels containing effective hand information while suppressing the features of redundant background channels. The final output is the enhanced feature to improve the accuracy of hand detection.

[0069] For example, the adaptive pooling layer is a custom `AdaptiveAvgPool2dCustom`, which compresses the 64-channel × 346 × 260 features into a 64-channel × 1 × 1 feature vector, preserving the core hand features. Then, the pooled (1, 64, 1, 1) tensor is flattened into a (1, 64) one-dimensional vector and input into the fully connected layer for dimensionality transformation. A dimensionality reduction FC layer compresses the number of channels from 64 to 8 (compression coefficient r = 8), and a ReLU activation function is used to introduce a non-linear feature transformation, enhancing the feature discrimination of the effective hand channels. Subsequently, a dimensionality increase FC layer restores the number of channels from 8 to 64, and a Sigmoid activation function maps the output values ​​to the 0~1 range, resulting in (1, 64) channel attention weights. For example, the weights for core hand channels such as fingertips and palms are > 0.7, while the weights for background redundant channels are < 0.3. Finally, the calculated attention weights are multiplied channel by channel with the original dilated depth convolution features to amplify the weights of important channels and suppress the weights of secondary channels, outputting a weighted feature map.

[0070] Optionally, the native PyTorch layer `nn.AdaptiveAvgPool2d` can be called, setting its `output_size` parameter to (1, 1) and performing adaptive average pooling on the preprocessed features. This layer adaptively matches the 320×240 feature map size, compressing the two-dimensional spatial features (320×240) of each channel into a 1×1 global feature vector without manually calculating the pooling kernel size, thus fully capturing the global information of the hand region corresponding to each channel. These 128 1×1 global feature vectors are then flattened and fed into an activation branch consisting of two `nn.Linear` fully connected layers. The first fully connected layer reduces the channel dimension from 128 to 32 and introduces non-linearity using the ReLU activation function. The second fully connected layer restores the dimension to 128, learning and generating channel attention weights that perfectly match the original number of feature channels. Finally, these weights are multiplied back into the preprocessed dilated depthwise convolutional features channel by channel, adaptively enhancing the feature strength of channels containing effective hand information (such as knuckles and palm contours).

[0071] It is understandable that the adaptive pooling layer in this embodiment is preferably a custom `AdaptiveAvgPool2dCustom`. PyTorch's native `nn.AdaptiveAvgPool2d` relies on automatic derivation of a dynamic computation graph. When exporting to ONNX, the implicit computation of "dynamic step size / kernel size" is recognized as an unsupported operation by the ONNX parser, leading to export failure or dimensionality errors during inference. In contrast, the custom `AdaptiveAvgPool2dCustom` transforms the implicit derivation of the native adaptive pooling into explicit numerical computation. ONNX can clearly recognize these fixed numerical operations, avoiding parsing errors. It also enforces a unified `output_size` format, converting `output_size` of int / non-int types to tuples, preventing ONNX export anomalies due to inconsistent input formats. Furthermore, it reuses `F.avg_pool2d` at the underlying level, ensuring that the exported model can perform normal inference in frameworks such as ONNX Runtime and TensorRT.

[0072] This application provides a gesture recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the gesture recognition method based on a pulse camera in the first embodiment described above.

[0073] The following is for reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the gesture recognition device in the embodiments of this application. The gesture recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The gesture recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0074] like Figure 4As shown, the gesture recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the gesture recognition device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the gesture recognition device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show gesture recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0075] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0076] The gesture recognition device provided in this application employs the pulse camera-based gesture recognition method described in the above embodiments, which can solve the technical problem of low gesture extraction accuracy. Compared with the prior art, the beneficial effects of the gesture recognition device provided in this application are the same as those of the pulse camera-based gesture recognition method provided in the above embodiments, and other technical features of this gesture recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0077] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0079] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the pulse camera-based gesture recognition method in the above embodiments.

[0080] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0081] The aforementioned computer-readable storage medium may be included in the gesture recognition device; or it may exist independently and not be assembled into the gesture recognition device.

[0082] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the gesture recognition device, cause the gesture recognition device to: Pulse image data acquired by a pulse camera is used as input to a hand detection network, and up-dimensional feature data is obtained based on dilated convolutional layers; In the inverted residual block of the hand detection network, the up-dimensional feature data is expanded based on a target expansion coefficient to obtain expanded depthwise convolutional features, wherein the target expansion coefficient is smaller than the default expansion coefficient of the hand detection network; Obtain the weighted feature map obtained by passing the expanded depthwise convolutional features through an embedded SE architecture; Based on the weighted feature map, a multi-dimensional collaborative hand region directional extraction process is performed to obtain the hand region of the pulse image data; Identify the gesture type of the hand area.

[0083] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0086] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described gesture recognition method based on a pulse camera, thereby solving the technical problem of low gesture extraction accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the gesture recognition method based on a pulse camera provided in the above embodiments, and will not be repeated here.

[0087] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A gesture recognition method based on a pulse camera, characterized in that, The gesture recognition method based on a pulse camera includes: Pulse image data acquired by a pulse camera is used as input to a hand detection network, and up-dimensional feature data is obtained based on dilated convolutional layers; In the inverted residual block of the hand detection network, the up-dimensional feature data is expanded based on a target expansion coefficient to obtain expanded depthwise convolutional features, wherein the target expansion coefficient is smaller than the default expansion coefficient of the hand detection network; Obtain the weighted feature map obtained by passing the expanded depthwise convolutional features through an embedded SE architecture; Based on the weighted feature map, a multi-dimensional collaborative hand region directional extraction process is performed to obtain the hand region of the pulse image data; Identify the gesture type of the hand area.

2. The gesture recognition method based on a pulse camera as described in claim 1, characterized in that, Before the step of using pulse image data acquired by a pulse camera as input to a hand detection network and obtaining up-dimensional feature data based on dilated convolutional layers, the pulse camera-based gesture recognition method further includes: Obtain a pre-labeled pulse dataset and an RGB image dataset, wherein the number of samples in the RGB image dataset is greater than that in the pre-labeled pulse dataset; Based on the imaging principle of a pulse camera, the RGB image dataset is converted into pulse simulation data; Within a preset parameter range, the contrast and brightness of the pulse simulation data are asymmetrically adjusted to simulate the pulse accumulation effect under different lighting conditions, and pulse noise is injected simultaneously. The hand detection network is trained using the pre-labeled pulse dataset and the pulse simulation data.

3. The gesture recognition method based on a pulse camera as described in claim 1, characterized in that, The step of obtaining the weighted feature map obtained by processing the expanded depthwise convolutional features through an embedded SE architecture includes: Based on the adaptive pooling layer, the dimensional feature map of each channel in the expanded depth convolution feature is globally averaged to obtain the channel-level feature vector of each channel. The channel-level feature vectors are nonlinearly mapped using a fully connected layer to obtain channel attention weights. The channel attention weights and the dilated depth convolution features are multiplied channel by channel to obtain the weighted feature map.

4. The gesture recognition method based on a pulse camera as described in claim 1, characterized in that, Before the step of obtaining the weighted feature map obtained by the extended depthwise convolutional features through the embedded SE architecture, the gesture recognition method based on the pulse camera further includes: After performing pointwise convolution, batch normalization, and activation processing on the expanded depthwise convolution features, depthwise convolution, batch normalization, and activation processing are performed to obtain preprocessed expanded depthwise convolution features. Following the step of obtaining the weighted feature map obtained by the extended depthwise convolutional features through an embedded SE architecture, the gesture recognition method based on a pulse camera further includes: The weighted feature map is subjected to depthwise convolution and batch normalization to obtain a preprocessed weighted feature map.

5. The gesture recognition method based on a pulse camera as described in claim 4, characterized in that, The step of performing multi-dimensional collaborative hand region directional extraction based on the weighted feature map to obtain the hand region of the pulse image data includes: If the stride of the preprocessed weighted feature map satisfies the judgment condition of residual connection, the weighted feature map and the preprocessed dilated depth convolution feature map are subjected to residual connection processing to obtain a residual fused feature map. Based on the backbone network of the hand detection network, feature extraction is performed on the residual fusion feature map, and the extracted features are fused based on the neck network. The fused features are then input into the fully connected layer. Based on the fully connected layer, the fused features are flattened and transformed by full connectivity, dimensionality compressed and fitted to suppress, normalized and nonlinearly enhanced, and then dimensionality reduced and output to obtain the hand region of the pulse image data.

6. The gesture recognition method based on a pulse camera as described in claim 1, characterized in that, The step of identifying the gesture type of the hand region includes: Obtain the joint point coordinate sequence of the hand joints in the hand region from the pulse image data; The gesture type corresponding to the key point coordinate sequence is determined based on preset detection rules.

7. The gesture recognition method based on a pulse camera as described in claim 1, characterized in that, Before the step of expanding the up-dimensional feature data in the inverted residual block of the hand detection network based on a target dilation coefficient to obtain expanded depthwise convolutional features, wherein the target dilation coefficient is less than the default dilation coefficient of the hand detection network, the gesture recognition method based on the pulse camera further includes: The amount of data computation required to acquire the pulse image data to be processed, and / or the network hierarchy of the hand detection network; The target expansion coefficient is determined based on the first expansion coefficient corresponding to the data computation volume and / or the second expansion coefficient corresponding to the network layer.

8. A gesture recognition device, characterized in that, The gesture recognition device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the gesture recognition method based on a pulse camera as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the gesture recognition method based on a pulse camera as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the gesture recognition method based on a pulse camera as described in any one of claims 1 to 7.

Images