Bimodal three-dimensional gesture attitude estimation method and system based on multi-frequency attention

Abstract

The invention discloses a bimodal three-dimensional gesture attitude estimation method and system based on multi-frequency attention, and the method comprises the steps: obtaining RGB data and event data corresponding to a to-be-recognized three-dimensional gesture, and carrying out the preprocessing of the RGB data and the event data, so as to obtain time-space aligned bimodal data; and inputting the bimodal data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture so as to estimate a three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized. According to the method and the system, complementarity of the RGB data and the event data can be fully utilized, high-frequency details are reserved, high estimation precision and robustness are still achieved under the extreme illumination condition, and reconstruction precision and efficiency of gesture attitude estimation under the extreme illumination condition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a dual-modal three-dimensional gesture pose estimation method and system based on multi-frequency attention. BACKGROUND

[0002] Three-dimensional gesture pose estimation aims to predict hand key point positions and palm poses from images or videos, and provides a basis for hand model reconstruction, gesture recognition and other tasks, and is a core technology for realizing natural interaction in AR / VR and other fields. With the development of deep learning, convolutional neural networks (CNNs), recurrent neural networks (RNNs) and the like are widely used in this field, forming multiple technical routes based on RGB cameras, event cameras and dual-modal fusion. However, the existing technology has significant limitations:

[0003] The method based on the RGB camera is constrained by the synchronous imaging mechanism, and is easy to lose hand movement information under extreme illumination conditions such as overexposure and strong light, resulting in pose estimation deviation;

[0004] The method based on the event camera is difficult to capture details such as finger edges due to sparse foreground and background noise interference of event data, affecting accurate segmentation of the hand region;

[0005] The existing dual-modal fusion method has obvious defects:

[0006] Patent No. CN115661941B (hereinafter referred to as D1), D1 discloses a gesture recognition method and electronic device, D1 acquires a video stream collected by a camera and an event stream collected by an event camera; the first video stream and the first event stream with the same collection time are fused to determine the gesture recognition result when the user's hand is facing the camera, the first video stream is part of the video stream collected by the camera, and the first event stream is part of the event stream collected by the event camera. D1 only performs time alignment and ignores spatial alignment, resulting in insufficient fusion accuracy;

[0007] Patent No. CN118505742A (hereinafter referred to as D2), D2 discloses a first-view hand tracking system and application based on an event camera and an RGB camera, D2 detects a hand bounding box in a high-resolution image by using a target detection algorithm, generates 3D coordinates of key points by using a feature point extraction and pose regression network, optimizes the key point positions according to event stream information, and generates a sparse trajectory of the key points; a time series filtering algorithm is applied to smooth and splice the trajectory, and high-frequency real-time hand key point position information is generated. D2 relies on manual adjustment of camera parameters to realize spatial alignment, which is poor in convenience; and most methods use standard attention mechanisms, only paying attention to the spatial correlation between modalities, losing high-frequency information carrying local details, and the performance drops sharply in extreme environments.

[0008] Therefore, how to fully utilize the complementarity of RGB data and event data, realize efficient spatio-temporal alignment and retain high-frequency details, becomes a key problem to improve the accuracy and robustness of three-dimensional gesture posture estimation. SUMMARY

[0009] To solve the technical problems in the background art, the application proposes a dual-modal three-dimensional gesture posture estimation method and system based on multi-frequency attention.

[0010] The dual-modal three-dimensional gesture posture estimation method based on multi-frequency attention proposed by the application comprises the following steps:

[0011] S1, acquiring RGB data and event data corresponding to a three-dimensional gesture to be recognized, and pre-processing the RGB data and event data to obtain dual-modal data aligned in space and time;

[0012] S2, inputting the dual-modal data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture, so as to estimate the three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized.

[0013] Preferably, the multi-frequency attention network comprises:

[0014] a multi-frequency attention fusion module for extracting and fusing multi-frequency features of the dual-modal data aligned in space and time to obtain fusion features;

[0015] a time sequence attention prediction module for predicting posture parameters and shape parameters of hand key points by using the fusion features;

[0016] a MANO reconstruction module for completing three-dimensional mesh model reconstruction of the hand based on the posture parameters and the shape parameters.

[0017] Preferably, the generation process of the fusion features comprises:

[0018] concatenating the dual-modal data aligned in space and time, obtaining shallow features through a shallow residual convolution layer, and dividing the shallow features into n sub-portions along the channel dimension;

[0019] extracting multi-frequency features of each sub-portion through two-dimensional discrete cosine transform (DCT), and merging the multi-frequency features of each sub-portion to obtain a multi-frequency feature vector;

[0020] adding the multi-frequency feature vector and the shallow features to obtain feature enhancement weights, enhancing the features corresponding to the dual-modal data through the feature enhancement weights, and obtaining the fusion features through a convolution layer after enhancement.

[0021] Preferably, the dual-modal data aligned in space and time is concatenated and shallow features are obtained through a shallow residual convolution layer, specifically as follows:

[0022]

[0023] in, Indicates shallow features; This represents the concatenated bimodal data; f D This indicates a shallow residual convolutional layer.

[0024] Preferably, the extraction of multi-frequency features of each sub-part using two-dimensional discrete cosine transform (DCT) specifically involves:

[0025]

[0026] in, Let $\mathbf{i}$ represent the compressed vector of the $i$-th sub-part of the shallow feature, i.e., the multi-frequency feature of the $i$-th sub-part; $H$ and $W$ represent the two remaining dimensions of the shallow feature besides the channel dimension, where $H$ represents the width of the shallow feature and $W$ represents the length of the shallow feature; i ,v i} represents the frequency index of the i-th sub-part of the corresponding shallow feature; F i denoted as the multi-frequency feature of the i-th sub-part; h represents any specific width of the shallow feature in the summation operation; w represents any specific length of the shallow feature in the summation operation.

[0027] Preferably, the step of merging the multi-frequency features of each sub-part to obtain a multi-frequency feature vector specifically involves:

[0028] F=Sigmoid(fc(Cat([F0,F1,…,F n-1 ])))#

[0029] Where fc represents a fully connected layer; sigmoid represents an activation function; Cat represents a concatenation operation; F0, F1, ..., F n-1 F represents the multi-frequency features corresponding to the n sub-parts; F is the multi-frequency feature vector.

[0030] Preferably, the training process of the multi-frequency attention network includes:

[0031] A 3D gesture pose dataset containing RGB data and event data is used. The 3D gesture pose dataset contains RGB data, event data, bimodal data, and a one-to-one mapping relationship between 3D gesture poses.

[0032] The bimodal data in the 3D gesture pose dataset is used as the input feature, and the 3D gesture pose corresponding to the bimodal data is used as the target variable. These are then input into the initial multi-frequency attention network for model training.

[0033] During the training process, the average joint position error, the average vertex position error and the position joint error are used as loss functions, and when the average joint position error, the average vertex position error and the position joint error all converge to a preset loss, the current model parameters are reserved to obtain the trained multi-frequency attention network.

[0034] Preferably, the average joint position error is calculated according to the following formula:

[0035]

[0036] wherein K represents the number of joints, and K is usually 21 in three-dimensional gesture reconstruction, is the predicted three-dimensional coordinate of the i-th joint, represents the real labeled three-dimensional coordinate of the i-th joint; ||.||2 represents the Euclidean distance calculation; L J is the average joint position error.

[0037] Preferably, the average vertex position error is calculated according to the following formula:

[0038]

[0039] wherein M represents the number of grid vertices, and M is usually 778 in three-dimensional gesture grid model reconstruction; is the predicted three-dimensional coordinate of the i-th grid vertex, represents the real labeled three-dimensional coordinate of the i-th grid vertex; ||.||2 represents the Euclidean distance calculation.

[0040] Preferably, the position joint error is calculated according to the following formula:

[0041]

[0042] wherein L all represents the position joint error, and represent the average joint position error and the average vertex position error of the predicted result relative to the labeled RGB frame; and represent the average joint position error and the average vertex position error of the predicted result relative to the labeled stacked event frame; λ J and λ V are constants, respectively representing the weight of and the weight of.

[0043] Preferably, the three-dimensional gesture posture dataset includes a training set, a test set and a validation set, wherein 70% of the three-dimensional gesture posture dataset is used as the training set, 20% of the three-dimensional gesture posture dataset is used as the test set, and 10% of the three-dimensional gesture posture dataset is used as the validation set.

[0044] Preferably, the preprocessing specifically includes:

[0045] Feature extraction is performed on the RGB data and the stacked event frame respectively to obtain corresponding RGB feature maps and event feature maps;

[0046] The RGB feature maps and the event feature maps are input into a time sequence prediction convolution layer to obtain spatial offset, and then the spatial offset and the event feature maps are input into a convolution layer to obtain an event feature map after space-time alignment;

[0047] The event feature map after space-time alignment and the RGB feature map are used as double-modal data after space-time alignment.

[0048] The double-modal three-dimensional gesture posture estimation system based on multi-frequency attention proposed by the application comprises:

[0049] The data acquisition module is configured to acquire RGB data and event data corresponding to a three-dimensional gesture to be recognized, and to preprocess the RGB data and the event data to obtain double-modal data after space-time alignment;

[0050] The data processing and output module is configured to input the double-modal data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture, so as to estimate a three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized.

[0051] The multi-frequency attention network comprises:

[0052] The multi-frequency attention fusion module is configured to perform multi-frequency feature extraction and fusion on the double-modal data after space-time alignment to obtain fusion features.

[0053] The time sequence attention prediction module is configured to predict posture parameters and shape parameters of hand key points by using the fusion features.

[0054] The MANO reconstruction module is configured to complete three-dimensional mesh model reconstruction of the hand based on the posture parameters and the shape parameters.

[0055] In the present application, the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention realizes three-dimensional gesture pose estimation using dual-modal data by adopting a multi-frequency attention network. The technology combines a deep learning algorithm, uses a timestamp and a time sequence convolution layer to complete the space-time alignment of dual-modal data. Then, a multi-frequency attention network is used to complete the estimation of the three-dimensional gesture pose. Compared with traditional gesture pose estimation methods, the present application can fully utilize the complementarity of RGB data and event data, retain high-frequency details, and still have high estimation accuracy and robustness under extreme lighting conditions, thereby improving the reconstruction accuracy and efficiency of gesture pose estimation under extreme lighting conditions. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A workflow schematic diagram of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure.

[0057] Figure 2 A preprocessing flowchart of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure.

[0058] Figure 3 An architecture schematic diagram of the multi-frequency attention network of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure.

[0059] Figure 4 A structure schematic diagram of the multi-frequency attention fusion module of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure.

[0060] Figure 5 A feature splicing operation structure diagram of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure.

[0061] Figure 6 An implementation comparison display schematic diagram of the proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention of the present application is shown in the figure. DETAILED DESCRIPTION

[0062] REFERENCE Figures 1-6 The proposed dual-modal three-dimensional gesture pose estimation method based on multi-frequency attention includes the following steps:

[0063] S1, acquiring RGB data and event data corresponding to a three-dimensional gesture to be recognized, and preprocessing the RGB data and the event data to obtain dual-modal data that is space-time aligned.

[0064] In the present embodiment, the preprocessing specifically includes:

[0065] The RGB data and the stacked event frames are respectively subjected to feature extraction to obtain corresponding RGB feature maps and event feature maps.

[0066] The RGB feature map and the event feature map are input into the temporal prediction convolutional layer to obtain the spatial offset. Then, the spatial offset and the event feature map are input into the convolutional layer to obtain the spatiotemporally aligned event feature map.

[0067] The spatiotemporally aligned event feature map and RGB feature map are used as spatiotemporally aligned bimodal data.

[0068] Specifically, during the generation of stacked event frames, the specific stacked event functions are as follows:

[0069]

[0070] Among them, I Ev (t,t i ,t s ) represents an event within a timestamp; t i Indicates the trigger time of the event; t s This indicates the calibration time of the nearest timestamp. The event trigger time within the event window is projected onto the range [0,1], thus quantifying the event's relative position within the time window; a larger value indicates a newer event. Simultaneously, only the latest event is retained at the same pixel position, thereby generating stacked event frames within the event window.

[0071] Specifically, the calculation formulas for the RGB feature map and the event feature map are as follows:

[0072]

[0073] Where F Im,t and F Ev,t These represent RGB data and stacked event frames, respectively, f c This represents a convolutional layer that initially extracts features from RGB data and stacked event frames. and This represents the RGB feature map and the event feature map.

[0074] Specifically, the formula for calculating the spatial offset is as follows:

[0075]

[0076] Where ΔP is the spatial position offset of the event camera; These are event feature maps and RGB image feature maps, respectively; ConvLSTM represents a temporal prediction convolutional layer; ConvLSTM directly processes image sequences (such as video frames) by replacing the fully connected layers in traditional LSTM with convolutional operations, utilizing convolution to extract spatial features and model temporal dependencies, thereby effectively capturing the spatiotemporal dynamics of images. This allows for data spatial alignment without requiring camera parameters.

[0077] In the embodiment, the specific calculation formula of the spatio-temporally aligned event feature map is as follows:

[0078]

[0079] wherein, is the spatio-temporally aligned event feature map, represents the event feature map; f DC represents a variable convolution layer, which dynamically adjusts the sampling position of a traditional convolution by introducing a learnable offset, calculates the feature value of a non-integer offset point by using bilinear interpolation, and combines deformable RoI pooling to enhance the modeling ability of geometric deformation, so as to more flexibly capture the details and complex shapes of irregular targets in an image.

[0080] S2, inputting the bimodal data into the pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture, so as to estimate the three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized.

[0081] In the embodiment, the multi-frequency attention network comprises:

[0082] a multi-frequency attention fusion module, configured to perform multi-frequency feature extraction and fusion on the spatio-temporally aligned bimodal data to obtain fusion features;

[0083] a time sequence attention prediction module, configured to predict posture parameters and shape parameters of hand key points by using the fusion features;

[0084] a MANO reconstruction module, configured to complete three-dimensional mesh model reconstruction of the hand based on the posture parameters and the shape parameters.

[0085] In the embodiment, as shown in Figure 5 the splicing operation comprises:

[0086] merging two or more vectors of the same shape into the same vector in any dimension, so as to realize merging of two or more vectors.

[0087] In the embodiment, the generation process of the fusion features comprises:

[0088] splicing the spatio-temporally aligned bimodal data to obtain shallow features through a shallow residual convolution layer, and dividing the shallow features into n subparts along the channel dimension;

[0089] extracting multi-frequency features of each subpart through two-dimensional discrete cosine transform (DCT), and merging the multi-frequency features of each subpart to obtain a multi-frequency feature vector;

[0090] adding the multi-frequency feature vector and the shallow features to obtain feature enhancement weights, and enhancing the features corresponding to the bimodal data through the feature enhancement weights to obtain the fusion features through a convolution layer.

[0091] In the embodiment, the spatio-temporally aligned bimodal data is spliced through a shallow residual convolution layer to obtain a shallow feature, specifically:

[0092]

[0093] wherein, represents the shallow feature; represents the spliced bimodal data; f D represents the shallow residual convolution layer.

[0094] In the embodiment, multi-frequency features of each sub-part are extracted through a two-dimensional discrete cosine transform (DCT), specifically:

[0095]

[0096] wherein, represents a compressed vector of the i-th sub-part of the shallow feature, i.e., the multi-frequency feature of the i-th sub-part; H and W represent two dimensions of the shallow feature except the channel dimension, wherein H represents the width of the shallow feature, and W represents the length of the shallow feature; {u i ,v i} represents the frequency index corresponding to the i-th sub-part of the shallow feature; F i is the multi-frequency feature of the i-th sub-part; h represents any specific width of the shallow feature in the summation operation; w represents any specific length of the shallow feature in the summation operation.

[0097] In the embodiment, the multi-frequency features of each sub-part are merged to obtain a multi-frequency feature vector, specifically:

[0098] F = Sigmoid(fc(Cat([F0, F1, …, Fn])) n-1 )))

[0099] wherein, fc represents a full connection layer; Sigmoid represents an activation function; Cat represents a splicing operation; F0, F1, …, Fn represent the multi-frequency features corresponding to n sub-parts; F represents the multi-frequency feature vector. n-1

[0100] In the embodiment, the multi-frequency feature vector is added to the shallow feature to obtain a feature enhancement weight, and the feature corresponding to the bimodal data is enhanced through the feature enhancement weight, and then a convolution layer is used to obtain a fusion feature, specifically including:

[0101] The multi-frequency feature vector F is added to an initial data block obtained through another convolution layer to obtain a feature enhancement weight, and the stacked event frame feature map and the RGB frame feature map ​The stacked event frame feature maps are multiplied by the feature enhancement weights respectively to obtain enhanced stacked event frame feature maps and RGB frame feature maps The enhanced stacked event frame feature maps and RGB frame feature maps are spliced, and then fused features are obtained through a convolution layer to complete data fusion.

[0102] In the embodiment, the training process of the multi-frequency attention network includes:

[0103] A three-dimensional gesture posture dataset containing RGB data and event data is used, wherein the three-dimensional gesture posture dataset contains RGB data, event data, bimodal data, and a one-to-one mapping relationship between the three-dimensional gesture posture;

[0104] The bimodal data in the three-dimensional gesture posture dataset is input as an input feature, and the three-dimensional gesture posture corresponding to the bimodal data is input as a target variable into the initial multi-frequency attention network for model training.

[0105] During the training process, the average joint position error, the average vertex position error, and the position joint error are used as loss functions, and when the average joint position error, the average vertex position error, and the position joint error all converge to a preset loss, the current model parameters are retained to obtain the trained multi-frequency attention network.

[0106] Specifically, the average joint position error is used to measure the average Euclidean distance error between the predicted joint position and the true label, and the calculation formula is as follows:

[0107]

[0108] wherein K represents the number of joints, and in three-dimensional gesture reconstruction, K is usually 21, is the predicted three-dimensional coordinate of the i-th joint, represents the true labeled three-dimensional coordinate of the i-th joint. ||.||2 represents the Euclidean distance calculation; L J is the average joint position error; therefore L J can directly reflect the geometric error between the predicted joint position and the true position, and the smaller the value is, the more accurate the prediction is, which can be used as one of the objective evaluation indexes of three-dimensional gesture mesh model reconstruction.

[0109] The average vertex position error is used to measure the average geometric error between the predicted three-dimensional mesh vertex and the true label, and the calculation formula is as follows:

[0110]

[0111] wherein M represents the number of mesh vertices, and M is usually 778 in the three-dimensional gesture mesh model reconstruction. predicated three-dimensional coordinates representing the i-th mesh vertex; true labeled three-dimensional coordinates representing the i-th mesh vertex. ||.||2 represents the Euclidean distance calculation. Compared with the average joint position error, the average vertex position error can evaluate the accuracy of all hand mesh vertices, and can more comprehensively reflect the shape and surface detail error of the hand mesh, so the average vertex position error is often used in the three-dimensional mesh reconstruction task, and the smaller the value is, the more accurate the prediction is, which can be used as one of the objective evaluation indexes of the three-dimensional gesture mesh model reconstruction.

[0112] Specifically, the position joint error is calculated by weighting the average joint position error and the average vertex position error. The position joint error is calculated as follows:

[0113]

[0114] wherein L all represents the position joint error, and represent the average joint position error and the average vertex position error of the predicted result relative to the labeled RGB frame. and represent the average joint position error and the average vertex position error of the predicted result relative to the labeled stacked event frame. λ J and λ V are constants, respectively representing the weight of the weight of

[0115] In the embodiment, the three-dimensional gesture posture dataset includes a training set, a test set, and a validation set, wherein 70% of the three-dimensional gesture posture dataset is used as the training set, 20% of the three-dimensional gesture posture dataset is used as the test set, and 10% of the three-dimensional gesture posture dataset is used as the validation dataset.

[0116] It should be noted that, compared with the conventional scheme in the art, the use of multi-frequency attention can effectively utilize the high-frequency information of the RGB and event dual modal data, thereby completing joint feature extraction and fusion, and highlighting the target features in the process of mining frequency domain information, and reducing the interference of background noise. The conventional scheme in the art uses a standard attention mechanism to extract and fuse features by focusing on the spatial correlation between modalities in the process of feature extraction and modeling, so it mainly captures low-frequency information representing the global structure, and it is difficult to retain high-frequency signals carrying local details. Therefore, in the face of strong light conditions, the conventional scheme in the art usually lacks high-frequency signal features carrying local details, so it cannot guarantee stable and effective fusion features, thereby affecting the accuracy and precision of subsequent three-dimensional gesture mesh model reconstruction.

[0117] The multi-frequency attention mechanism used in the multi-frequency attention fusion module can work in cooperation with the subsequent time sequence attention prediction module. The data feature extraction in the frequency domain and the time domain of the dual-modal data can be more helpful to enhance the modal-specific features, so as to achieve the best tracking effect. The use of the multi-frequency attention fusion module to complete the feature fusion can effectively utilize the high-frequency detailed information of the dual-modal data, can retain the hand details in extreme conditions, and thus makes the pose parameters and shape parameters required by the subsequent MANO reconstruction module more accurate.

[0118] It should be noted that, compared with the conventional scheme in the art, the use of the time sequence attention prediction module can work in cooperation with the multi-frequency attention fusion module through frequency domain and time domain processing, which is more helpful to enhance the modal-specific features, so as to achieve the best tracking effect. The hand movement shows obvious space-time continuity, and the event stream contains rich time sequence and motion information, so the use of the time sequence attention prediction module can also work in cooperation with the subsequent MANO reconstruction module, so that the extracted pose parameters and shape parameters are more coherent and smooth in the time dimension.

[0119] Specifically, compared with the conventional scheme in the art, the use of the MANO reconstruction module can work in cooperation with the previous multi-frequency attention fusion module and time sequence attention prediction module, and the use of the parameterized hand mesh model can constrain the anatomical rationality of the reconstruction result through parameters, which shows stronger robustness compared with the non-parameterized method.

[0120] In the embodiment, the three-dimensional reconstruction method of the application is compared with the existing evaluation methods. The existing evaluation methods include the following six methods: MG, FR, EH, FE, Vanilla, EvRGBHand and DaEvRGBHand. Among them, MG (MeshGraphormer) and FR (FastMETRO-RGB) are used as baseline methods of the three-dimensional gesture mesh model reconstruction algorithm based on the RGB camera. EH (EventHands) and FE (FastMETRO-Event) are used as baseline methods of the three-dimensional gesture mesh model reconstruction algorithm based on the event camera. Vanilla (EvRGBHand Vanilla), EvRGBHand and DaEvRGBHand are used as baseline methods of the three-dimensional gesture mesh model reconstruction algorithm based on the fusion of the event camera and the RGB camera. DaEvRGBHand is a three-dimensional gesture mesh model reconstruction algorithm for gesture estimation using a bidirectional attention mechanism. The above methods are used to obtain the data in the following table after training for 200 rounds. The experimental results are shown in the following table:

[0121]

[0122]

[0123] In the embodiment, the objective indicators of the method of the present application under strong light conditions are obviously improved compared with other methods. Due to the inter-modal joint feature extraction and modeling of the visual attention mechanism structure, high-frequency information may be lost and noise may be introduced, thereby affecting the accurate tracking of the target. In the process of establishing the relationship between the template and the search area, the multi-frequency attention network introduces DTC to extract multi-frequency information of the RGB and event two modalities, which is used to make up for the defect of ignoring the visual domain information in the multi-head attention mechanism.

[0124] With reference to Figures 1-6 The dual-modality three-dimensional gesture pose estimation system based on multi-frequency attention proposed by the present application comprises:

[0125] A data acquisition module is configured to acquire RGB data and event data corresponding to a three-dimensional gesture to be recognized, and to pre-process the RGB data and the event data to obtain dual-modality data aligned in time and space.

[0126] A data processing and output module is configured to input the dual-modality data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture, so as to estimate a three-dimensional gesture pose corresponding to the three-dimensional gesture to be recognized.

[0127] The multi-frequency attention network comprises:

[0128] A multi-frequency attention fusion module is configured to extract and fuse multi-frequency features from the dual-modality data aligned in time and space, so as to obtain fused features.

[0129] A time sequence attention prediction module is configured to predict pose parameters and shape parameters of hand key points by using the fused features.

[0130] A MANO reconstruction module is configured to complete three-dimensional mesh model reconstruction of a hand based on the pose parameters and the shape parameters.

[0131] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can make equivalent substitutions or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A dual-modality three-dimensional gesture pose estimation method based on multi-frequency attention, characterized in that, The method comprises the following steps: S1, obtaining RGB data and event data corresponding to a three-dimensional gesture to be recognized, and pre-processing the RGB data and the event data to obtain spatio-temporally aligned dual-mode data; S2, inputting the dual-mode data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture, so as to estimate a three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized; The multi-frequency attention network comprises: A multi-frequency attention fusion module for extracting and fusing multi-frequency features of the spatio-temporally aligned dual-mode data to obtain fusion features; A time sequence attention prediction module for predicting posture parameters and shape parameters of hand key points by using the fusion features; A MANO reconstruction module for completing three-dimensional mesh model reconstruction of a hand based on the posture parameters and the shape parameters.

2. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 1, characterized in that, The generation process of the fusion features comprises: concatenating the spatio-temporally aligned dual-mode data, passing the concatenated data through a shallow residual convolution layer to obtain shallow features, and dividing the shallow features into n sub-portions along a channel dimension; extracting multi-frequency features of each sub-portion by two-dimensional discrete cosine transform (DCT), and merging the multi-frequency features of each sub-portion to obtain a multi-frequency feature vector; adding the multi-frequency feature vector to the shallow features to obtain feature enhancement weights, enhancing features corresponding to the dual-mode data by using the feature enhancement weights, and then passing the enhanced features through a convolution layer to obtain the fusion features.

3. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 2, characterized in that, The shallow features are obtained by concatenating the spatio-temporally aligned dual-mode data and passing the concatenated data through a shallow residual convolution layer. wherein, represents a shallow feature; represents the stitched bimodal data;f D represents a shallow residual convolutional layer.

4. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 2, characterized in that, The multi-frequency features of each sub-portion are extracted by two-dimensional discrete cosine transform (DCT). wherein, represents the compressed vector of the i-th sub-part of the shallow feature, i.e., the multi-frequency feature of the i-th sub-part; H and W represent the two dimensions of the shallow feature except the channel dimension, wherein H represents the width of the shallow feature and W represents the length of the shallow feature; {u i ,v i} represents the frequency index corresponding to the i-th sub-part of the shallow feature; F i is the multi-frequency feature of the i-th sub-part; h represents any specific width of the shallow feature in the summation operation; and w represents any specific length of the shallow feature in the summation operation.

5. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 2, characterized in that, The multi-frequency features of each sub-portion are merged to obtain a multi-frequency feature vector. F = Sigmoid(fc(Cat([F0, F1, …, F n-1 ])))# wherein f c represents a fully connected layer; Sigmoid represents an activation function; Cat represents a concatenation operation; F0, F1, …, F n-1 are multi-frequency features corresponding to n sub-parts; F is a multi-frequency feature vector.

6. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 1, characterized in that, The training process of the multi-frequency attention network comprises: using a three-dimensional gesture posture dataset containing RGB data and event data, wherein the three-dimensional gesture posture dataset contains RGB data, event data, dual-mode data, and a one-to-one mapping relationship between the three-dimensional gesture posture; inputting the dual-mode data in the three-dimensional gesture posture dataset as input features and the three-dimensional gesture posture corresponding to the dual-mode data as target variables into an initial multi-frequency attention network to perform model training; in the training process, using average joint position error, average vertex position error, and position joint error as loss functions, and when the average joint position error, the average vertex position error, and the position joint error all converge to a preset loss, retaining the current model parameters to obtain a trained multi-frequency attention network.

7. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 6, characterized in that, The average joint position error is calculated according to the following formula: where K represents the number of joints, and K is usually 21 in three-dimensional gesture reconstruction, is the predicted three-dimensional coordinate of the i-th joint, is the real labeled three-dimensional coordinate of the i-th joint; ||.||2represents the Euclidean distance calculation; L J is the average joint position error.

8. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 6, characterized in that, The average vertex position error is calculated according to the following formula: where M represents the number of mesh vertices, and M is usually 778 in the reconstruction of three-dimensional gesture mesh model; represents the predicted three-dimensional coordinates of the i-th mesh vertex; represents the real labeled three-dimensional coordinates of the i-th mesh vertex; and ||.||2represents the Euclidean distance calculation.

9. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 6, characterized in that, The position joint error is calculated according to the following formula: where L all denotes the position joint error, and denotes the average joint position error and the average vertex position error of the predicted result relative to the labeled RGB frame; and denotes the average joint position error and the average vertex position error of the predicted result relative to the labeled stacked event frame; λ J and λ V are constants, respectively representing the weight of the weight of 10. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 6, characterized in that, The three-dimensional gesture posture dataset comprises a training set, a test set, and a validation set, wherein 70% of the three-dimensional gesture posture dataset is used as the training set, 20% of the three-dimensional gesture posture dataset is used as the test set, and 10% of the three-dimensional gesture posture dataset is used as the validation set.

11. The multi-frequency attention based dual-modal three-dimensional gesture pose estimation method according to claim 1, characterized in that, The pre-processing specifically comprises: performing feature extraction on the RGB data and stacked event frames to obtain corresponding RGB feature maps and event feature maps; The RGB feature map and the event feature map are input into a time sequence prediction convolutional layer to obtain a spatial offset, and then the spatial offset and the event feature map are input into a convolutional layer to obtain an event feature map after space-time alignment; The event feature map after space-time alignment and the RGB feature map are used as double-modal data after space-time alignment.

12. A dual modality three-dimensional gesture pose estimation system based on multi-frequency attention, characterized in that, It comprises: a data acquisition module configured to acquire RGB data and event data corresponding to a three-dimensional gesture to be recognized, and to pre-process the RGB data and the event data to obtain double-modal data after space-time alignment; a data processing and output module configured to input the double-modal data into a pre-trained multi-frequency attention network to complete three-dimensional reconstruction of the gesture and estimate a three-dimensional gesture posture corresponding to the three-dimensional gesture to be recognized; The multi-frequency attention network comprises: a multi-frequency attention fusion module configured to extract and fuse multi-frequency features from the double-modal data after space-time alignment to obtain fused features; a time sequence attention prediction module configured to predict posture parameters and shape parameters of hand key points using the fused features; a MANO reconstruction module configured to complete three-dimensional mesh model reconstruction of a hand based on the posture parameters and the shape parameters.

Images