A multimodal human motion recognition method and system based on RGB-D frame images

Abstract

This invention provides a multimodal human action recognition method and system based on RGB-D frame images, belonging to the field of image processing technology. The method includes: acquiring RGB-D video data under first-view action conditions; extracting time-aligned RGB video frame sequences and Depth video frame sequences; inputting them into a dual-stream convolutional neural network to extract RGB high-dimensional spatial features and Depth high-dimensional spatial features; performing channel dimensionality reduction, aggregating them through one-dimensional temporal convolution of consecutive frames, and then flattening them along the spatial dimension; injecting spatial position encoding, and then inputting it into a cross-modal fusion Transformer network for feature fusion to obtain a fused feature sequence; inputting it into a classifier to output the corresponding action recognition result. This invention collaboratively solves three major problems under first-view conditions: dynamic blur, dimensionality explosion, and feature contamination, significantly reducing computational overhead while ensuring high recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a multimodal human motion recognition method and system based on RGB-D frame images. Background Technology

[0002] With the continuous development of intelligent and human-machine collaborative systems, robots' ability to understand fine-grained movements from a first-person perspective has become the core foundation for achieving autonomous teaching and learning as well as complex interactive tasks.

[0003] However, in first-person perspective scenarios, existing mainstream deep learning-based multimodal recognition methods generally use temporal compression techniques (such as rank pooling) to fuse multi-frame information. This can lead to motion blur and ghosting at the hand-object edges under severe shaking in a first-person perspective, destroying crucial microscopic motion details. If a complete multi-frame sequence is directly input to avoid blurring, the surge in sequence length can trigger the curse of dimensionality, causing a quadratic increase in computational overhead for advanced models relying on self-attention (such as the Transformer), making real-time operation impossible on edge devices like robots. Furthermore, RGB images in complex indoor environments are rich in background noise. Existing early or direct cross-modal fusion methods easily allow this noise to contaminate the pure geometric structure information in the deep modalities, leading to a decrease in the discriminative power of the fused features. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a multimodal human motion recognition method and system based on RGB-D frame images, thereby solving at least one problem in the background technology.

[0005] This invention provides a multimodal human motion recognition method based on RGB-D frame images, the method comprising: Collect RGB-D video data of the robot's actions from a first-person perspective, extract time-aligned RGB video frame sequences and Depth video frame sequences, and perform preprocessing. The preprocessed RGB video frame sequence and the Depth video frame sequence are respectively input into a dual-stream convolutional neural network with non-shared parameters to extract RGB high-dimensional space features and Depth high-dimensional space features respectively. Channel dimensionality reduction is performed on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively to obtain RGB single-frame features and Depth single-frame features. Then, after one-dimensional temporal convolution aggregation of several consecutive frames, RGB spatiotemporal hybrid features and Depth spatiotemporal hybrid features are generated. Subsequently, they are flattened along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. The RGB feature sequence and the Depth feature sequence are injected into the spatial location code, and then input into the cross-modal fusion Transformer network for feature fusion to obtain the fused feature sequence. In this process, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. The fused feature sequence is subjected to global feature aggregation to obtain video-level global features, which are then input into a classifier to output the corresponding action recognition results.

[0006] According to one aspect of the above technical solution, the dual-stream convolutional neural network uses a pre-trained ResNet-50 as the backbone network. By removing the last classification layer and the global average pooling layer, the output of the fourth bottleneck layer is extracted as the high-dimensional spatial feature and the Depth high-dimensional spatial feature. The dimensions of the high-dimensional spatial feature and the Depth high-dimensional spatial feature are both C×H×W, where C, H, and W are the number of channels, height, and width of the feature map, respectively.

[0007] According to one aspect of the above technical solution, the channel dimensionality reduction reduces the number of channels of the RGB high-dimensional spatial features and the Depth high-dimensional spatial features from C dimensions to 1 / 4C dimensions through a 1×1 convolutional layer, resulting in the RGB single-frame features and the Depth single-frame features with dimensions of 1 / 4C×H×W.

[0008] According to one aspect of the above technical solution, the step of one-dimensional temporal convolution aggregation includes: The RGB single-frame features and the Depth single-frame features of N consecutive frames with dimensions of 1 / 4C×H×W are concatenated along the time dimension to obtain RGB feature tensors and Depth feature tensors with dimensions of 1 / 4C×H×W×N. A one-dimensional convolution kernel with a kernel size of M and a stride of 1 is used to perform convolution operations on the RGB feature tensor and the Depth feature tensor along the time dimension, respectively, to output the RGB spatiotemporal mixed feature and the Depth spatiotemporal mixed feature with a dimension of 1 / 4C×H×W, where M is a positive integer less than N.

[0009] According to one aspect of the above technical solution, the spatial location encoding is a location encoding based on a sine function, used to characterize the absolute position of feature points in an H×W spatial grid.

[0010] According to one aspect of the above technical solution, based on the multi-head self-attention mechanism, global dependency modeling and feature purification are performed on the RGB feature sequence and the Depth feature sequence within a single modality, respectively, to complete the self-attention calculation within the modality; Based on the multi-head cross-attention mechanism, a first interactive calculation is performed using the purified RGB feature sequence as the query vector and the purified Depth feature sequence as the key and value vector, and a second interactive calculation is performed using the purified Depth feature sequence as the query vector and the purified RGB feature sequence as the key and value vector, to complete the intermodal cross-attention calculation.

[0011] According to one aspect of the above technical solution, after the intramodal self-attention calculation and after the intermodal cross-attention calculation, a bottleneck-excited feedforward network is connected to perform channel recalibration on the input features.

[0012] According to one aspect of the above scheme, the calculation steps of the bottleneck-excitation feedforward network include: Global average pooling is performed on the input features to obtain channel descriptors; The channel descriptors are sequentially subjected to dimensionality reduction linear transformation, nonlinear activation, dimensionality increase linear transformation, and Sigmoid activation to generate channel weights; The channel weights are multiplied channel by channel by the input features.

[0013] According to one aspect of the above technical solution, the step of performing global feature aggregation on the fused feature sequence to obtain video-level global features, and inputting these features into a classifier to output the corresponding action recognition result specifically includes: Global average pooling is performed on the fused feature sequence to obtain a video-level global feature vector; The video-level global feature vector is input into a linear classifier, and after being normalized by the Softmax function, the action class probability is output.

[0014] Another aspect of the present invention provides a multimodal human motion recognition system based on RGB-D frame images, the system being used to implement the aforementioned multimodal human motion recognition method based on RGB-D frame images, the system comprising: The preprocessing module is used to acquire RGB-D video data of the robot's first-person perspective actions, extract time-aligned RGB video frame sequences and depth video frame sequences from them, and perform preprocessing. The feature extraction module is used to input the preprocessed RGB video frame sequence and the Depth video frame sequence into a dual-stream convolutional neural network with non-shared parameters, and extract RGB high-dimensional space features and Depth high-dimensional space features respectively. The temporal aggregation module is used to perform channel dimensionality reduction on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively to obtain RGB single-frame features and Depth single-frame features. Then, after one-dimensional temporal convolution aggregation of several consecutive frames, RGB spatiotemporal hybrid features and Depth spatiotemporal hybrid features are generated. Subsequently, they are flattened along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. The cross-modal fusion module is used to inject RGB feature sequences and Depth feature sequences into spatial location encoding, and then input them into the cross-modal fusion Transformer network for feature fusion to obtain a fused feature sequence. In this process, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. The classification module is used to perform global feature aggregation on the fused feature sequence to obtain video-level global features, and input them into the classifier to output the corresponding action recognition results.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By using lightweight one-dimensional convolution along the time axis to perform local fusion on the high-dimensional semantic features extracted by the dual-stream convolutional neural network, it can model the continuous semantic evolution of actions and avoid pixel-level blurring. It can also smoothly compress multi-frame sequences into single-frame features, which can drastically reduce the sequence length processed by the subsequent cross-modal fusion Transformer network and reduce the computational overhead by orders of magnitude. Thus, while preserving fine spatiotemporal information, it breaks through the computing power bottleneck of deployment on edge devices.

[0016] 2. To address the issue of background noise in RGB images interfering with depth information in real-world scenarios, this invention forces RGB features and Depth features to undergo independent intramodal self-attention cleanup in each layer of fusion, and then interacts through bidirectional cross-attention. This asymmetric cascaded design establishes a cleanup buffer for both features before cross-modal information exchange, blocking the early propagation path of noise and ensuring the purity of depth geometric information. Combined with the channel recalibration of the bottleneck-excited feedforward network, useful features are further enhanced, thereby significantly improving the recognition accuracy and stability of the model in cluttered and variable environments. Attached Figure Description

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a schematic diagram illustrating the principle framework of the multimodal human motion recognition method for RGB-D frame images in Embodiment 1 of the present invention. Figure 2 This is a framework diagram of the MEM in Embodiment 1 of the present invention; Figure 3 This is a partial framework diagram of the MIM in Embodiment 1 of the present invention; Figure 4 This is a framework diagram of the bottleneck excitation feedforward network in Embodiment 1 of the present invention. Detailed Implementation

[0018] To make the objectives, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0019] Example 1 Embodiment 1 of the present invention provides a multimodal human action recognition method based on RGB-D frame images, such as... Figure 1 As shown, the method includes steps S1-S7: Step S1: Collect RGB-D video data of the robot's actions from the first-person perspective, extract time-aligned RGB video frame sequences and Depth video frame sequences, and perform preprocessing. Specifically, the robot uses an RGB-D camera (such as an NVIDIA D435i camera or Kinect) to simultaneously capture RGB and Depth video streams from a first-person perspective, with the video frame rate set to 30fps. N consecutive frames are extracted (N=8 in this embodiment), and the two video streams are strictly time-stamp aligned. Linear normalization is performed on the Depth video stream to map the original depth values ​​to the grayscale range of [0, 255], making the Depth video stream format consistent with the RGB video stream format; Adjust the resolution of all frames in the Depth and RGB video streams to a uniform 224×224 pixels.

[0020] To address the blurring issue caused by severe shaking in the first-person perspective, this step does not introduce any preprocessing that compresses the time dimension (such as dynamic image generation), and directly outputs the original, high-fidelity continuous frame sequence. This avoids motion blur caused by time dimension information compression from the source, preserving complete microscopic spatiotemporal information for subsequent steps.

[0021] Step S2: Input the preprocessed RGB video frame sequence and the Depth video frame sequence into a dual-stream convolutional neural network with non-shared parameters to extract RGB high-dimensional space features and Depth high-dimensional space features respectively. Specifically, the dual-stream convolutional neural network uses a pre-trained ResNet-50 as its backbone network. By removing the last classification layer and the global average pooling layer, the output of the fourth bottleneck layer is extracted as the high-dimensional spatial feature and the Depth high-dimensional spatial feature to obtain a high-resolution, detailed feature map. The dimensions of both the high-dimensional spatial feature and the Depth high-dimensional spatial feature are C×H×W, where C, H, and W are the number of channels, height, and width of the feature map, respectively.

[0022] It should be noted that by leveraging the powerful representation capabilities of the pre-trained ResNet-50, high-level spatial semantic features are extracted for each frame of the image. At the same time, by removing subsequent downsampling layers, high spatial resolution is maintained, which is beneficial for subsequent modeling of fine-grained actions (such as finger poses).

[0023] As an example, not a limitation, the dimensions of the high-dimensional spatial features and the Depth high-dimensional spatial features are 2048×7×7, that is, they have 2048 channels and a spatial size of 7×7.

[0024] Step S3: Perform channel dimensionality reduction on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively to obtain RGB single-frame features and Depth single-frame features. Then, perform one-dimensional temporal convolution aggregation on several consecutive frames to generate RGB spatiotemporal hybrid features and Depth spatiotemporal hybrid features. Subsequently, flatten them along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. Specifically, in step S30, channel dimensionality reduction uses a 1×1 convolutional layer to reduce the number of channels in the RGB high-dimensional spatial features and the Depth high-dimensional spatial features from C dimensions to 1 / 4C dimensions, resulting in RGB single-frame features and Depth single-frame features with dimensions of 1 / 4C×H×W. Channel dimensionality reduction significantly reduces the channel dimensions that the subsequent one-dimensional temporal convolutional aggregation and cross-modal fusion Transformer network needs to process, greatly reducing the number of model parameters and computational cost; the 1×1 convolutional layer itself can be regarded as a linear combination and information integration between channels, which can perform preliminary refinement of the initial high-dimensional features.

[0025] Examples, not limitations, include the following: the dimensions of the high-dimensional spatial features and the Depth high-dimensional spatial features are 2048×7×7, and the RGB single-frame features and Depth single-frame features are obtained by channel dimensionality reduction to 512×7×7.

[0026] Step S31, the one-dimensional temporal convolution aggregation step, includes: The RGB single-frame features and the Depth single-frame features of N consecutive frames with dimensions of 1 / 4C×H×W are concatenated along the time dimension to obtain RGB feature tensors and Depth feature tensors with dimensions of 1 / 4C×H×W×N. A one-dimensional convolution kernel with a kernel size of M and a stride of 1 is used to perform convolution operations on the RGB feature tensor and the Depth feature tensor along the time dimension, respectively, to output the RGB spatiotemporal mixed feature and the Depth spatiotemporal mixed feature with a dimension of 1 / 4C×H×W, where M is a positive integer less than N.

[0027] This is an example, not a limitation. For instance, with N=8 and M=3, the original 8-frame sequence (if flattened, the sequence length = 8×7×7 = 392) would be fused into a single-frame feature (sequence length = 7×7 = 49). In other words, due to the use of uncompressed 8-frame input, directly flattening the feature sequence and feeding it into the cross-modal fusion Transformer network would result in a sequence length of 392, leading to a quadratic explosion in self-attention computation. By introducing a one-dimensional temporal convolution aggregation branch, which utilizes local receptive fields to extract inter-frame motion evolution in the deep semantic space, the 3D data is reduced to 2D (sequence length converges to 49). This constructs a dimensionality reduction bridge connecting multi-frame input and the subsequent cross-modal fusion Transformer network with extremely low computational overhead.

[0028] Furthermore, unlike motion overlay in the original pixel space (which leads to blurring), one-dimensional temporal convolutional aggregation performs temporal fusion in a high-level semantic feature space extracted by the convolutional network. Through learned convolutional kernels, it adaptively captures the evolution of semantic features between frames (such as continuous changes in hand position and object state), thus implicitly modeling motion information and avoiding pixel-level ghosting and blurring. Moreover, a kernel size of 3 means that it only focuses on the context of three adjacent frames at a time. This locality assumption aligns with the short-term continuity of most actions and is computationally lighter.

[0029] Step S32: Flatten along the spatial dimensions (height and width) to obtain the RGB feature sequence and the Depth feature sequence.

[0030] As an example, not a limitation, the 512×7×7 RGB spatiotemporal blending features and the Depth spatiotemporal blending features, when flattened along the spatial dimensions (height and width), yield two 512×49 RGB feature sequences and Depth feature sequences.

[0031] Step S4: Inject the RGB feature sequence and the Depth feature sequence into the spatial location encoding, and then input them into the cross-modal fusion Transformer network for feature fusion to obtain the fused feature sequence. In this step, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. Specifically, in step S40, the spatial location encoding is a location encoding based on a sine function, used to characterize the absolute position of feature points in the H×W spatial grid, without adding temporal location encoding. This is because the temporal evolution information of multiple frames has been aggregated through one-dimensional temporal convolution, fully and implicitly embedded in each channel of the RGB or Depth spatiotemporal hybrid features.

[0032] Furthermore, since the cross-modal fusion Transformer network lacks spatial awareness and cannot capture the relative relationships between different spatial positions in a frame image, directly modeling the features of the RGB and Depth feature sequences would result in the loss of fine-grained spatial information at the frame level, affecting the accuracy of action recognition. Therefore, a spatial position encoding based on a sine function is introduced to inject the spatial position information of the frame image into the features of the RGB and Depth feature sequences, ensuring that the spatial structure information is not destroyed in subsequent modeling processes.

[0033] Step S41: Input the RGB feature sequence carrying spatial location encoding and the Depth feature sequence into the cross-modal fusion Transformer network for feature fusion to obtain the fused feature sequence.

[0034] As an example and not a limitation, the number of stacked fusion layers in the cross-modal fusion Transformer network is L layers, where L=4. Each fusion layer follows a cascaded execution path of first cleaning the internal unimodal data and then bidirectional interaction across modalities, and a bottleneck-excited feedforward network is introduced for dynamic feature recalibration.

[0035] Specifically, based on the multi-head self-attention mechanism, global dependency modeling and feature purification are performed on the RGB feature sequences and Depth feature sequences within a single modality to complete the self-attention calculation within the modality. The RGB modality focuses on important objects and regions in the image (such as the hand or the object being manipulated) and suppresses irrelevant background noise, while the Depth modality strengthens the three-dimensional geometric structure (such as the outline of the object and its distance relationship with the hand). This is the purification process.

[0036] Specifically, such as Figure 2 As shown, MEM (Modal Augmentation Network) multiplies the RGB feature sequence and the Depth feature sequence with three learnable parameter matrices respectively, thereby mapping them to a query matrix through a subspace. Key matrix Sum matrix : , , , , in, It is a unimodal feature sequence, an intermediate feature sequence obtained by fusing the RGB feature sequence and the Depth feature sequence. For each attention head, the feature dimensions, For the number of attention heads, =8, The original feature dimension of 512 is divided into eight independent, lower-dimensional (64-dimensional) subspaces for parallel processing. This allows the model to focus on information from different locations in different representation subspaces, enhancing the model's representational power. =64.

[0037] For the One attention point ( =1, 2, ..., The corresponding queries, key-value slices are denoted as... , , Calculate the first The output of each attention head Use multi-head focus to capture key areas, as follows:

[0038] in, This is a scaling factor used to stabilize the dot product when the result is large. The gradient of a function.

[0039] All The outputs of each attention head are concatenated along the feature dimension and then processed by a learnable linear transformation matrix. Linear fusion and projection are performed to fuse information from each subspace, yielding the final output of Multi-Head Self-Attention (MHSA): , in, This is a feature dimension concatenation operation. The output of Multi-Head Self-Attention (MHSA) express OK, The linear space of columns.

[0040] To optimize information flow and training stability, the multi-head self-attention mechanism is integrated with residual connections, layer normalization (LN), and the aforementioned bottleneck-incentivized feedforward network (BEF). The complete forward propagation process of MEM is as follows: ， , in, This represents the enhanced and cleaned single-modal features output by MEM. The first line of the formula indicates: Input features First, global self-attention is performed using multi-head self-attention (MHSA) to capture long-range dependencies, and then compared with the input. The residuals are summed to preserve the original features, and then layer normalization is performed to obtain intermediate features. The second line of the formula indicates: [to...] Input a bottleneck-excited feedforward (BEF) network to perform local feature extraction and channel attention weighting, and then its output is fed into... The residuals are summed to preserve intermediate features, and then subjected to a final layer normalization to obtain the final output. This completes the enhancement and purification of features.

[0041] Furthermore, based on the multi-head cross-attention mechanism, a first interactive calculation is performed using the purified RGB feature sequence as the query vector and the purified Depth feature sequence as the key and value vector, and a second interactive calculation is performed using the purified Depth feature sequence as the query vector and the purified RGB feature sequence as the key and value vector, to complete the intermodal cross-attention calculation.

[0042] Specifically, such as Figure 3 As shown, MIM (Modal Interaction Network) consists of a multi-head cross-attention layer, a bottleneck-excited feedforward network (BEF), residual connections, and layer normalization. Through bidirectional query-key-value interaction, features from one modality actively query features from another modality, thereby establishing a precise cross-modal correspondence. Let the RGB feature sequence and the purified Depth feature sequence after the aforementioned MEM be respectively... ,in, The number of spatial locations (49). express OK, The linear space of columns.

[0043] Before cross-modal interaction, the original spatial location information of the query and key-value features is explicitly injected. , in, For the modal features of the query, Modal features of key values, , These are the corresponding, identical spatial location encoding matrices based on the sine function.

[0044] For the Each attention head is projected using an independent parameter matrix: , , in, This is the learnable parameter matrix.

[0045] Calculate the first The output of each attention head To achieve multi-head attention capturing of key areas, as follows:

[0046] Integrate cross-modal association information learned by all attention heads from different subspaces.

[0047] The outputs of all attention heads are concatenated and linearly fused to achieve cross-modal feature fusion, resulting in the final output of multi-head cross-attention: , in, ， This is the output of multi-head cross-attention.

[0048] Next, the residual connections, layer normalization, and bottleneck excitation feedforward network (BEF) are recalibrated. This step is similar to MEM (Modal Reinforcement Network), integrating residual connections, layer normalization (LN), and bottleneck excitation feedforward network (BEF) to form a stable processing unit. , , in, These are enhanced features obtained after a unidirectional interaction (e.g., RGB→Depth). The first line of the formula represents: Input features First, through multi-head cross-attention, then with input The residuals are summed, and then layer normalization is performed to obtain intermediate features. The second line of the formula indicates: [to...] The input bottleneck-excited feedforward network (BEF) is then fed into the output network. The residuals are summed, and then subjected to layer normalization to obtain the final output. .

[0049] Next, calculate the cross-attention in both directions: Using the purified Depth feature sequence as the query and the purified RGB feature sequence as the key, the above steps are performed to obtain... The purified RGB feature sequence is used as the query, and the purified Depth feature sequence is used as the key. The above MIM (Modal Interaction Network) steps are then performed to obtain... The results from the two directions are then added element by element and merged:

[0050] The element-wise addition fusion method can equally preserve the complementary information transmitted by the two interaction paths, and unlike the concatenation operation, it does not increase the feature dimension, making it an efficient and stable fusion strategy. This is the final output of a cross-modal fusion layer, which is rich in purified information from both modalities and the deep semantic association between them.

[0051] Furthermore, such as Figure 4 As shown, the bottleneck-excited feedforward network (BEF) first aggregates the number of spatial locations through global average pooling (GAP). Generate channel descriptors from global information on the channel: , in, For the first Channel descriptors for each channel, For the input features in the th The spatial location, the first The characteristic values ​​of each channel.

[0052] The channel descriptors are transformed by bottleneck feature compression dimensionality reduction, nonlinear activation, and dynamic channel excitation dimensionality increase to learn complex nonlinear relationships between channels with a low parameter count. , , in, for Intermediate features after dimensionality reduction of each channel For the first Recovery features after dimensionality upgrade of each channel , These are the learnable reduced-dimensional matrix and the increased-dimensional matrix, respectively. It is a non-linear activation function. To achieve the dimensionality reduction ratio, it is typically set to 8. express OK, The linear space of columns.

[0053] The Sigmoid function is used to map the transformed features to a range of 0 to 1, resulting in adaptive channel activation weights for each channel. : ， in, The size directly reflects The importance of each channel to the current task.

[0054] The learned adaptive channel activation weights are multiplied with the input features channel by channel (element by element) to complete the adaptive recalibration: ， in, This is an element-wise multiplication operation. For the recalibrated features, For input features, Adaptive channel excitation weights for all channels.

[0055] Step S5: Perform global feature aggregation on the fused feature sequence to obtain video-level global features, and input them into the classifier to output the corresponding action recognition result; The fused feature sequence is obtained after processing through a 4-layer cross-modal fusion Transformer network. To perform video-level classification, these features need to be aggregated into a single video-level global feature.

[0056] Specifically, for the fused feature sequence Perform global average pooling to obtain the video-level global feature vector, as follows: , in, This is a global average pooling operation. For the fusion feature sequence of the first row (i.e., the first row) All elements of a spatial location, This is a video-level global feature vector.

[0057] Inputting the video-level global feature vector into a linear classifier (fully connected layer) yields the raw prediction score for the action category, represented as: ， in, The original prediction score. , These are all classifier parameters. The total number of action categories. express OK, The linear space of columns.

[0058] Finally, the scores are converted into a probability distribution using the Softmax function, and the category with the highest probability is taken as the final recognition result. To achieve a high-performance action recognition model, this invention employs an end-to-end joint training strategy. The entire network, including a two-stream convolutional neural network, a one-dimensional temporal convolutional network, a cross-modal fusion Transformer network, and the final classifier, is optimized as a whole. Training uses RGB-D video data pairs containing consecutive frames as training sample units, and supervision signals are applied only to the output of the final fusion classification layer.

[0059] The model training uses the cross-entropy loss function as the optimization objective. This loss function effectively measures the difference between the probability distribution predicted by the model and the true action labels, guiding the model parameters to be updated towards correct classification, and is expressed as: , in, The loss value. This refers to the batch size, i.e., the number of samples input into the network at one time. In a preferred embodiment of the invention, it is set to [default value]. =64, The total number of action categories. For the first The true action labels for each training sample are represented using one-hot encoding, where elements corresponding to the true class are 1, and the rest are 0. For the model to the first The training sample belongs to the first... The original predicted scores for each category, It is obtained by normalizing the original predicted score using the Softmax function, and belongs to the [missing information]. The predicted probabilities of each category, The summation index for the denominator of the Softmax function. express A linear space.

[0060] Furthermore, the weights of the two-stream convolutional neural network (based on ResNet-50) are initialized using model parameters pre-trained on the large ImageNet dataset. This provides the model with powerful prior knowledge of general visual feature extraction, which helps to accelerate convergence and improve performance. The parameters of the rest of the network (such as one-dimensional temporal convolution aggregation, cross-modal fusion Transformer networks, etc.) are initialized using randomized methods suitable for deep learning models.

[0061] A mini-batch stochastic gradient descent optimizer with momentum was used for optimization, with a total of 60 training epochs. The momentum factor was set to 0.9 to smooth the gradient update direction and accelerate the training process.

[0062] Different initial learning rates and decay strategies are adopted based on the characteristics of different parts of the network.

[0063] The two-stream convolutional neural network part: The initial learning rate is set to 0.01. To ensure stable convergence, the learning rate is reduced to 0.1 times its original value (i.e., 0.001) at the 25th epoch.

[0064] Cross-modal fusion Transformer network part: Since this part is trained from scratch, the initial learning rate is also set to 0.01 to align it with the convolutional part. For finer adjustments, the learning rate is reduced to 0.1 times its original value at the 30th and 50th epochs, respectively.

[0065] To prevent overfitting, an early stopping strategy is adopted, taking into account the characteristics of the datasets used in this embodiment of the invention (such as THU-READ). Specifically, instead of directly using the model trained at the end of the 60th epoch, the model trained at the 55th epoch is selected as the final inference model based on its performance on the validation set. This helps to obtain a model with stronger generalization ability.

[0066] During the training phase, various data augmentation techniques were employed on the input RGB frame images to enhance the model's robustness and generalization ability against various disturbances. The augmentation strategies employed included: random horizontal flipping, random cropping, and color jittering (adjusting brightness, contrast, and saturation). These operations increased the diversity of the training data and simulated appearance changes in real-world scenes without altering the semantics of the actions.

[0067] Example 2 Embodiment 2 of the present invention provides a multimodal human motion recognition system based on RGB-D frame images, the system comprising: The preprocessing module is used to acquire RGB-D video data of the robot's first-person perspective actions, extract time-aligned RGB video frame sequences and depth video frame sequences from them, and perform preprocessing. The feature extraction module is used to input the preprocessed RGB video frame sequence and the Depth video frame sequence into a dual-stream convolutional neural network with non-shared parameters, and extract RGB high-dimensional space features and Depth high-dimensional space features respectively. The temporal aggregation module is used to perform channel dimensionality reduction on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively to obtain RGB single-frame features and Depth single-frame features. Then, after one-dimensional temporal convolution aggregation of several consecutive frames, RGB spatiotemporal hybrid features and Depth spatiotemporal hybrid features are generated. Subsequently, they are flattened along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. The cross-modal fusion module is used to inject RGB feature sequences and Depth feature sequences into spatial location encoding, and then input them into the cross-modal fusion Transformer network for feature fusion to obtain a fused feature sequence. In this process, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. The classification module is used to perform global feature aggregation on the fused feature sequence to obtain video-level global features, and input them into the classifier to output the corresponding action recognition results.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0070] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A multimodal human motion recognition method based on RGB-D frame images, characterized in that, The method includes: Collect RGB-D video data of the robot's actions from a first-person perspective, extract time-aligned RGB video frame sequences and Depth video frame sequences, and perform preprocessing. The preprocessed RGB video frame sequence and the Depth video frame sequence are respectively input into a dual-stream convolutional neural network with non-shared parameters to extract RGB high-dimensional space features and Depth high-dimensional space features respectively. Channel dimensionality reduction is performed on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively to obtain RGB single-frame features and Depth single-frame features. These are then aggregated through one-dimensional temporal convolution of several consecutive frames to generate RGB spatiotemporal mixed features and Depth spatiotemporal mixed features. Subsequently, these features are flattened along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. The one-dimensional temporal convolution aggregation step includes: The RGB single-frame features and the Depth single-frame features of N consecutive frames with dimensions of 1 / 4C×H×W are concatenated along the time dimension to obtain RGB feature tensors and Depth feature tensors with dimensions of 1 / 4C×H×W×N. A one-dimensional convolution kernel with a kernel size of M and a stride of 1 is used to perform convolution operations on the RGB feature tensor and the Depth feature tensor along the time dimension, respectively, to output RGB spatiotemporal mixed features and Depth spatiotemporal mixed features with dimensions of 1 / 4C×H×W, where M is a positive integer less than N; The RGB feature sequence and the Depth feature sequence are injected into the spatial location code, and then input into the cross-modal fusion Transformer network for feature fusion to obtain the fused feature sequence. In this process, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. The fused feature sequence is subjected to global feature aggregation to obtain video-level global features, which are then input into a classifier to output the corresponding action recognition results.

2. The multimodal human motion recognition method based on RGB-D frame images according to claim 1, characterized in that, The dual-stream convolutional neural network uses a pre-trained ResNet-50 as its backbone network. By removing the last classification layer and the global average pooling layer, the output of the fourth bottleneck layer is extracted as the high-dimensional spatial feature and the Depth high-dimensional spatial feature. The dimensions of the high-dimensional spatial feature and the Depth high-dimensional spatial feature are both C×H×W, where C, H, and W are the number of channels, height, and width of the feature map, respectively.

3. The multimodal human motion recognition method based on RGB-D frame images according to claim 2, characterized in that, The channel dimensionality reduction reduces the number of channels of the RGB high-dimensional spatial features and the Depth high-dimensional spatial features from C dimensions to 1 / 4C dimensions through a 1×1 convolutional layer, resulting in the RGB single-frame features and the Depth single-frame features with dimensions of 1 / 4C×H×W.

4. The multimodal human motion recognition method based on RGB-D frame images according to claim 1, characterized in that, The spatial location encoding is a location encoding based on a sine function, used to characterize the absolute position of feature points in an H×W spatial grid.

5. The multimodal human motion recognition method based on RGB-D frame images according to claim 1, characterized in that, Based on the multi-head self-attention mechanism, global dependency modeling and feature purification are performed on the RGB feature sequence and the Depth feature sequence within a single modality, respectively, to complete the self-attention calculation within the modality; Based on the multi-head cross-attention mechanism, a first interactive calculation is performed using the purified RGB feature sequence as the query vector and the purified Depth feature sequence as the key and value vector, and a second interactive calculation is performed using the purified Depth feature sequence as the query vector and the purified RGB feature sequence as the key and value vector, to complete the intermodal cross-attention calculation.

6. The multimodal human motion recognition method based on RGB-D frame images according to claim 5, characterized in that, After the intramodal self-attention calculation and after the intermodal cross-attention calculation, a bottleneck-excited feedforward network is connected to perform channel recalibration on the input features.

7. The multimodal human motion recognition method based on RGB-D frame images according to claim 6, characterized in that, The calculation steps of the bottleneck-excited feedforward network include: Global average pooling is performed on the input features to obtain channel descriptors; The channel descriptors are sequentially subjected to dimensionality reduction linear transformation, nonlinear activation, dimensionality increase linear transformation, and Sigmoid activation to generate channel weights; The channel weights are multiplied channel by channel by the input features.

8. The multimodal human motion recognition method based on RGB-D frame images according to claim 1, characterized in that, The steps of performing global feature aggregation on the fused feature sequence to obtain video-level global features, inputting these features into a classifier, and outputting the corresponding action recognition results specifically include: Global average pooling is performed on the fused feature sequence to obtain a video-level global feature vector; The video-level global feature vector is input into a linear classifier, and after being normalized by the Softmax function, the action class probability is output.

9. A multimodal human motion recognition system based on RGB-D frame images, characterized in that, The system is used to implement the multimodal human motion recognition method based on RGB-D frame images as described in any one of claims 1 to 8, and the system includes: The preprocessing module is used to acquire RGB-D video data of the robot's first-person perspective actions, extract time-aligned RGB video frame sequences and depth video frame sequences from them, and perform preprocessing. The feature extraction module is used to input the preprocessed RGB video frame sequence and the Depth video frame sequence into a dual-stream convolutional neural network with non-shared parameters, and extract RGB high-dimensional space features and Depth high-dimensional space features respectively. The temporal aggregation module is used to perform channel dimensionality reduction on the RGB high-dimensional spatial features and the Depth high-dimensional spatial features respectively, to obtain RGB single-frame features and Depth single-frame features. These are then aggregated through one-dimensional temporal convolution on several consecutive frames to generate RGB spatiotemporal hybrid features and Depth spatiotemporal hybrid features. Subsequently, these features are flattened along the spatial dimension to obtain RGB feature sequences and Depth feature sequences. The one-dimensional temporal convolution aggregation step includes: The RGB single-frame features and the Depth single-frame features of N consecutive frames with dimensions of 1 / 4C×H×W are concatenated along the time dimension to obtain RGB feature tensors and Depth feature tensors with dimensions of 1 / 4C×H×W×N. A one-dimensional convolution kernel with a kernel size of M and a stride of 1 is used to perform convolution operations on the RGB feature tensor and the Depth feature tensor along the time dimension, respectively, to output RGB spatiotemporal mixed features and Depth spatiotemporal mixed features with dimensions of 1 / 4C×H×W, where M is a positive integer less than N; The cross-modal fusion module is used to inject RGB feature sequences and Depth feature sequences into spatial location encoding, and then input them into the cross-modal fusion Transformer network for feature fusion to obtain a fused feature sequence. In this process, each fusion layer of the cross-modal fusion Transformer network sequentially performs intra-modal self-attention calculation and inter-modal cross-attention calculation. The classification module is used to perform global feature aggregation on the fused feature sequence to obtain video-level global features, and input them into the classifier to output the corresponding action recognition results.

Images