A spatiotemporal perception-based multi-modal action recognition method

Abstract

This invention discloses a spatiotemporally aware multimodal action recognition method, relating to the field of computer vision. The method includes: acquiring an original RGB video stream and generating a corresponding depth video stream using a monocular depth estimation model; outputting high-level features from different paths through a multimodal feature extraction network; wherein the multimodal feature extraction network includes a Slow path, a Fast path, and a Depth path, and is equipped with a cross-modal feature fusion module for aligning and fusing intermediate features extracted from the three paths at multiple preset stages; performing global spatiotemporal pooling and concatenation on the high-level features from different paths to generate global multimodal fused features; and inputting the global multimodal fused features into a classifier to obtain the prediction result of the action category. By fusing RGB appearance, temporal motion, and algorithm-generated deep structural information, and performing multi-level interactive fusion within the network, this method improves the spatial understanding ability and environmental robustness of the action recognition model.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a multimodal action recognition method based on spatiotemporal awareness. Background Technology

[0002] Action recognition, as a core task of computer vision and video understanding, aims to automatically identify and classify specific actions or behaviors from continuous video sequences. This technology has been widely applied in many important fields such as intelligent monitoring, human-computer interaction, and video content analysis.

[0003] Current mainstream action recognition methods rely heavily on appearance and motion information extracted from RGB video frames. Models such as two-stream networks and 3D convolutional networks, as well as the high-performance SlowFast network in recent years, are mainly designed around the RGB modality. However, RGB information is essentially a reflection record of light. In real-world scenarios with low light, overexposure, or complex dynamic lighting, its reliability and stability will drop significantly, leading to significant fluctuations in model performance. This constitutes the primary limitation of existing technologies in terms of environmental adaptability.

[0004] Furthermore, RGB images are projections of the three-dimensional world onto a two-dimensional plane, losing crucial geometric structure and depth information. This makes it difficult for existing models to effectively perceive how actions unfold in real three-dimensional space. For example, they lack the ability to distinguish between actions like sitting down and squatting, which may be similar in RGB color and texture but have fundamental differences in spatial posture and relative positions of body joints. At the same time, when the viewpoint changes or the target is partially occluded, robust inference is difficult to make based solely on RGB information, limiting the application effect of the model in real complex scenes.

[0005] In addition, in order to improve performance, existing technologies have attempted to introduce other modal information, such as optical flow and human skeleton, but they often have shortcomings in the fusion strategy of multimodal information. Common post-decision fusion or simple feature splicing fails to achieve multi-level and deep interaction and complementarity between different modalities in the feature extraction process. This shallow fusion method cannot fully explore the collaborative potential between modalities and is difficult to form a more discriminative unified feature representation, thus restricting the further improvement of model performance.

[0006] Therefore, how to design a spatiotemporally aware multimodal action recognition method to achieve more accurate and robust action recognition in complex real-world environments through deep interaction and collaboration of multimodal features is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a spatiotemporal awareness-based multimodal action recognition method, which aims to solve the technical problems of existing RGB video-based action recognition methods, such as weak three-dimensional spatial understanding, sensitivity to changes in lighting and viewpoint, and difficulty in distinguishing actions with similar geometric structures. By introducing a depth modality generated by a monocular depth estimation model and designing a feature extraction network including Slow, Fast, and Depth paths and a cross-modal fusion mechanism, more accurate and stable action recognition is achieved in complex real-world scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A spatiotemporal awareness-based multimodal action recognition method includes the following steps: S1. Obtain the original RGB video stream and generate the corresponding depth video stream using a monocular depth estimation model; S2. Output high-level features under different paths through a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a Slow path, a Fast path and a Depth path, and is equipped with a cross-modal feature fusion module, which is used to align and fuse the intermediate features extracted from the three paths in multiple preset stages to generate intermediate multimodal fusion features that are input to subsequent network layers. S3. Perform global spatiotemporal pooling and concatenation on high-level features under different paths to generate global multimodal fusion features; S4. Input the global multimodal fusion features into the classifier to obtain the prediction result of the action category.

[0010] Preferably, S1 includes: Acquire RGB video stream including T consecutive frames ; Utilizing a pre-trained monocular depth estimation model for each frame of RGB image Processing is performed to generate the corresponding depth map. To obtain the depth video stream The monocular depth estimation model maintains a fixed weight during the training of the multimodal feature extraction network.

[0011] Preferably, in S2, the multimodal feature extraction network is based on an improved SlowFast network architecture, wherein the Slow path, Fast path and Depth path are set in parallel and are all constructed using 3D convolution; The Slow path is used to achieve a first sampling rate. Process RGB video streams and extract texture color semantic features; The Fast path is used at the second sampling rate. Process RGB video streams to extract temporal motion semantic features; among which... α is a frame rate factor greater than 1; The Depth path is used at a first sampling rate. Process depth video streams to extract spatial structure features.

[0012] Preferably, the input dimension of the Slow path is The input dimension of the Fast path is The input dimension of the Depth path is ; The network structure of the Depth path is the same as that of the Slow path, but it has independent trainable weights; the number of feature channels in the Fast path is set to... β is a channel reduction factor less than 1; , These represent the image height and width, respectively.

[0013] Preferably, in step S2, the cross-modal feature fusion module processing includes: With the intermediate features of the Slow path Based on the dimension, the intermediate features of the Fast path are analyzed. Intermediate features of the Depth path Perform timing interpolation, spatial scaling, and channel adjustment respectively to make it consistent with... The dimensions are consistent; among them, channel adjustment is achieved through 1×1×1 3D convolution; Aligned features , , The fusion process is performed to obtain intermediate multimodal fusion features. .

[0014] Preferably, the cross-modal feature fusion module includes an appearance-motion branch and a structure injection branch, and the fusion process includes: Features are derived through the appearance-motion pathway. With features By performing element-by-element addition, preliminary fusion characteristics are obtained. Appearance-motion interaction features are generated through convolutional layers and activation functions. ; Using structure injection branches to inject features Spatial structure attention weight map is generated by processing through convolutional layers and the Sigmoid activation function. ; Appearance-motion interaction features Spatial Structure Attention Weight Map Perform element-wise multiplication to output intermediate multimodal fusion features. .

[0015] Preferably, S3 includes: Global average pooling is performed on the high-level features output by the Slow, Fast, and Deep paths respectively to obtain the corresponding feature vectors. , , ; The feature vector , , By stitching along the channel dimension, global multimodal fusion features are obtained. .

[0016] Preferably, S4 includes: Global multimodal fusion features The input is fed into a fully connected layer, and then normalized using the Softmax function to obtain the probability distribution. ; The category with the highest probability As the final recognition result output, This represents the predicted probability of the c-th category.

[0017] As can be seen from the above technical solution, compared with the prior art, the technical solution of the present invention has the following beneficial effects: 1. This method introduces a depth video stream generated by a monocular depth estimation model and uses an independent depth path to extract spatial structure features, providing the model with key geometric and depth information. This enables the model to effectively distinguish actions that look similar but have different spatial structures in the traditional RGB mode, and it has stronger robustness to appearance changes caused by changes in viewpoint or object occlusion.

[0018] 2. By constructing a multimodal feature extraction network with three parallel paths (Slow, Fast, and Depth), and using a cross-modal feature fusion module in the middle layer of the network for dynamic feature alignment and interaction, deep fusion of texture, motion, and structural information at multiple abstract levels is achieved. This not only overcomes the inherent defects of a single RGB modality being sensitive to changes in lighting and easily affected by background interference, but also enhances the stability and discriminative power of the overall feature expression through the synergy and complementarity between features.

[0019] 3. The core depth information of the technical solution comes from the algorithmic estimation of the input RGB video stream, rather than relying on expensive dedicated depth sensing equipment. While maintaining high performance, it reduces the requirements for hardware acquisition conditions and system costs, making it easier to apply and deploy in a wide range of real-world scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 A flowchart of a spatiotemporal awareness-based multimodal action recognition method provided in an embodiment of the present invention; Figure 2 This is a diagram of the multimodal feature extraction network framework provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown, this embodiment provides a multimodal action recognition method based on spatiotemporal awareness, including the following steps: S1. Obtain the original RGB video stream and generate the corresponding depth video stream using a monocular depth estimation model; S2. Output high-level features under different paths through a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a Slow path, a Fast path and a Depth path, and is equipped with a cross-modal feature fusion module, which is used to align and fuse the intermediate features extracted from the three paths in multiple preset stages to generate intermediate multimodal fusion features that are input to subsequent network layers. S3. Perform global spatiotemporal pooling and concatenation on high-level features under different paths to generate global multimodal fusion features; S4. Input the global multimodal fusion features into the classifier to obtain the prediction result of the action category.

[0024] This method generates depth information through an algorithm and coordinates it with RGB video streams to construct a three-path parallel, multi-level cross-modal fusion action recognition network. It enhances the model's ability to perceive and distinguish actions in three-dimensional space, effectively improves the recognition robustness under complex lighting, changing viewpoints and background interference, and has stronger practicality and ease of deployment because it does not rely on dedicated depth sensing hardware.

[0025] The following provides further explanation of each step in the above method; In this embodiment, S1, the original RGB video stream is acquired, and a corresponding depth video stream is generated using a monocular depth estimation model; this includes: Acquire RGB video stream including T consecutive frames ; Utilizing a pre-trained monocular depth estimation model for each frame of RGB image Processing is performed to generate the corresponding depth map. To obtain the depth video stream ; Among them, the monocular depth estimation model can be the Depth Anything model, whose weights remain fixed during the training of the multimodal feature extraction network.

[0026] In practical implementation, to ensure the consistency of multimodal data, a monocular depth estimation model is used to generate depth maps. Afterwards, a rigorous data alignment operation must be performed, including aligning the depth map... The size is precisely scaled to match the original RGB frame using bilinear interpolation. The same spatial resolution, and ensure that the two correspond exactly in timestamps; In addition, an online depth refinement module is added after this step, which uses the original RGB frame. and depth map As input, residual maps are learned through a small convolutional network. It is used to correct noise or blurring that may exist in the depth map at object edges and in areas with weak texture. Its purpose is to improve the quality of the depth map input to the Depth path, thereby providing the network with more reliable spatial structure priors and further enhancing the model's ability to utilize geometric information.

[0027] This step replaces expensive depth sensing hardware with algorithms to acquire depth information, successfully upgrading the action recognition task from relying on single RGB appearance information to utilizing RGB-depth dual-modal information, providing the model with the ability to understand the three-dimensional geometry of the scene.

[0028] In this embodiment S2, a multimodal feature extraction network outputs high-level features for different paths; such as... Figure 2 As shown, the multimodal feature extraction network includes a Slow path, a Fast path, and a Depth path, and is equipped with a cross-modal feature fusion module, which is used to align and fuse the intermediate features extracted by the three paths in multiple preset stages to generate intermediate multimodal fusion features that are input to subsequent network layers. The multimodal feature extraction network is based on an improved SlowFast network architecture, in which the Slow path, Fast path and Depth path are set in parallel and are all constructed using 3D convolution. Slow path, used at the first sampling rate Process RGB video streams and extract texture color semantic features; Fast path, used at the second sampling rate Process RGB video streams to extract temporal motion semantic features; among which... α is a frame rate factor greater than 1; Depth path, used at the first sampling rate Process depth video streams to extract spatial structure features.

[0029] Furthermore, the input dimension of the Slow path is The input dimension of the Fast path is The input dimension of the Depth path is ; The network structure of the Depth path is the same as that of the Slow path, but it has independent trainable weights; the number of feature channels in the Fast path is set to... β is a channel reduction factor less than 1; , These represent the image height and width, respectively.

[0030] Although the Slow, Fast, and Deep paths in the multimodal feature extraction network are parallel, their design has clear physical meaning and engineering considerations; the Slow path uses a low frame rate. With smaller temporal convolutional kernels, it aims to capture rich texture and color semantic information in keyframes over larger time intervals. Its network channel count C=256 focuses on spatial details; the Fast path employs a high frame rate. With larger temporal convolution kernels, the aim is to densely sample to capture subtle temporal motion patterns, as described in this embodiment. The number of channels is significantly reduced, which is based on the assumption that motion information is relatively low-level and redundant, thus reducing the amount of computation. The Depth path reuses the architecture of the Slow path to process the depth map. Its independent weights ensure that the network can specifically learn to extract spatial structure features from the depth modality, thus achieving decoupling from appearance feature learning.

[0031] Furthermore, the cross-modal feature fusion module processes the following: With the intermediate features of the Slow path Based on the dimension, the intermediate features of the Fast path are analyzed. Intermediate features of the Depth path Perform timing interpolation, spatial scaling, and channel adjustment respectively to make it consistent with... The dimensions are consistent; among them, channel adjustment is achieved through 1×1×1 3D convolution; Aligned features , , The fusion process is performed to obtain intermediate multimodal fusion features. .

[0032] Furthermore, the cross-modal feature fusion module includes an appearance-motion branch and a structure injection branch, and the fusion process includes: Features are derived through the appearance-motion pathway. With features By performing element-by-element addition, preliminary fusion characteristics are obtained. Appearance-motion interaction features are generated through convolutional layers and activation functions. ; Using structure injection branches to inject features Spatial structure attention weight map is generated by processing through convolutional layers and the Sigmoid activation function. ; Appearance-motion interaction features Spatial Structure Attention Weight Map Perform element-wise multiplication to output intermediate multimodal fusion features. .

[0033] The cross-modal feature fusion module dynamically injects geometric attention into appearance-motion features, enabling the network to focus on regions that are meaningful in three-dimensional space for recognition, rather than simply relying on RGB textures. This module can be flexibly inserted into multiple stages of the network to achieve multi-level fusion from low to high layers.

[0034] The three-path parallel and multi-layered fusion architecture constructed in this step introduces the Depth path, which endows the model with the three-dimensional spatial perception capability that is lacking in the native SlowFast network. The cross-modal fusion module ensures that the information of the three modalities is not simply spliced ​​together in the later stage, but rather interacts and complements each other deeply and dynamically during the feature extraction process. This enables the model to comprehensively utilize appearance, motion and structural information, thereby showing significantly stronger discrimination ability and robustness when facing complex backgrounds and different actions with similar appearances.

[0035] In this embodiment S3, global spatiotemporal pooling and concatenation are performed on high-level features under different paths to generate global multimodal fusion features; including: Global average pooling is performed on the high-level features output by the Slow, Fast, and Deep paths respectively to obtain the corresponding feature vectors. , , ; The feature vector , , By stitching along the channel dimension, global multimodal fusion features are obtained. .

[0036] This step, combined with the aforementioned feature enhancement design, can output highly condensed and discriminative global multimodal fusion features. It integrates global statistical information on appearance details, motion patterns, and spatial structure, providing a comprehensive basis for the final classifier to make accurate decisions.

[0037] In this embodiment, S4, the global multimodal fusion features are input into the classifier to obtain the prediction result of the action category; This step, before inputting the global multimodal fusion features into the classifier, also includes: The global multimodal fusion features are input into a multilayer perceptron, which learns an independent feature transformation matrix for texture color semantic features, temporal motion semantic features, and spatial structure features respectively. The transformed modal features are residually concatenated with the original global multimodal fusion features to obtain enhanced global multimodal fusion features, which are then input into the classifier.

[0038] Furthermore, the input classifier processing includes: Global multimodal fusion features The input is fed into a fully connected layer, and then normalized using the Softmax function to obtain the probability distribution. ; The category with the highest probability As the final recognition result output, This represents the predicted probability of the c-th category.

[0039] Furthermore, in the specific training process, a composite loss function can be used. Optimize the multimodal feature extraction network and classifier:

[0040] in, Cross-entropy loss for action classification, The knowledge distillation loss is used to constrain the Depth path to learn effective structural priors from the monocular depth estimation model. The maximum mean difference loss is used to constrain the features extracted by the Slow and Depth paths to be complementary in the representation space without excessive redundancy. , , The balancing weights for each loss term.

[0041] In model training, a phased strategy can be adopted. First, fix some weights of the Depth Anything model and the Depth path, mainly training the Slow and Fast paths to enable the network to initially establish the spatiotemporal understanding ability of RGB modalities. Then, release all weights and use a composite loss function. Perform end-to-end joint optimization; where cross-entropy loss Dominant classification task; knowledge distillation loss Ensures the Depth path inherits effective geometric priors from a robust pre-trained deep model; Maximum mean difference loss. This prompts the Slow and Depth paths to learn complementary rather than redundant features.

[0042] The spatiotemporal awareness-based multimodal action recognition method in this embodiment combines monocular depth estimation with an improved SlowFast network architecture to construct a recognition framework with three parallel paths and multi-level feature fusion. It introduces geometric information from the data source, realizes dynamic interaction and enhancement of appearance, motion and structure in the middle of the network, and performs adaptive integration of global information at the decision front end. It improves the accuracy and robustness of action recognition in complex real-world scenarios, especially the ability to distinguish geometrically sensitive action categories. At the same time, since it only requires monocular RGB input and does not require additional hardware, it has excellent practicality and broad deployment potential.

[0043] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0044] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal action recognition method based on spatiotemporal awareness, characterized in that, Includes the following steps: S1. Obtain the original RGB video stream and generate the corresponding depth video stream using a monocular depth estimation model; S2. Output high-level features under different paths through a multimodal feature extraction network; wherein, the multimodal feature extraction network includes a Slow path, a Fast path and a Depth path, and is equipped with a cross-modal feature fusion module, which is used to align and fuse the intermediate features extracted from the three paths in multiple preset stages to generate intermediate multimodal fusion features that are input to subsequent network layers. S3. Perform global spatiotemporal pooling and concatenation on high-level features under different paths to generate global multimodal fusion features; S4. Input the global multimodal fusion features into the classifier to obtain the prediction result of the action category.

2. The multimodal action recognition method based on spatiotemporal awareness according to claim 1, characterized in that, S1 includes: Acquire RGB video stream including T consecutive frames ; Utilizing a pre-trained monocular depth estimation model for each frame of RGB image Processing is performed to generate the corresponding depth map. To obtain the depth video stream The monocular depth estimation model maintains a fixed weight during the training of the multimodal feature extraction network.

3. The multimodal action recognition method based on spatiotemporal awareness according to claim 1, characterized in that, In S2, the multimodal feature extraction network is based on an improved SlowFast network architecture, in which the Slow path, Fast path and Depth path are set in parallel and are all constructed using 3D convolution. The Slow path is used to achieve a first sampling rate. Process RGB video streams and extract texture color semantic features; The Fast path is used at the second sampling rate. Process RGB video streams to extract temporal motion semantic features; among which... α is a frame rate factor greater than 1; The Depth path is used at a first sampling rate. Process depth video streams to extract spatial structure features.

4. The multimodal action recognition method based on spatiotemporal awareness according to claim 3, characterized in that, The input dimension of the Slow path is The input dimension of the Fast path is The input dimension of the Depth path is ; The network structure of the Depth path is the same as that of the Slow path, but it has independent trainable weights; the number of feature channels in the Fast path is set to... β is a channel reduction factor less than 1; , These represent the image height and width, respectively.

5. The multimodal action recognition method based on spatiotemporal awareness according to claim 1, characterized in that, In S2, the cross-modal feature fusion module processing includes: With the intermediate features of the Slow path Based on the dimension, the intermediate features of the Fast path are analyzed. Intermediate features of the Depth path Perform timing interpolation, spatial scaling, and channel adjustment respectively to make it consistent with... The dimensions are consistent; among them, channel adjustment is achieved through 1×1×1 3D convolution; Aligned features , , The intermediate multimodal fusion features are obtained by performing fusion processing. .

6. The multimodal action recognition method based on spatiotemporal awareness according to claim 5, characterized in that, The cross-modal feature fusion module includes an appearance-motion branch and a structure injection branch, and the fusion process includes: Features are derived through the appearance-motion pathway. With features By performing element-by-element addition, preliminary fusion characteristics are obtained. Appearance-motion interaction features are generated through convolutional layers and activation functions. ; Using structure injection branches to inject features Spatial structure attention weight map is generated by processing through convolutional layers and the Sigmoid activation function. ; Appearance-motion interaction features Spatial Structure Attention Weight Map Perform element-wise multiplication to output intermediate multimodal fusion features. .

7. The multimodal action recognition method based on spatiotemporal awareness according to claim 1, characterized in that, S3 includes: Global average pooling is performed on the high-level features output by the Slow, Fast, and Deep paths respectively to obtain the corresponding feature vectors. , , ; The feature vector , , By stitching along the channel dimension, global multimodal fusion features are obtained. .

8. The multimodal action recognition method based on spatiotemporal awareness according to claim 1, characterized in that, S4 includes: Global multimodal fusion features The input is fed into a fully connected layer, and then normalized using the Softmax function to obtain the probability distribution. ; The category with the highest probability As the final recognition result output, This represents the predicted probability of the c-th category.

Images