Video action recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a video action recognition method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed action video, detecting and tracking a target object in the action video, and generating a motion track of the target object; screening an optimal motion trail from the motion trails according to a preset screening standard; according to the optimal motion track, performing matting processing on the original motion video to generate a reconstructed video containing the target object; inputting the reconstructed video into a plurality of convolution layers for feature extraction, wherein each convolution layer comprises a convolution structure and an attention mechanism module; and fusing the features extracted by each convolution layer, carrying out space-time down-sampling on the features extracted by the convolution layers to obtain features after dimension reduction, inputting the features after dimension reduction into the full connection layer, and outputting the motion category of the target object. According to the method, the calculation amount and memory occupation can be reduced, the spatial-temporal feature extraction capability of the model is improved, and the accuracy of action recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of video recognition technology, and in particular to a video action recognition method, device, electronic device and storage medium. Background Art

[0002] At present, video action recognition technology based on convolutional neural network (CNN) has made significant progress and is widely used in fields such as autonomous driving and surveillance. Since video action recognition not only needs to extract the spatial information of a single frame, but also needs to extract the temporal correlation between multiple frames, traditional methods mainly use 3D convolution to capture the spatiotemporal characteristics of video data. Although 3D convolution can extract the spatiotemporal information of actions well, due to its high computational complexity and large number of parameters, the model computational cost and memory usage are very high, making it difficult to effectively apply in resource-constrained scenarios (such as embedded devices or edge devices).

[0003] In order to solve the problem of large amount of calculation, several improvement schemes have been proposed in the prior art. For example, 3D convolution is combined with lightweight models (such as MobileNetV2) to reduce the computational burden and is suitable for scenarios with limited resources. In addition, another improvement method is based on the residual structure and uses the P3D model, which combines 3D convolution and residual network, so that the network depth is improved while the number of parameters is reduced.

[0004] However, the existing improvement schemes still have certain limitations. The computational complexity of 3D convolution is still high, and the scalability of the network model is limited. In practical applications, how to further reduce the amount of computation and parameters while maintaining the ability to extract spatiotemporal features is still a technical problem that needs to be solved. For resource-constrained scenarios, there is an urgent need for a solution that can maintain high action recognition accuracy while reducing computational costs. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a video action recognition method, device, electronic device and storage medium to solve the problems of large computational complexity, high memory usage, poor spatiotemporal feature extraction capability and reduced action recognition accuracy existing in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a video action recognition method is provided, comprising: obtaining an action video to be processed, detecting and tracking a target object in the action video, and generating a motion trajectory of the target object; selecting an optimal motion trajectory from the motion trajectory according to a preset screening criterion; performing a cutout process on the original action video according to the optimal motion trajectory to generate a reconstructed video containing the target object; inputting the reconstructed video into multiple convolutional layers for feature extraction, wherein the convolutional layer comprises a convolutional structure and an attention mechanism module, wherein the convolutional structure splits a three-dimensional convolution into two convolution operations, which are respectively used to extract spatial features and temporal features, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract the motion features of the target object; fusing the features extracted by each convolutional layer, and performing spatiotemporal downsampling on the features extracted by the convolutional layer to obtain reduced-dimensional features, inputting the reduced-dimensional features into a fully connected layer, and outputting the action category of the target object.

[0007] According to a second aspect of an embodiment of the present application, a video action recognition device is provided, comprising: a detection module, configured to obtain an action video to be processed, detect and track a target object in the action video, and generate a motion trajectory of the target object; a screening module, configured to screen out an optimal motion trajectory from the motion trajectory according to a preset screening standard; a generation module, configured to perform a cutout process on the original action video according to the optimal motion trajectory, and generate a reconstructed video containing the target object; an extraction module, configured to input the reconstructed video into multiple convolutional layers for feature extraction, the convolutional layer comprising a convolutional structure and an attention mechanism module, wherein the convolutional structure splits a three-dimensional convolution into two convolution operations, which are respectively used to extract spatial features and temporal features, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract the motion features of the target object; an output module, configured to fuse the features extracted by each convolutional layer, and perform spatiotemporal downsampling on the features extracted by the convolutional layer to obtain reduced-dimensional features, input the reduced-dimensional features into a fully connected layer, and output the action category of the target object.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0011] By acquiring the action video to be processed, detecting and tracking the target object in the action video, and generating the motion trajectory of the target object; according to the preset screening criteria, selecting an optimal motion trajectory from the motion trajectory; according to the optimal motion trajectory, performing cutout processing on the original action video, and generating a reconstructed video containing the target object; inputting the reconstructed video into multiple convolutional layers for feature extraction, the convolutional layer includes a convolutional structure and an attention mechanism module, wherein the convolutional structure splits the three-dimensional convolution into two convolution operations, which are used to extract spatial features and temporal features respectively, and the attention mechanism module is used to weight the features extracted by the convolutional structure in order to extract the motion features of the target object; the features extracted by each convolutional layer are fused, and the features extracted by the convolutional layer are downsampled in time and space to obtain the features after dimensionality reduction, and the features after dimensionality reduction are input into the fully connected layer, and the action category of the target object is output. The present application can reduce the amount of calculation and memory usage, improve the spatiotemporal feature extraction capability of the model, and improve the accuracy of action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a schematic diagram of the overall implementation process of the video action recognition method based on lightweight 3D convolution provided in an embodiment of the present application;

[0014] Figure 2 It is a flowchart of a video action recognition method provided by an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of the structure of the convolutional layer provided in an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of a convolution structure provided in an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of a 3D depth-separable convolution provided in an embodiment of the present application;

[0018] Figure 6 It is a structural diagram of the attention mechanism module provided in an embodiment of the present application;

[0019] Figure 7 is a structural diagram of a video action recognition device provided in an embodiment of the present application;

[0020] Figure 8It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] Video action recognition is an important field in computer vision, especially in monitoring, autonomous driving and other scenarios. In recent years, technologies based on convolutional neural networks (CNNs) have made significant progress. The difficulty of video action recognition is that it is necessary not only to extract the spatial information of each frame, but also to extract the temporal association through multiple frames, that is, to capture the spatiotemporal features at the same time. To this end, existing methods usually use 3D convolution, which can effectively extract the spatiotemporal features of the action, but it also brings the following problems:

[0023] 1. High computational cost: Since 3D convolution has to deal with the time dimension, the number of parameters is significantly larger than that of 2D convolution, which leads to large model calculation and slow inference speed.

[0024] 2. Large memory usage: 3D convolutional models require a lot of memory and are not suitable for use on resource-constrained devices, such as embedded systems or in-vehicle systems.

[0025] 3. Limited model expansion: Due to the large number of 3D convolution parameters, building deeper networks will further increase the complexity, limiting the scalability and flexibility of the model.

[0026] At present, there are some solutions that attempt to solve the above problems, for example, the introduction of lightweight models, such as MobileNetV2 and other lightweight models combined with 3D convolution, the purpose is to reduce the number of model parameters and calculations while maintaining the ability to extract spatiotemporal features. Another example is the P3D model based on the residual structure. The P3D model uses different residual structures combined with ResNet-152 to maintain a deeper network structure, while having fewer parameters than the traditional C3D model.

[0027] Although these methods have reduced the computational complexity to a certain extent, how to further reduce the amount of computation and parameters based on 3D convolutional models is still an urgent problem that needs to be solved.

[0028] Therefore, the prior art still has the following problems:

[0029] Large amount of computation and high memory usage: Existing 3D convolutional models have a large number of parameters, making them difficult to be effectively applied in resource-constrained scenarios (such as embedded devices or vehicle systems).

[0030] The model’s spatiotemporal feature extraction is insufficient: Existing methods mainly rely on 3D convolution, but when processing spatiotemporal features, the computational efficiency is low and details are easily overlooked.

[0031] It is difficult to ensure accuracy while reducing the amount of calculation: Existing lightweight methods may sacrifice a certain degree of action recognition accuracy while reducing the computational and memory burdens, and it is difficult to reduce the computational complexity while maintaining high accuracy.

[0032] In view of the problems existing in the prior art, in order to improve the spatiotemporal feature extraction effect of the action recognition system while reducing the amount of calculation and the amount of parameters, this application proposes a video action recognition method based on lightweight 3D convolution. In order to further lightweight the convolution structure, this application introduces a (2+1)D convolution structure, and on its basis, improves the 3D convolution therein to a 3D depth-separable convolution to further reduce the amount of calculation. Due to the reduction in the model volume, the recognition accuracy of the model will be reduced. Therefore, this application additionally introduces an attention mechanism module, which focuses on the extraction of motion features for action recognition, which can make up for the shortcomings of spatiotemporal features in characterization, thereby improving the recognition accuracy of the model. In addition, the video preprocessing proposed in this application can extract valid video frames and remove redundant frames, which can greatly reduce the amount of calculation for action recognition. Real-time prediction can be achieved on some edge devices with insufficient computing power.

[0033] The overall implementation process of the video action recognition method based on lightweight 3D convolution of the present application is summarized below with reference to the accompanying drawings. Figure 1 3D convolution-based video action recognition method provided by the embodiment of the present application. Figure 1 As shown, the video action recognition method based on lightweight 3D convolution may specifically include the following steps:

[0034] 1. Input a pre-segmented action video clip.

[0035] 2. Decode a single frame image and feed it into the human body and hand detection model.

[0036] 3. Then match the human body and the hand, and perform human body tracking to generate the trajectory of each ID.

[0037] 4. After the video is processed, select the best trajectory. The more frames with hands in the trajectory, the better the trajectory.

[0038] 5. Based on the selected trajectories, cut out the original video and reconstruct a new video. The reconstructed new video can ensure that only one person's movements are shown, and the most obvious movement trajectory is retained.

[0039] 6. The new video is input to Layer 1-5. Layer 1-5 uses lightweight (2+1)D convolution + attention mechanism modules in parallel to extract spatiotemporal features. The output is subjected to spatiotemporal downsampling, and the temporal and spatial dimensions of the features are halved.

[0040] 7. Layer 6 uses spatiotemporal average pooling, and after passing through the dropout layer, it enters the fully connected layer to obtain the final prediction result of the action category.

[0041] The contents of the technical solution of the present application are described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Figure 2 is a flow chart of the video action recognition method provided by the embodiment of the present application. Figure 2 As shown, the video action recognition method may specifically include:

[0043] S201, obtaining an action video to be processed, detecting and tracking a target object in the action video, and generating a motion trajectory of the target object;

[0044] S202, selecting an optimal motion trajectory from the motion trajectories according to a preset screening criterion;

[0045] S203, performing cutout processing on the original action video according to the optimal motion trajectory to generate a reconstructed video containing the target object;

[0046] S204, inputting the reconstructed video into a plurality of convolutional layers for feature extraction, wherein the convolutional layers include a convolutional structure and an attention mechanism module, wherein the convolutional structure splits the three-dimensional convolution into two convolution operations, which are respectively used to extract spatial features and temporal features, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract motion features of the target object;

[0047] S205, the features extracted by each convolution layer are fused, and the features extracted by the convolution layer are downsampled in time and space to obtain reduced-dimensional features, and the reduced-dimensional features are input into the fully connected layer to output the action category of the target object.

[0048] In some embodiments, obtaining an action video to be processed and detecting and tracking a target object in the action video include:

[0049] Obtain the pre-segmented action video, decode the action video into multiple single-frame images, input the single-frame images into the pre-trained target object detection model, and match the target objects output by the detection model to generate the motion trajectory corresponding to each target object.

[0050] Specifically, a pre-processed action video clip is obtained, and the action video clip has been pre-processed and segmented into segments suitable for action recognition. In this embodiment, the pre-segmented action video clip may be from a monitoring device or other camera device, and includes the action process of a specific target object.

[0051] Furthermore, the system performs a decoding operation on the action video. The decoding process is to decode the continuous action video into multiple single-frame images, which can be more effectively used as input for subsequent target detection. After decoding, the obtained video frames are processed by a pre-trained target object detection model. The target object detection model can include a trained human body detection model and a hand detection model for identifying and locating human body and hand features in the video frame.

[0052] Furthermore, when a single frame is input into the detection model, the model will detect the target object (such as a human body or a hand) in each frame according to the trained data set and generate the detection result of the target object. The detection result includes the position coordinates of the target object in each frame and the corresponding confidence. In order to ensure the continuity of the motion trajectory, the system further matches the detected target objects, that is, based on the detection results, identifies the same target objects that appear in different frames.

[0053] Furthermore, the system performs a tracking operation of the target object based on the matching results. Specifically, the system generates a motion trajectory of each target object according to the position of the detected target object in the continuous frames. In this embodiment, the system generates a corresponding trajectory for each human ID in the video and saves the trajectory data for subsequent processing.

[0054] Through the steps of the above embodiment, the embodiment can effectively obtain the motion trajectory of each target object in the action video, providing reliable basic data for subsequent video processing and action recognition. This method not only improves the accuracy of target object detection, but also ensures the integrity of the motion trajectory through matching and tracking.

[0055] In some embodiments, selecting an optimal trajectory from the motion trajectories according to a preset screening criterion includes:

[0056] The number of frames containing the target object's action in each motion trajectory is counted, and the motion trajectory with the largest number of frames containing the target object's action is taken as the best trajectory.

[0057] Specifically, after video detection and target object tracking are completed, the system generates multiple motion trajectories of the target object. Each motion trajectory corresponds to the position change of the target object detected in the video in consecutive frames, reflecting the movement process of the target object. In order to improve the accuracy of subsequent action recognition, this embodiment screens these motion trajectories and selects the best motion trajectory from them.

[0058] Furthermore, the system counts the number of frames in each motion trajectory, with a focus on the number of frames containing significant movements of the target object in each motion trajectory. Significant movement frames are those frames in which the target object undergoes significant movement changes, usually identified by movement amplitude or other preset movement features.

[0059] Furthermore, the system sorts the motion trajectories according to the number of significant action frames contained in each motion trajectory. In order to ensure that the selected trajectory can reflect the motion process of the target object as comprehensively as possible, the system determines the motion trajectory containing the most significant action frames as the best motion trajectory. The selection criteria of this best trajectory ensure that the trajectory based on which the subsequent motion analysis is based is not only complete, but also preferentially contains the main motion information of the target object.

[0060] Through the screening method in this embodiment, the system can effectively select the trajectory that best reflects the target object's motion characteristics from multiple motion trajectories, reduce redundant information, and provide more accurate and simplified input data for subsequent motion recognition steps. This screening mechanism greatly improves the overall computing efficiency of the system while maintaining the accuracy of motion recognition.

[0061] In some embodiments, the original action video is subjected to a cutout process according to the optimal motion trajectory to generate a reconstructed video containing the target object, including:

[0062] Obtain the image frame where the target object appears in the best motion trajectory;

[0063] According to the contour information of the target object, a cutout operation is performed on the target object in the image frame to remove background information of non-target objects;

[0064] A reconstructed video is generated according to the cutout operation, wherein the reconstructed video contains action information in an optimal motion trajectory of the target object.

[0065] Specifically, the system obtains image frames corresponding to the motion trajectory from the original video based on the best motion trajectory selected in the previous stage. These image frames are key frames where the target object performs significant actions in the video and can effectively reflect the motion characteristics of the target object.

[0066] Furthermore, the system performs a cutout process on each frame of the image based on the contour information of the target object. Specifically, the system uses the position information output by the target detection and tracking model to accurately determine the boundary of the target object. On this basis, the cutout algorithm is used to extract the target object from the image frame, while removing the background information outside the target object. The cutout operation ensures that only the action of the target object is retained in the generated image frame, while other background content or non-related objects are filtered out, thereby simplifying the video content.

[0067] Furthermore, the system reconstructs the processed image frames into a new video. The reconstructed video only contains the significant action information of the target object, and can ensure that there is only one target object in the video, so that the subsequent action analysis is more concise and efficient. The reconstructed video retains all the key action frames of the target object in the optimal motion trajectory, ensuring the complete presentation of the action process, and removing the redundant information and irrelevant background content in the original video.

[0068] Through the method of the above embodiment, the generated reconstructed video can not only simplify the complexity of action recognition, but also improve the accuracy and efficiency of action recognition. In practical applications, this reconstruction method based on cutout processing can effectively reduce unnecessary computing resource consumption, and is particularly suitable for action recognition tasks in resource-constrained environments such as edge devices or mobile devices.

[0069] In some embodiments, this embodiment provides a spatiotemporal feature extraction method based on lightweight convolution and attention mechanism module, which is suitable for processing newly generated videos. The method extracts spatiotemporal features by inputting new videos into Layer 1 to Layer 5, and each layer uses a lightweight (2+1)D convolution structure in parallel with an attention mechanism module. Figure 3 Schematic diagram of the structure of the convolutional layer provided in the embodiment of the present application. Figure 3 As shown, the convolutional layer structure may specifically include the following contents:

[0070] First, the reconstructed video after the cutout process is input to Layer 1. In Layer 1, the system first uses two lightweight (2+1)D convolution structures for serial processing. Each (2+1)D convolution structure decomposes the 3D convolution into two independent convolution operations: a two-dimensional convolution (1×d×d) for extracting spatial features, followed by a one-dimensional convolution (t×1×1) for extracting temporal correlation features. In this way, the convolution layer can maintain the effective extraction of spatiotemporal features while reducing the amount of calculation and parameters.

[0071] Then, based on the output of the two lightweight (2+1)D convolutional structures connected in series, the system combines the output with the parallel attention mechanism module. The attention mechanism module is used to further weight the extracted features, especially focusing on extracting the motion features of the target object. Specifically, the attention mechanism analyzes the spatiotemporal differences between frames and enhances the motion information in the key frames, allowing the system to more accurately capture the motion features of the target object.

[0072] Next, the system fuses the output results of the lightweight convolution branch and the attention mechanism branch. After adding the results of the two branches, the fused result is input into the Relu activation layer, which further enhances the nonlinear expression ability of the features and obtains the final output.

[0073] This process is repeated in Layer 1 to Layer 5, and each layer uses the same lightweight convolution and attention mechanism structure. The output of each layer serves as the input of the next layer, extracting more refined spatiotemporal features layer by layer. In the process of feature extraction, the system can effectively reduce computational overhead while maintaining high requirements for action recognition accuracy.

[0074] Through this parallel lightweight convolution and attention mechanism design, the system can quickly and efficiently extract the spatiotemporal features of the target object from the reconstructed video, thereby providing high-quality feature input for subsequent action classification and recognition.

[0075] In some embodiments, the convolution structure splits the three-dimensional convolution into two convolution operations, which are used to extract spatial features and temporal features respectively, including:

[0076] Performing convolution processing on each frame of the input video data using a two-dimensional convolution operation of a first convolution kernel size to extract spatial features;

[0077] Performing convolution processing on the extracted spatial features in the time dimension using a one-dimensional convolution operation of the second convolution kernel size to extract time-related information;

[0078] Between the two-dimensional convolution operation and the one-dimensional convolution operation, the output of the convolution operation is normalized, and the normalized features are enhanced using an activation function.

[0079] Specifically, the convolution structure adopts a method of splitting the three-dimensional convolution into two convolution operations to extract the spatial features and temporal features of the video data respectively. Figure 4 Schematic diagram of the convolution structure provided in the embodiment of the present application. Figure 4 As shown, the convolution structure may specifically include the following contents:

[0080] First, the system processes the input video data. The dimensions of the input data include Ni Number of channels, T i The number of frames and the size of each frame image is H i ×W i The original 3D convolution layer usually uses a convolution kernel of size t×d×d. Although this 3D convolution can extract spatial and temporal features at the same time, it has a large amount of calculation and a large number of parameters. To optimize this process, this embodiment splits the single-layer 3D convolution.

[0081] The first step is to use two-dimensional convolution to extract spatial features. The convolution kernel size is 1×d×d, and the convolution operation is performed on each frame of the input video data. The function of this two-dimensional convolution kernel is to extract the spatial features of each frame image and retain the spatial information in each frame image, such as the outline and texture of the object. Through this operation, an intermediate result is generated, and the size of the result remains unchanged, which is still N. i ×T i ×H i ×W i .

[0082] In the second step, one-dimensional convolution is used to extract temporal correlation features. The convolution kernel size is t×1×1. This convolution operation acts on the intermediate spatial feature results obtained in the previous step and mainly processes information in the time dimension. By capturing the changes and associations between different frames, the temporal correlation between video frames is extracted to ensure accurate representation of dynamic action features. After this operation, the output size is still N i+1 ×T i ×H i ×W i .

[0083] Between the two convolution operations mentioned above, the system introduces normalization and activation processing. First, batch normalization is performed on the output of each convolution operation, which helps to speed up the convergence of the model and reduce the overfitting problem during training. Then, the Relu activation function is used to enhance the nonlinear expression ability of the convolution result, ensuring that the extracted features can better represent complex action information.

[0084] By splitting a single-layer 3D convolution into (1×d×d) and (t×1×1) convolution structures, the system significantly reduces the number of parameters and computation without changing the dimension of the output feature map. This improved convolution structure improves computational efficiency while ensuring the ability to extract spatiotemporal features, making it very suitable for applications in devices with limited resources, such as edge computing devices or action recognition tasks in embedded systems.

[0085] Combination Figure 4 As shown, the number of input channels is N i , the frame number is T i, the image size is H i ×W i , the number of output channels is N i+1 This design not only simplifies the calculation, but also maintains the accuracy of the output, effectively addressing the computational burden of 3D convolution in action recognition.

[0086] In some embodiments, the method further comprises:

[0087] Perform convolution operation on each channel of the input video data, extract the spatiotemporal features of each channel using a convolution filter with a third convolution kernel size, obtain the convolution features of each channel, and stack the output feature layers of all channels;

[0088] The convolution filter with the fourth convolution kernel size is used to perform point-by-point convolution operation on the stacked feature map, and each channel is fused and reduced in dimension to generate an output feature map.

[0089] Specifically, this embodiment also proposes a feature extraction method based on 3D depth separable convolution to optimize the spatiotemporal feature processing of input video data. This method greatly reduces the amount of calculation and the amount of parameters by combining depth-wise convolution and point-wise convolution, while ensuring the accurate extraction of spatiotemporal features. Figure 5 3D depth separable convolution is a schematic diagram of the embodiment of the present application. Figure 5 As shown, the 3D depth-separable convolution may specifically include the following contents:

[0090] First, the system performs depth-wise convolution on the input video data. The dimension of the input video data is N i ×T i ×H i ×W i , where N i is the number of input channels, T i is the number of video frames, H i ×W i is the spatial size of each frame. Depth-wise convolution uses a convolution filter with the third convolution kernel size, that is, a convolution kernel of 1×t×d×d, to perform convolution operations on each input channel separately. Specifically, the operation mode of depth-wise convolution is:

[0091] A convolution kernel of size 1×t×d×d is applied to each channel. The convolution kernel operates only on the features of this channel and does not share calculations with other channels.

[0092] The convolution operation of each channel extracts its corresponding spatiotemporal features and generates a 1×T i ×H i ×W i The feature map of

[0093] After completing the convolution operation of all channels, the feature maps obtained by each channel are stacked to generate an overall output feature map with a size of N i ×T i ×H i ×W i .

[0094] Through this depth-by-depth convolution method, the system can extract spatiotemporal features for each channel independently, avoiding cross-channel parameter redundancy and effectively reducing the amount of computation.

[0095] Furthermore, after completing the depth-wise convolution and stacking the output feature maps, the system further performs point-wise convolution operations. Point-wise convolution uses a convolution filter with a fourth convolution kernel size, that is, a size of N. i+1 ×1×1×1 convolution kernel. The purpose of this operation is:

[0096] Through the 1×1×1 convolution kernel, the stacked feature map is convolved point by point. The point-by-point convolution fuses and reduces the dimension of the features of all channels. The point-by-point convolution integrates the feature information of different channels to reduce the number of channels and enhance the expressiveness of the features. The size of the final output feature map is N. i+1 ×T i ×H i ×W i , where N i+1 is the number of channels after fusion, and represents the feature vector after dimensionality reduction.

[0097] Through the combined operation of depth-wise convolution and point-by-point convolution, this embodiment achieves an improvement in 3D depth-separable convolution. In depth-wise convolution, the convolution filter operates independently on each channel, ensuring efficient extraction of spatiotemporal features and greatly reducing the amount of parameters; in point-by-point convolution, the convolution filter achieves dimensionality reduction by fusing cross-channel features point by point. This separable convolution structure not only retains the feature extraction capability of 3D convolution, but also significantly reduces the computational cost. It is suitable for processing complex video action recognition tasks, especially in resource-constrained scenarios. While maintaining efficient convolution operations, this embodiment ensures the model's ability to capture spatiotemporal features and improves the accuracy and efficiency of overall action recognition.

[0098] In some embodiments, the attention mechanism module is used to weight the features extracted by the convolution structure so as to extract the motion features of the target object, including:

[0099] For the input vector of the first dimension, the channel dimension of the input vector is reduced by using two-dimensional convolution to obtain an output vector of the second dimension;

[0100] Performing a differential operation on two adjacent frames to generate multiple output vectors of the third dimension, filling and splicing the output vectors of the third dimension, and restoring them to output vectors of the fourth dimension;

[0101] Perform spatial pooling on the spliced ​​output vector of the fourth dimension and perform channel expansion to obtain a mask of the fifth dimension;

[0102] The activation function is used to activate the mask to obtain a weighted mask, the weighted mask is used to weight the original input vector, and the weighted features are added to the original input vector to generate an optimized input vector.

[0103] Specifically, this embodiment also provides a feature weighting method based on the attention mechanism module to improve the motion feature extraction capability of the target object. The method performs weighted processing on the features extracted by the convolution structure to enhance the representation capability of key motion information in the video. Figure 6 Schematic diagram of the structure of the attention mechanism module provided in the embodiment of the present application. Figure 6 As shown, the attention mechanism module can specifically include the following contents:

[0104] For the input high-dimensional vector, the dimension is N×T×C×H×W, where N is the batch size, T is the number of time frames, C is the number of channels, and H and W are the spatial dimensions (i.e., the height and width of the image). First, the channel dimension of the input vector is reduced by using a 1×1 two-dimensional convolution to compress the channel with a reduction factor of α, and the output vector dimension is N×T×C / α×H×W. The main function of this step is to reduce the amount of data and reduce the computational complexity while retaining important channel features.

[0105] Furthermore, the system performs a differential operation on the output vectors of two adjacent frames, that is, the difference between adjacent frames in the time dimension is calculated one by one, and (T-1) output vectors with dimensions of N×T×C / α×H×W are obtained. The purpose of the differential operation is to capture the dynamic changes between video frames, that is, the motion information of the target object. After the differential calculation is completed, the result is padded and spliced ​​with a 0 vector to restore the output vector with dimensions of N×T×C / α×H×W.

[0106] Furthermore, a spatial pooling operation is performed on the output vector of the fourth dimension after splicing, which extracts key motion features by reducing the amount of information in the spatial dimension. Subsequently, the system performs channel expansion on the pooled result to generate a mask of dimension N×T×C×1×1. This mask is the core of the subsequent weighted operation, which is used to emphasize motion features that are significant in time and space.

[0107] Furthermore, the generated mask is activated, and the mask value is normalized by the Sigmoid activation function so that the mask output is distributed between [0,1]. The activated mask is used to perform a weighted operation on the input vector. Specifically, the weighted mask is multiplied element by element with the original input vector to achieve weighted processing of each feature. The weighted features can better characterize the motion information of the target object.

[0108] Finally, the weighted features are added element by element to the original input vector to obtain the final optimized features. This fusion operation retains the integrity of the original features, while strengthening the motion characteristics of the target object through the attention mechanism, improving the model's ability to recognize dynamic actions.

[0109] Through the method of the above embodiment, the system can more effectively focus on the significant motion features of the target object in the video, while avoiding excessive attention to irrelevant information. This weighted method based on the attention mechanism module not only improves the accuracy of feature extraction, but also significantly optimizes the subsequent action recognition performance.

[0110] In some embodiments, the features extracted by each convolutional layer are fused, and the features extracted by the convolutional layer are temporally and spatially downsampled, including:

[0111] In the convolution layer, the convolution structure and the attention mechanism module work in parallel to fuse the spatial and temporal features extracted by convolution with the weighted motion features to obtain spatiotemporal feature representation;

[0112] The spatiotemporal features are subjected to spatiotemporal downsampling. In the time dimension, a downsampling operation is applied to halve the time dimension of the feature; in the spatial dimension, a downsampling operation is applied to halve the spatial resolution of the feature.

[0113] Specifically, this embodiment also provides a method for fusing and spatiotemporal downsampling the features extracted by each convolutional layer, so as to improve the computational efficiency of the video action recognition system and ensure accurate extraction of spatiotemporal features. The method may include the following steps:

[0114] First, the system processes the spatiotemporal features extracted by each convolutional layer (Layer1 to Layer5). In each layer, the lightweight convolutional structure and the attention mechanism module work in parallel to extract spatial features and temporal features from the input video frames, respectively, and weight the motion features of the target object through the attention mechanism module. At this point, the spatial and temporal information extracted by the convolutional structure is fused with the weighted motion features to generate a complete spatiotemporal feature representation.

[0115] Next, the extracted spatiotemporal features are subjected to spatiotemporal downsampling. Specifically, the system first downsamples the temporal dimension of the features, halving the resolution of the features in the temporal dimension, which can reduce redundant information between video frames while retaining key temporal correlations. Subsequently, the system downsamples the spatial dimension of the features, halving the spatial resolution of the features, thereby reducing the size of the feature map and further reducing computational complexity.

[0116] Finally, after all convolutional layers are processed, the output downsampled features are input to Layer6, which globally summarizes the spatiotemporal features through spatiotemporal average pooling operations to further compress the spatial and temporal dimensions of the features. Subsequently, the processed features pass through the Dropout layer, which is used to prevent the model from overfitting, especially when there is less training data or the feature dimension is high. Finally, the features processed by Dropout are input to the fully connected layer, and the system predicts the action category based on the feature information and outputs the action category result of the target object.

[0117] Through this layered spatiotemporal feature fusion and downsampling processing, this embodiment can fully utilize the spatiotemporal features of the input video for action recognition while ensuring efficient calculation, thereby ensuring the accuracy and reliability of the prediction results.

[0118] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0119] Figure 7 Schematic diagram of the structure of the video action recognition device provided by the embodiment of the present application. Figure 7 As shown, the video action recognition device includes:

[0120] The detection module 701 is configured to obtain the action video to be processed, detect and track the target object in the action video, and generate a motion trajectory of the target object;

[0121] The screening module 702 is configured to screen out an optimal motion trajectory from the motion trajectories according to a preset screening criterion;

[0122] The generation module 703 is configured to perform a cutout process on the original action video according to the optimal motion trajectory to generate a reconstructed video containing the target object;

[0123] The extraction module 704 is configured to input the reconstructed video into a plurality of convolutional layers for feature extraction, wherein the convolutional layers include a convolutional structure and an attention mechanism module, wherein the convolutional structure splits the three-dimensional convolution into two convolution operations for extracting spatial features and temporal features respectively, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract the motion features of the target object;

[0124] The output module 705 is configured to fuse the features extracted by each convolution layer, and perform spatiotemporal downsampling on the features extracted by the convolution layer to obtain reduced-dimensional features, input the reduced-dimensional features into the fully connected layer, and output the action category of the target object.

[0125] In some embodiments, Figure 7 The detection module 701 obtains the pre-segmented action video, decodes the action video into multiple single-frame images, inputs the single-frame images into a pre-trained detection model of the target object, matches the target object output by the detection model, and generates a motion trajectory corresponding to each target object.

[0126] In some embodiments, Figure 7 The generation module 703 obtains the image frame where the target object appears in the optimal motion trajectory; performs a cutout operation on the target object in the image frame according to the contour information of the target object to remove the background information of the non-target object; generates a reconstructed video according to the cutout operation, wherein the reconstructed video includes the action information in the optimal motion trajectory of the target object.

[0127] In some embodiments, Figure 7 The extraction module 704 performs convolution processing on each frame of the input video data using a two-dimensional convolution operation of a first convolution kernel size to extract spatial features; performs convolution processing on the extracted spatial features in the time dimension using a one-dimensional convolution operation of a second convolution kernel size to extract time-related information; between the two-dimensional convolution operation and the one-dimensional convolution operation, the output of the convolution operation is normalized, and the normalized features are enhanced using an activation function.

[0128] In some embodiments, Figure 7 The extraction module 704 performs convolution operation on each channel of the input video data respectively, extracts the spatiotemporal features of each channel using a convolution filter of a third convolution kernel size, obtains the convolution features of each channel, and stacks the output feature layers of all channels; performs point-by-point convolution operation on the stacked feature map using a convolution filter of a fourth convolution kernel size, fuses and reduces the dimensions of each channel, and generates an output feature map.

[0129] In some embodiments, Figure 7The extraction module 704 uses a two-dimensional convolution to reduce the channel dimension of the input vector of the first dimension to obtain an output vector of the second dimension; performs a differential operation on two adjacent frames to generate multiple output vectors of the third dimension, fills and splices the output vectors of the third dimension, and restores them to output vectors of the fourth dimension; performs a spatial pooling operation on the spliced ​​output vectors of the fourth dimension, and performs channel expansion processing to obtain a mask of the fifth dimension; activates the mask using an activation function to obtain a weighted mask, uses the weighted mask to weight the original input vector, and adds the weighted features to the original input vector to generate an optimized input vector.

[0130] In some embodiments, Figure 7 The output module 705 uses the convolution structure and the attention mechanism module in parallel in the convolution layer to fuse the spatial features and temporal features extracted by the convolution with the weighted motion features to obtain the spatiotemporal feature representation; the spatiotemporal features are subjected to spatiotemporal downsampling. In the time dimension, a downsampling operation is applied to halve the time dimension of the features; in the spatial dimension, a downsampling operation is applied to halve the spatial resolution of the features.

[0131] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0132] Figure 8 Schematic diagram of the structure of the electronic device 8 provided in the embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 801 executes the computer program 803, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0133] Exemplarily, the computer program 803 may be divided into one or more modules / units, which are stored in the memory 802 and executed by the processor 801 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 803 in the electronic device 8.

[0134] The electronic device 8 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 8 may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will appreciate that Figure 8It is only an example of the electronic device 8 and does not constitute a limitation of the electronic device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0135] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0136] The memory 802 may be an internal storage unit of the electronic device 8, for example, a hard disk or memory of the electronic device 8. The memory 802 may also be an external storage device of the electronic device 8, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 8. Further, the memory 802 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 802 is used to store computer programs and other programs and data required by the electronic device. The memory 802 may also be used to temporarily store data that has been output or is to be output.

[0137] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0138] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0143] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0144] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A video action recognition method, characterized in that: include: Acquire an action video to be processed, detect and track a target object in the action video, and generate a motion trajectory of the target object; According to a preset screening criterion, selecting an optimal motion trajectory from the motion trajectories; According to the optimal motion trajectory, the original action video is subjected to a cutout process to generate a reconstructed video containing the target object; Inputting the reconstructed video into a plurality of convolutional layers for feature extraction, wherein the convolutional layers include a convolutional structure and an attention mechanism module, wherein the convolutional structure splits a three-dimensional convolution into two convolution operations for extracting spatial features and temporal features respectively, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract motion features of the target object; The features extracted by each convolutional layer are fused, and the features extracted by the convolutional layer are down-sampled in time and space to obtain reduced-dimensional features, and the reduced-dimensional features are input into the fully connected layer to output the action category of the target object.

2. The method according to claim 1, characterized in that The step of obtaining an action video to be processed and detecting and tracking a target object in the action video includes: Obtain a pre-segmented action video, decode the action video into multiple single-frame images, input the single-frame images into a pre-trained detection model of the target object, and match the target objects output by the detection model to generate a motion trajectory corresponding to each target object.

3. The method according to claim 1, characterized in that The step of performing cutout processing on the original action video according to the optimal motion trajectory to generate a reconstructed video containing the target object includes: Acquire an image frame in which a target object appears in the optimal motion trajectory; According to the contour information of the target object, performing a cutout operation on the target object in the image frame to remove background information of non-target objects; A reconstructed video is generated according to the cutout operation, wherein the reconstructed video contains action information in the optimal motion trajectory of the target object.

4. The method according to claim 1, characterized in that: The convolution structure splits the three-dimensional convolution into two convolution operations, which are used to extract spatial features and temporal features respectively, including: Performing convolution processing on each frame of the input video data using a two-dimensional convolution operation of a first convolution kernel size to extract spatial features; Performing convolution processing on the extracted spatial features in the time dimension using a one-dimensional convolution operation of the second convolution kernel size to extract time-related information; Between the two-dimensional convolution operation and the one-dimensional convolution operation, the output of the convolution operation is normalized, and the normalized features are enhanced using an activation function.

5. The method according to claim 4, characterized in that The method further comprises: Perform convolution operation on each channel of the input video data, extract the spatiotemporal features of each channel using a convolution filter with a third convolution kernel size, obtain the convolution features of each channel, and stack the output feature layers of all channels; A convolution filter with a fourth convolution kernel size is used to perform a point-by-point convolution operation on the stacked feature map, and each of the channels is fused and reduced in dimension to generate an output feature map.

6. The method according to claim 1, characterized in that The attention mechanism module is used to weight the features extracted by the convolution structure so as to extract the motion features of the target object, including: For an input vector of a first dimension, using a two-dimensional convolution to reduce the channel dimension of the input vector to obtain an output vector of a second dimension; Performing a differential operation on two adjacent frames to generate a plurality of output vectors of the third dimension, filling and splicing the output vectors of the third dimension, and restoring them to output vectors of the fourth dimension; Perform spatial pooling on the spliced ​​output vector of the fourth dimension and perform channel expansion to obtain a mask of the fifth dimension; The mask is activated by using an activation function to obtain a weighted mask, the original input vector is weighted by using the weighted mask, and the weighted features are added to the original input vector to generate an optimized input vector.

7. The method according to claim 1, characterized in that The fusing of the features extracted by each convolutional layer and performing spatiotemporal downsampling of the features extracted by the convolutional layer includes: In the convolution layer, the convolution structure is used to work in parallel with the attention mechanism module to fuse the spatial features and temporal features extracted by convolution with the weighted motion features to obtain a spatiotemporal feature representation; The spatiotemporal features are subjected to spatiotemporal downsampling processing. In the time dimension, a downsampling operation is applied to reduce the time dimension of the features by half; in the space dimension, a downsampling operation is applied to reduce the spatial resolution of the features by half.

8. A video action recognition device, characterized in that: include: A detection module is configured to obtain an action video to be processed, detect and track a target object in the action video, and generate a motion trajectory of the target object; A screening module is configured to screen out an optimal motion trajectory from the motion trajectories according to a preset screening criterion; A generation module is configured to perform a cutout process on the original action video according to the optimal motion trajectory to generate a reconstructed video containing the target object; An extraction module is configured to input the reconstructed video into a plurality of convolutional layers for feature extraction, wherein the convolutional layers include a convolutional structure and an attention mechanism module, wherein the convolutional structure splits a three-dimensional convolution into two convolution operations for extracting spatial features and temporal features respectively, and the attention mechanism module is used to weight the features extracted by the convolutional structure so as to extract motion features of the target object; The output module is configured to fuse the features extracted by each of the convolutional layers, and perform spatiotemporal downsampling on the features extracted by the convolutional layers to obtain reduced-dimensional features, input the reduced-dimensional features into the fully connected layer, and output the action category of the target object.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images