Dual-channel behavior identification method and device of adaptive mask, and storage device

By using an adaptive mask dual-channel behavior recognition method, combined with the SlowFast network and the Videomae algorithm, the problem of insufficient accuracy and robustness of existing technologies in complex scenarios is solved, and efficient video behavior recognition results are achieved.

CN119007064AActive Publication Date: 2024-11-22CHINA UNIV OF GEOSCIENCES (WUHAN)

Patent Information

Application Number
CN202411010690.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

Existing video action recognition methods suffer from insufficient accuracy and robustness when dealing with complex scenes. In particular, the SlowFast network cannot fully capture dynamic changes, and Videomae's masking strategy may lead to the omission of key action information. Furthermore, it relies on the richness and diversity of training data.

Method used

A dual-channel behavior recognition method with adaptive masking is adopted, which combines the SlowFast network and the Videomae algorithm. By dynamically adjusting the masking rate and pseudo-random mask, and utilizing self-attention mechanism and fully connected layers for feature reconstruction and fusion, efficient processing of video frames is achieved.

Benefits of technology

It improves the accuracy and robustness of video behavior recognition, reduces computational complexity, minimizes data loss, and enhances the model's ability to understand complex scenes, especially with excellent performance in situations where data is insufficient.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007064A_ABST
    Figure CN119007064A_ABST
Patent Text Reader

Abstract

The invention provides a dual-channel behavior recognition method and device of an adaptive mask and a storage device, and relates to the field of video behavior recognition, and the method comprises the steps: employing a SlowFast network model, and extracting a fast channel feature and a slow channel feature from a video frame; dynamically adjusting the mask rate of the video frame, and performing pseudo-random mask to obtain unmasked and masked feature blocks; assigning mask tokens to the masked feature blocks, and calculating non-masked feature blocks by applying a self-attention mechanism to obtain an attention feature matrix Z; inputting the attention feature matrix and the mask token into a full connection layer, and outputting a feature reconstruction matrix; a matrix is reconstructed according to the features of the two channels, information fusion of the channels is achieved through transverse connection, and finally behavior recognition is conducted through a Softmax classifier. The method has the beneficial effects that the behavior recognition capability of the model in a complex scene is improved, the behavior prediction confidence is improved, and the calculation complexity of an original algorithm is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video behavior recognition, and in particular to an adaptive mask dual-channel behavior recognition method, device and storage device. Background Art

[0002] Due to the complexity of video content and the limitations of a single algorithm in processing video behavior recognition, a single model often finds it difficult to fully capture the dynamic and static information in the video.

[0003] Existing video behavior recognition methods can be divided into methods based on skeleton points, methods based on RGB pixel information, and methods based on encoders according to the implementation principle. Due to the complexity of the principle of video behavior recognition tasks, methods based on RGB pixel information cannot fully understand the scene. In real application scenarios, algorithms based on skeleton points often have a high false alarm rate and are difficult to apply in practice. Encoder-based algorithms can mask the feature information in the image to improve the model's ability to understand human behavior in the video. For example, algorithms based on RGB pixel information can fully learn all the feature information of the video and better learn the feature details of the video, but are easily limited by external factors such as lighting conditions and environmental complexity. Correspondingly, the encoder-based algorithm does not learn all the feature information of the video like the former, but learns part of the image with more advanced information; its idea is to pseudo-randomly mask the image, and mask the video with an extremely high mask rate (70%-75%) while reducing information loss. The algorithm learns more advanced semantic information with less information as much as possible to enhance its ability to handle complex scenes. In terms of average accuracy, the algorithm based on RGB pixel information is far superior to the algorithm based on the encoder; and in terms of the ability to understand complex scenes, especially considering the complexity of many real-life scenes, the latter is far superior to the former in robustness and accuracy.

[0004] Although the SlowFast network and Videomae algorithm provide advanced technical frameworks for video action recognition, they still face some significant limitations in practical applications:

[0005] 1) For the SlowFast network, although its design can extract fast and slow features in videos to enhance the accuracy of action recognition, in scenes with extreme dynamic changes, the SlowFast network cannot fully capture all key dynamics, which limits its application effect in complex environments.

[0006] 2) For Videomae, it predicts the masked part of the video through an innovative masking strategy, thereby promoting the model to learn deep video content. However, the masking strategy of this method has a certain degree of randomness, which may lead to the omission of key action information, especially when the action changes slightly or quickly in the video. In addition, the effect of Videomae depends heavily on the richness and diversity of training data. Therefore, its performance may not be satisfactory in scenarios with less data or insufficient data diversity.

[0007] In summary, although these technologies each have their own unique advantages and application areas, they still have limitations when processing video content in rich real-life scenarios. Summary of the invention

[0008] In order to provide a more comprehensive and efficient video behavior recognition solution to adapt to a wider range of application scenarios and requirements, the present invention provides an adaptive mask dual-channel behavior recognition method, device and storage device. An adaptive mask dual-channel behavior recognition method mainly includes:

[0009] S1: preprocess the input video frame sequence;

[0010] S2: Use the SlowFast network model to extract fast channel features and slow channel features from video frames, and then divide the video information into two channels for separate processing;

[0011] S3: dynamically adjust the mask rate of the video frame according to the fast channel features and the slow channel features, perform pseudo-random masking, and obtain unmasked and masked feature blocks;

[0012] S4: assign mask tokens to the masked feature blocks, apply the self-attention mechanism to calculate the unmasked feature blocks, and finally obtain the attention feature matrix Z;

[0013] S5: Input the attention feature matrix and mask token into the fully connected layer of the Videomae network and output the feature reconstruction matrix of the two channels;

[0014] S6: Reconstruct the matrix based on the two features and realize information fusion and behavior recognition through the lateral connection of dual channels.

[0015] A storage device stores instructions and data for implementing an adaptive mask dual-channel behavior recognition method.

[0016] A dual-channel behavior recognition device with an adaptive mask comprises: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an adaptive mask dual-channel behavior recognition method.

[0017] The technical solution provided by the present invention has the following beneficial effects: the SlowFast network in the present invention can simultaneously extract fast actions and background static information in the video, ensuring accurate capture of dynamic behaviors; Videomae enhances the model's understanding depth of the global content of the video by predicting the video frame content of the mask. The present invention reduces the computational complexity of the Videomae algorithm through a dual-channel method, further improves the accuracy and robustness of the original algorithm through an adaptive pseudo-random mask, and reduces the possibility of data loss. The present invention can alleviate the dependence of the original algorithm on the richness and diversity of training data. In scenarios with less data or insufficient data diversity, the method proposed by the present invention has better performance. By integrating the feature learning method of slowfast and the mask learning mechanism of Videomae, the behavior prediction confidence of the original algorithm is further improved, the recognition ability of video behaviors and the behavior prediction confidence are improved, and the computational complexity of the original algorithm is reduced, which has broad application prospects and practical application needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0019] Figure 1 It is a framework diagram of a dual-channel behavior recognition method with adaptive mask in an embodiment of the present invention.

[0020] Figure 2 4 is a flow chart of a dual-channel behavior recognition method in an embodiment of the present invention.

[0021] Figure 3 Schematic diagram of feature splicing in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the operation of the hardware device in the embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0024] Please refer to Figure 1-2 , Figure 1 is a framework diagram of a dual-channel behavior recognition method with adaptive masking in an embodiment of the present invention, Figure 2This is a flow chart of a dual-channel behavior recognition method. The present invention effectively integrates the advantages of the two technologies by combining the dual-speed processing characteristics of the SlowFast network and the masked self-encoding characteristics of the Videomae algorithm, thereby improving the comprehensive performance of video behavior recognition. This technical fusion enables the present invention to maintain high accuracy and strong robustness when processing videos with complex backgrounds and rapid motion changes. The present invention is particularly suitable for tasks that require advanced visual analysis. In the field of public safety, such as surveillance video analysis, the present invention can identify abnormal behaviors or emergencies in real time, greatly improving response speed and prevention capabilities. In traffic monitoring systems, through accurate behavior recognition, the present invention can effectively monitor and manage the flow of vehicles and pedestrians on the road. In summary, the present invention has broad application prospects and practical application needs.

[0025] The method specifically includes:

[0026] S1: The input video data needs to be converted into a format suitable for processing, and the video resolution may need to be adjusted to meet the input requirements of the model. Common resolution adjustment methods include downsampling, such as downsampling the original video's 1920×1080 resolution to 256×256.

[0027] S2: After preprocessing the video frames, use the SlowFast network model to extract fast channel features and slow channel features from the video frames. In real scenes, actions often change quickly, so they need to be processed at a higher frame rate of the fast channel, while background information often changes very little, so its semantic information is captured at a low frame rate of the slow channel. Therefore, the final implementation method is: the fast channel focuses on capturing fast actions in the video and uses a higher frame rate; the slow channel uses a lower frame rate to capture the background to provide contextual information.

[0028] S3: Dynamically adjust the mask rate according to the complexity of the video content (such as the amount of motion and scene changes). This can be automatically determined by analyzing the degree of pixel changes in the previous frames. The specific changes can be expressed by formula (1):

[0029]

[0030] Among them, Mask-Rate (MR) represents the degree of pixel change, I represents the standard deviation of frame pixel change, and threshold is the preset threshold. When the frame pixel changes more, the masking rate is smaller, so that the model can learn more image information; when the frame pixel changes less, the masking rate is larger, which reduces the amount of model calculation while reducing data loss. In general, since the fast channel contains the movement information of the human body, its masking rate is generally smaller than that of the slow channel. The slow channel contains background information and often changes less.

[0031] After obtaining the adaptive mask rate, the video frame needs to be pseudo-randomly masked according to its value. Pseudo-random masking is to reduce data loss, because if the masked area cannot find an unmasked feature block in the adjacent time domain or is too far away from the unmasked feature block in the time domain, the risk of information loss will increase. Therefore, the implementation method of pseudo-random masking is: do not mask the same feature block that is close in the time domain.

[0032] After obtaining the adaptive masking rate and performing pseudo-random masking, the unmasked and masked feature blocks are obtained.

[0033] S4: After obtaining the unmasked and masked feature blocks, the masked part is replaced by a mask token, whose mathematical expression is the value; for the unmasked feature blocks (i.e., the non-masked area in the feature matrix), the self-attention mechanism is applied for calculation, and finally the attention feature matrix Z is obtained.

[0034] The calculation of self-attention is mainly divided into three steps: calculating the query (Query, Q), key (Key, K) and value (Value, V). First, generate Q, K, V according to formula (2):

[0035] Q=W Q X,K=W K X,V=W V X(2)

[0036] Where X is the input feature matrix, W Q , W K , W V is a learnable weight matrix. Then use these three values ​​to calculate the attention weight, as shown in formula (3):

[0037]

[0038] Among them, d k QK is the dimension of the key vector, used to scale the result of the dot product to prevent the vanishing gradient problem. T The similarity between the query and the key is calculated, and the softmax function ensures that the sum of the output weights is 1. Applying the attention weights yields the output, as shown in formula (4):

[0039] Z=AV(4)

[0040] Here, the attention weight A is multiplied by the value matrix V to generate the final output matrix Z, which contains the weighted representation of the input data and obtains a feature matrix of the unmasked part. Encoding a small amount of information and obtaining high-level features enables the model to capture the dependencies between different feature blocks and strengthen the internal correlation and discrimination of key information.

[0041] The output of the self-attention mechanism is used to reconstruct the occluded part, and the content of the occluded area is inferred based on the contextual information of the unoccluded part. This step improves the model's ability to predict the content of the occluded area and helps to fully restore the feature representation of the video frame.

[0042] S5: In the feature reconstruction step, the self-attention feature matrix and mask token obtained in the previous step are used, which are input into the fully connected layer to output the final feature reconstruction matrix. Since the video frame is divided into two channels, two feature reconstruction matrices will be obtained here in the end, preparing for the next step of lateral connection.

[0043] S6: Since the video frame is divided into two channels in S2, namely action information and background information, and their information is not interoperable in the intermediate processing process, in order to achieve the final behavior prediction, it is necessary to connect them horizontally to achieve feature fusion. The strategy of fusing from the fast channel to the slow channel is adopted, and the feature shape of the slow channel and the feature shape of the fast channel are respectively expressed as shown in formula (5):

[0044]

[0045] Among them, T represents the time scale; since the fast channel samples more frames than the slow channel in the same time, α>1; S2 represents the size of the video frame. Since it is adjusted to 256×256 when downsampling S1, it is a square of S2; C represents the number of channels. Since the video frames processed by the fast channel are several times that of the slow channel, in order to reduce the amount of calculation, β<1 here, that is, the fast channel has fewer channels.

[0046] Since the steps in the time domain are inconsistent and cannot be directly fused, a convolution operation is performed on the fast channel αT, and a convolution is performed every α frames, and finally reshaped into the formula (6):

[0047] {T,S 2 ,βC}(6)

[0048] At this point, the two channels have been synchronized in the time domain, and the feature reconstruction matrices of the two channels are spliced, such as Figure 3 After feature concatenation, the concatenated matrix is ​​input into the Softmax classifier, as shown in Softmax formula (7):

[0049]

[0050] Among them, Z i represents the original score of the i-th category, Z jrepresents the original score of the jth category, and K represents the total number of categories. The output of the Softmax function is the probability of each behavior category, and the sum of the probabilities of all categories is 1.

[0051] In this embodiment, the public ava_kinetics data set is selected to train the behavior recognition model disclosed in the present invention which has strong video understanding ability and is suitable for complex scenes, and the weight of the trained model is tested. In the experiment, this type of spatiotemporal behavior detection method is quantitatively compared with the method proposed in the present invention.

[0052] Quantitative comparison: The two most common indicators in this field are selected to evaluate the model performance, namely top-1 (the proportion of real behaviors appearing in the first predicted behavior) and top-5 (the proportion of real behaviors appearing in the first five predicted behaviors), to evaluate the quantitative indicator differences between the staged processing scheme and the method proposed in this invention. The results are shown in Table 1. The higher the value, the better the performance. The bold represents the optimal indicator. mAP represents the average accuracy of behavior prediction.

[0053] Table 1 Quantitative comparison results

[0054] Model Top-1 Top-5 mAP SlowFastR50 79.9 94.0 42.0 Swim-Transformer 79.2 94.5 --- TimeSformer 73.1 93.7 36.6 ViT 81.9 94.9 46.2 MViT 82.0 95.4 47.9 Videomae 83.6 95.7 48.3 Ours 85.3 96.3 49.6

[0055] See also Figure 4 , Figure 4 40 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically comprises: an adaptive mask dual-channel behavior recognition device 401, a processor 402 and a storage device 403.

[0056] An adaptive masked dual-channel behavior recognition device 401: The adaptive masked dual-channel behavior recognition device 401 implements the adaptive masked dual-channel behavior recognition method.

[0057] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the dual-channel behavior recognition method with an adaptive mask.

[0058] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the dual-channel behavior recognition method with an adaptive mask.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dual-channel behavior recognition method with adaptive masking, Features: include: S1: preprocess the input video frame sequence; S2: Use the SlowFast network model to extract fast channel features and slow channel features from video frames, and then divide the video information into two channels for separate processing; S3: dynamically adjust the mask rate of the video frame according to the fast channel features and the slow channel features, perform pseudo-random masking, and obtain unmasked and masked feature blocks; S4: assign mask tokens to the masked feature blocks, and apply the self-attention mechanism to calculate the unmasked feature blocks to obtain the attention feature matrix Z; S5: Input the attention feature matrix and mask token into the fully connected layer of the Videomae network and output the feature reconstruction matrix of the two channels; S6: Reconstruct the matrix based on the two features and realize information fusion and behavior recognition through the lateral connection of dual channels.

2. The method of dual-channel behavior recognition with adaptive masking as claimed in claim 1, Features: In S1, the preprocessing refers to adjusting the video resolution by downsampling the original video resolution of 1920×1080 to 256×256.

3. The method for dual-channel behavior recognition with adaptive masking as claimed in claim 1, Features: In S3, the calculation formula for dynamically adjusting the mask rate of the video frame is: Among them, Mask-Rate represents the degree of pixel change, I represents the standard deviation of frame pixel change, and threshold is a preset threshold.

4. The method of dual-channel behavior recognition with adaptive masking as claimed in claim 1, Features: In S4, the calculation of the self-attention mechanism is divided into three parts: calculating the query Q, key K and value V: Q=W Q X,K=W K X,V=W V X (2) Among them, X is the input feature matrix, W Q , W K , W V is a learnable weight matrix.

5. The method of dual-channel behavior recognition with adaptive masking as claimed in claim 4, Features: In S4, these three values ​​are used to calculate the attention weight A, as shown in formula (3): Among them, d k is the dimension of the key vector, used to scale the result of the dot product to prevent the gradient vanishing problem, QK T represents the similarity between the query and the key, and softmax represents the classifier, which is used to ensure that the sum of the output weights is 1; Applying the attention weights yields the output as shown in formula (4): Z=AV (4) Among them, Z represents the attention feature matrix, A represents the attention weight, and V represents the value matrix.

6. The method of dual-channel behavior recognition with adaptive masking as claimed in claim 1, Features: In S5, the characteristic shapes of the slow channel and the fast channel are respectively shown in formula (5): {T,S 2 ,C}{αT,S 2 ,βC}(5) The final feature reconstruction matrix is: {T,S 2 ,βC}(6)Where T represents the time scale, S2 represents the size of the video frame, C represents the number of channels, α>1, β<1.

7. The method of dual-channel behavior recognition with adaptive masking as claimed in claim 1, Features: In S6, the dual-channel feature reconstruction matrix is ​​concatenated, and the concatenated matrix is ​​input into the Softmax classifier. The Softmax classifier is shown in formula (7): Among them, Z i represents the original score of the i-th category, Z j represents the original score of the jth category, K represents the total number of categories, and the output of the Softmax function is the probability of each behavior category, and the sum of the probabilities of all categories is 1.

8. A storage device, Features: The storage device stores instructions and data for implementing the adaptive mask dual-channel behavior recognition method according to any one of claims 1 to 7.

9. A dual-channel behavior recognition device with adaptive masking, Features: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the dual-channel behavior recognition method with adaptive mask as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Taking identification method and device based on double-channel cross attention mechanism

    CN113936339A

  • Combined action space-time detection method based on interaction object motion enhancement

    CN115100737A

  • Audiovisual speech recognition method and device based on self-supervised learning, and storage medium

    CN118248147A

  • Person Re-Identification Method Combining Random Batch Mask and Multi-Scale Representation Learning

    JP6830707B1

  • Method and system for generating adversarial attacks on video recognition models

    US11776241B1

Cited By

  • Civil aviation pilot abnormal behavior identification method based on diffusion large model fine tuning

    CN119625844A

  • Abnormal behavior recognition method for civil aviation pilots based on fine-tuning of diffusion large model

    CN119625844B