Adaptive mask dual-channel behavior recognition method, device and storage device

Abstract

The application provides a kind of self-adapting mask dual-channel behavior identification method, equipment and storage device, it is related to video behavior identification field, this method includes: using SlowFast network model, respectively from video frame extraction fast channel feature and slow channel feature;And dynamically adjust the mask rate of video frame, carry out pseudo-random mask, obtain unmasked and masked feature block;Masked feature block is given mask token, for unmasked feature block, apply self-attention mechanism to calculate, obtain attention feature matrix Z;Attention feature matrix and mask token are input into fully connected layer, and output feature reconstruction matrix;According to the feature reconstruction matrix of two channels, realize the information fusion of channel by transverse connection, finally through Softmax classifier carries out behavior identification.The beneficial effects of the application are: improve the behavior recognition ability of model in complex scene, improve the behavior prediction confidence, reduce the calculation complexity of original algorithm.

Description

Technical Field

[0001] This invention relates to the field of video behavior recognition, and more particularly to a dual-channel behavior recognition method, device, and storage device with adaptive masking. Background Technology

[0002] Due to the complexity of video content and the limitations of a single algorithm in processing video behavior recognition, a single model often struggles to comprehensively capture both dynamic and static information in a video.

[0003] Existing video action recognition methods can be categorized into skeleton-based methods, RGB pixel-based methods, and encoder-based methods based on their implementation principles. Due to the inherent complexity of video action recognition tasks, RGB pixel-based methods cannot fully understand the scene. In real-world applications, skeleton-based algorithms often have high false alarm rates, making them difficult to implement. Encoder-based algorithms, on the other hand, can mask the feature information in the image, thereby improving the model's ability to understand human behavior in videos. For example, RGB pixel-based algorithms can comprehensively learn all the feature information of a video and learn the feature details well, but they are easily limited by external factors such as lighting conditions and environmental complexity. In contrast, encoder-based algorithms do not learn all the feature information of the video like the former, but rather learn more advanced information from the image. The idea is to perform pseudo-random masking on the image, masking the video with a very high masking rate (70%-75%) while minimizing information loss. The algorithm learns more advanced semantic information with less information, thereby enhancing its ability to handle complex scenes. In terms of average accuracy, algorithms based on RGB pixel information are far superior to those based on encoders; and in terms of understanding complex scenes, especially considering the complexity of many real-world scenarios, the latter is far superior to the former in terms of robustness and accuracy.

[0004] Although the SlowFast network and Videomae algorithm provide an advanced technical framework for video action recognition, they still face some significant limitations in practical applications:

[0005] 1) Although the SlowFast network is designed to extract fast and slow features from videos to enhance the accuracy of action recognition, it cannot fully capture all key dynamics in extremely dynamic scenarios, which limits its application in complex environments.

[0006] 2) Videomae employs an innovative masking strategy to predict masked portions of a video, thereby facilitating the model's learning of deeper video content. However, this masking strategy has a degree of randomness, which may lead to the omission of key action information, especially when actions change subtly or rapidly in the video. Furthermore, Videomae's performance heavily relies on the richness and diversity of its training data; therefore, its performance may be unsatisfactory in scenarios with limited or insufficiently diverse data.

[0007] In conclusion, although these technologies each have their unique advantages and application areas, they still have limitations when processing video content in rich real-world scenarios. Summary of the Invention

[0008] To provide a more comprehensive and efficient video behavior recognition solution to adapt to a wider range of application scenarios and requirements, this invention provides an adaptive mask dual-channel behavior recognition method, device, and storage device. An adaptive mask dual-channel behavior recognition method mainly includes:

[0009] S1: Preprocess the input video frame sequence;

[0010] S2: Using the SlowFast network model, fast channel features and slow channel features are extracted from video frames respectively, and then the video information is divided into two channels for separate processing;

[0011] S3: Based on the characteristics of the fast channel and the slow channel, dynamically adjust the masking rate of the video frame, perform pseudo-random masking, and obtain unmasked and masked feature blocks;

[0012] S4: Assign a mask token to the masked feature blocks. For the unmasked feature blocks, apply a self-attention mechanism to calculate and finally obtain the attention feature matrix Z.

[0013] S5: Input the attention feature matrix and mask token into the fully connected layer of the Videomee network, and output the feature reconstruction matrix of two channels;

[0014] S6: Based on the reconstruction matrix of two features, information fusion and behavior recognition are achieved through the horizontal connection of the two channels.

[0015] A storage device that stores instructions and data for implementing a dual-channel behavior recognition method with an adaptive mask.

[0016] An adaptive mask dual-channel behavior recognition device includes: a processor and a storage device; the processor loads and executes instructions and data in the storage device to implement an adaptive mask dual-channel behavior recognition method.

[0017] The beneficial effects of the technical solution provided by this invention are as follows: The SlowFast network in this invention can simultaneously extract fast actions and static background information from videos, ensuring accurate capture of dynamic behaviors; Videomae enhances the model's understanding of the global video content by predicting the video frame content through a mask. This invention reduces the computational complexity of the Videomae algorithm through a dual-channel approach, and further improves the accuracy and robustness of the original algorithm through adaptive pseudo-random masks, reducing the possibility of data loss. This invention alleviates the dependence of the original algorithm on the richness and diversity of training data; in scenarios with limited data or insufficient data diversity, the method proposed in this invention performs better. By integrating the feature learning method of SlowFast and the mask learning mechanism of Videomae, the behavior prediction confidence of the original algorithm is further improved, enhancing the recognition ability and behavior prediction confidence of video behaviors while reducing the computational complexity of the original algorithm, thus possessing broad application prospects and meeting practical application needs. Attached Figure Description

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0019] Figure 1 This is a framework diagram of a dual-channel behavior recognition method with adaptive masking in an embodiment of the present invention.

[0020] Figure 2 This is a flowchart of the dual-channel behavior recognition method in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of feature splicing in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the hardware device working in an embodiment of the present invention. Detailed Implementation

[0023] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] Please refer to Figure 1-2 , Figure 1 This is a framework diagram of a dual-channel behavior recognition method with adaptive masking according to an embodiment of the present invention. Figure 2This is a flowchart of a dual-channel behavior recognition method. This invention effectively combines the advantages of the SlowFast network's dual-speed processing characteristics and the Videomae algorithm's masking autoencoder feature, thus improving the overall performance of video behavior recognition. This technological fusion enables the invention to maintain high accuracy and robustness when processing videos with complex backgrounds and rapid motion changes. This invention is particularly suitable for tasks requiring advanced visual analysis. In the field of public safety, such as surveillance video analysis, this invention can identify abnormal behavior or emergency events in real time, greatly improving response speed and prevention capabilities. In traffic monitoring systems, through accurate behavior recognition, this invention can effectively monitor and manage vehicle and pedestrian traffic on roads. In summary, this invention has broad application prospects and meets practical application needs.

[0025] The method specifically includes:

[0026] S1: The input video data needs to be converted into a format suitable for processing. The video resolution may need to be adjusted to meet the model's input requirements. Common resolution adjustment methods include downsampling, such as downsampling the original video's 1920×1080 resolution to 256×256.

[0027] S2: After preprocessing the video frames, the SlowFast network model is used to extract fast channel features and slow channel features from the video frames. In real-world scenarios, actions often change rapidly, so a higher frame rate is needed for the fast channel. Background information, on the other hand, often changes very little, so a lower frame rate is used in the slow channel to capture its semantic information. Therefore, the final implementation method is: the fast channel focuses on capturing fast actions in the video using a higher frame rate; the slow channel uses a lower frame rate to capture the background and provide contextual information.

[0028] S3: Dynamically adjust the mask rate based on the complexity of the video content (such as the amount of motion and scene changes). This can be automatically determined by analyzing the degree of pixel change in the previous few frames, and the specific degree of change can be expressed by formula (1):

[0029]

[0030] In this context, Mask-Rate (MR) represents the degree of pixel variation, I represents the standard deviation of frame pixel variation, and threshold is a preset threshold value. When the frame pixel variation is greater, the masking rate is smaller, allowing the model to learn more image information; conversely, when the frame pixel variation is smaller, the masking rate is greater, reducing the computational cost of the model while minimizing data loss. Generally, because the fast channel contains human motion information, its masking rate is usually smaller than that of the slow channel, which contains background information and tends to have smaller variations.

[0031] After obtaining the adaptive mask rate, the video frames need to be pseudo-randomly masked based on its value. Pseudo-random masking aims to reduce data loss because if a masked area cannot find an unmasked feature block in the adjacent time domain, or is too far away from an unmasked feature block in the time domain, the risk of information loss increases. Therefore, the method for implementing pseudo-random masking is to avoid masking the same feature block that is close in the time domain.

[0032] After obtaining the adaptive mask rate and performing pseudo-random masking, we obtain the unmasked and masked feature blocks.

[0033] S4: After obtaining the unmasked and masked feature blocks, a mask token is assigned to the masked part to replace it, and its mathematical expression is the value; for the unmasked feature blocks (i.e. the non-masked regions in the feature matrix), the self-attention mechanism is applied to calculate and finally obtain the attention feature matrix Z.

[0034] The calculation of self-attention mainly consists of three steps: calculating the query (Q), key (K), and value (V). First, Q, K, and V are generated according to formula (2):

[0035] Q = W Q X,K=W K X,V=W V X(2)

[0036] Where X is the input feature matrix, W Q W K W V This is a learnable weight matrix. Then, these three values ​​are used to calculate the attention weights, as shown in formula (3):

[0037]

[0038] Where, d k QK is the dimension of the key vector, used to scale the result of the dot product to prevent the gradient vanishing problem. T The similarity between the query and the key is calculated, and the softmax function ensures that the sum of the weights in the output is 1. The attention weights are then applied to obtain the output, as shown in Equation (4):

[0039] Z = AV(4)

[0040] Here, the attention weights A are multiplied by the value matrix V to generate the final output matrix Z, which contains a weighted representation of the input data, resulting in a feature matrix of the unmasked portion. Encoding a small portion of information to obtain high-level features allows the model to capture the dependencies between different feature blocks, strengthening the internal correlation and discriminative power of key information.

[0041] The occluded portion is reconstructed using the output of a self-attention mechanism, and the content of the occluded region is inferred based on the contextual information of the unoccluded portion. This step improves the model's ability to predict the content of the occluded region, contributing to the complete recovery of the feature representation of the video frame.

[0042] S5: In the feature reconstruction step, the self-attention feature matrix and mask token obtained in the previous step are used. They are input into the fully connected layer, and the final feature reconstruction matrix is ​​output. Since the video frame is divided into two channels, two feature reconstruction matrices will be obtained here, which will prepare for the next lateral connection.

[0043] S6: Since the video frame is divided into two channels in S2, namely action information and background information, and their information is not interconnected in the intermediate processing, in order to achieve the final behavior prediction, they need to be horizontally connected to achieve feature fusion. The strategy of fusing from the fast channel to the slow channel is adopted, and the feature shapes of the slow channel and the fast channel are expressed as shown in formula (5):

[0044]

[0045] Where T represents the time scale; since the fast channel samples more frames per second than the slow channel, α>1; S2 represents the size of the video frame, which is a square of S2 because it is adjusted to 256×256 when downsampling in S1; C represents the number of channels. Since the fast channel processes several times more video frames than the slow channel, β<1 is used here to reduce the computational load, meaning the fast channel has fewer channels.

[0046] Because their temporal steps are inconsistent, they cannot be directly fused. Therefore, a convolution operation is performed on the αT of the fast channel, and a convolution is performed once every α frames, which is finally reshaped into the form shown in formula (6):

[0047] {T,S 2 ,βC}(6)

[0048] At this point, the two channels are synchronized in the time domain. The feature reconstruction matrices of the two channels are then concatenated, such as... Figure 3 As shown. After feature concatenation, the concatenated matrix is ​​input into the Softmax classifier, as shown in Softmax formula (7):

[0049]

[0050] Among them, Z i Z represents the original score of the i-th class. jLet represent the raw score of class j, and K represent the total number of classes. The output of the Softmax function is the probability of each behavior class, with the sum of the probabilities of all classes being 1.

[0051] In this embodiment, the publicly available ava_kinetics dataset is selected to train the behavior recognition model of this invention, which has strong video understanding capabilities and is suitable for complex scenarios. The weights of the trained model are then tested, and the spatiotemporal behavior detection method of this type is quantitatively compared with the method proposed in this invention.

[0052] Quantitative comparison: Two of the most common metrics in this field were selected to evaluate the model performance, namely top-1 (the proportion of the real behavior appearing in the first predicted behavior) and top-5 (the proportion of the real behavior appearing in the first five predicted behaviors), to evaluate the quantitative differences between the phased processing scheme and the method proposed in this invention. The results are shown in Table 1. The higher the value, the better the performance. Bold text represents the optimal metric. mAP represents the average accuracy of behavior prediction.

[0053] Table 1 Quantitative Comparison Results

[0054] SlowFastR50 79.9 94.0 42.0 Swim-Transformer 79.2 94.5 --- TimeSformer 73.1 93.7 36.6 ViT 81.9 94.9 46.2 MViT 82.0 95.4 47.9 Videomae 83.6 95.7 48.3 Ours 85.3 96.3 49.6

[0055] Please see Figure 4 , Figure 4 This is a schematic diagram of the hardware device in operation according to an embodiment of the present invention. The hardware device specifically includes: a dual-channel behavior recognition device 401 with adaptive masking, a processor 402, and a storage device 403.

[0056] An adaptive mask dual-channel behavior recognition device 401: The adaptive mask dual-channel behavior recognition device 401 implements the adaptive mask dual-channel behavior recognition method.

[0057] Processor 402: The processor 402 loads and executes the instructions and data in the storage device 403 to implement the adaptive mask dual-channel behavior recognition method.

[0058] Storage device 403: The storage device 403 stores instructions and data; the storage device 403 is used to implement the adaptive mask dual-channel behavior recognition method.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dual-channel behavior recognition method using adaptive masks, characterized in that: include: S1: Preprocess the input video frame sequence; S2: Using the SlowFast network model, fast channel features and slow channel features are extracted from video frames respectively, and then the video information is divided into two channels for separate processing; S3: Based on the characteristics of the fast channel and the slow channel, dynamically adjust the masking rate of the video frame, perform pseudo-random masking, and obtain unmasked and masked feature blocks; S4: Assign a mask token to the masked feature blocks. For the unmasked feature blocks, apply a self-attention mechanism to calculate and obtain the attention feature matrix Z. S5: Input the attention feature matrix and mask token into the fully connected layer of the Videomee network, and output the feature reconstruction matrix of two channels; S6: Reconstruct the matrix based on two features, and achieve information fusion and behavior recognition through the horizontal connection of the two channels; In S3, the formula for calculating the mask rate of dynamically adjusted video frames is: （1） in, Indicates the degree of pixel change. I It represents the standard deviation of the frame pixel change, and threshold is the preset threshold.

2. The dual-channel behavior recognition method with adaptive mask as described in claim 1, characterized in that: In S1, the preprocessing refers to adjusting the video resolution by downsampling, which reduces the original video resolution from 1920×1080 to 256×256.

3. The dual-channel behavior recognition method with adaptive mask as described in claim 1, characterized in that: In S4, the computation of the self-attention mechanism is divided into three parts: computing the query Q, the key K, and the value V. （2） in, X It is the input feature matrix. W Q 、W K 、W V It is a learnable weight matrix.

4. The dual-channel behavior recognition method with adaptive mask as described in claim 3, characterized in that: In S4, these three values ​​are used to calculate the attention weight A, as shown in formula (3): （3） in, d k This is the dimension of the key vector, used to scale the result of the dot product to prevent the gradient vanishing problem. QK T This represents the similarity between the query and the key, and softmax represents the classifier used to ensure that the sum of the weights of the output is 1. The output is obtained by applying attention weights, as shown in formula (4): (4) Where Z represents the attention feature matrix, A represents the attention weights, and V represents the value matrix.

5. The dual-channel behavior recognition method with adaptive mask as described in claim 1, characterized in that: In S5, the characteristic shapes of the slow channel and the fast channel are shown in formula (5): （5） The final feature reconstruction matrix is: （6） in, T Indicates the time scale, S² represents the size of the video frame, and C represents the number of channels. α >1, β <1.

6. The dual-channel behavior recognition method with adaptive mask as described in claim 1, characterized in that: In S6, the feature reconstruction matrix of the two channels is concatenated, and the concatenated matrix is ​​input into the Softmax classifier, which is shown in formula (7): (7) in, Z i Indicates the first i The original score of the class, Z j Indicates the first j The original score of the class, K represents the total number of classes, and the output of the Softmax function is the probability of each behavior class. The sum of the probabilities of all classes is 1.

7. A storage device, characterized in that: The storage device stores instructions and data to implement the dual-channel behavior recognition method with adaptive mask as described in any one of claims 1 to 6.

8. A dual-channel behavior recognition device with adaptive masking, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the dual-channel behavior recognition method with adaptive mask as described in any one of claims 1 to 6.

Images