Channel feature interaction time modeling behavior recognition method based on BERT model

Abstract

The application discloses a channel feature interaction time modeling behavior recognition method based on a BERT model, and belongs to the technical fields of computer vision, deep learning and behavior recognition. The motion video is decomposed into corresponding RGB image sequences, which are input into a two-dimensional convolutional neural network; based on the features extracted by the two-dimensional convolutional neural network, self-attention calculation is performed on the sub-channel feature sequences through a channel reorganization module and a channel BERT model, key sub-channel features with large differences changing with time and the interaction correlation therebetween are extracted, key semantic features and the correlation thereof distinguishing motion categories are obtained, and thus the behavior classification precision is improved. Channel BERT and time BERT are combined, the key channel semantic features in key frames are further paid attention to, and thus higher behavior recognition accuracy is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, deep learning and behavior recognition, and in particular relates to a channel feature interaction time modeling behavior recognition method based on a BERT (Bidirectional Encoder Representation from Transformers) model. Background Art

[0002] Action recognition is a fundamental task in computer vision, with widespread applications in human-computer interaction, video retrieval, and intelligent security monitoring. Action recognition technology primarily relies on processing and analyzing video data to enable computers to understand human movements and behaviors. Therefore, the key to action recognition lies in modeling video semantic features along the temporal dimension during action changes. First, feature modeling requires extracting the spatiotemporal relationships between human actions in a video. This modeling of spatiotemporal features can be used to describe the differences in appearance between different human behaviors. Second, the channel relationships between video frames at different times must be considered. Effective channel feature interactions can more completely represent video semantic information. Temporal modeling of channel feature interactions in videos is an effective method for improving the accuracy of action recognition tasks.

[0003] After analysis and experiments, it was found that embedding the BERT self-attention mechanism module in the two-dimensional convolutional network can learn to extract the temporal correlation between image frames and improve the accuracy of action recognition. On this basis, a channel reorganization module is proposed, which separates the channel features of continuous frames into N sub-channels and splices the corresponding sub-channel features of each frame along the time dimension to form a sub-channel feature time series. Then, the channel BERT self-attention mechanism is used to establish a channel feature interaction model for the reconstructed sub-channel group. It extracts the key sub-channel feature sequences with large differences in time changes and the interaction correlation between them based on the similarity calculation between the sub-channel groups, enhances the interaction relationship between the sub-channel group and the adjacent video frames, and obtains the key semantic features for distinguishing action categories. In order to simultaneously extract the interaction correlation of the channel and time dimensions in the frame image features, a joint-BERT model is proposed. By fusing the two branches of extracting channel correlation and extracting time correlation, it can further focus on the key channel semantic features in the key frames and improve the accuracy of action recognition. At the same time, the channel BERT module (70) and the time BERT module (71) adopt a weight sharing strategy to reduce the number of weights of the overall model.

[0004] In recent years, similar research and patents to the BERT-based channel feature interaction time modeling behavior recognition method proposed in this paper are as follows:

[0005] At the 2020 ECCV (European Conference on Computer Vision) conference, Kalfaoglu et al. published an article titled “Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition”, which used BERT to replace the TGAP layer at the end of the 3D CNN to learn important temporal features in the video frames, thereby enhancing the late temporal modeling capabilities of the CNN backbone network. In contrast, the present invention uses 2D CNN to extract the spatial features of the video, and uses BERT to obtain the correlation between the time frames of the video, while also adding a channel BERT to extract the correlation between the channel features of the frame images. First, a channel reorganization module is designed to extract the channel information of all adjacent frames within the sub-channel feature sequence, and then the self-attention mechanism of the channel BERT is used to calculate the similarity of different sub-channel feature sequences to learn the key channel semantic features in the video frames. In addition, the channel feature interactive temporal modeling behavior recognition method based on the BERT model designed by the present invention uses two-dimensional convolution to replace three-dimensional convolution to extract the semantic features of each frame of the image, which greatly saves computational costs, reduces the number of parameters, and improves the computational efficiency of the model.

[0006] Gao, Y. et al. published an article titled “Channel interaction networks for fine-grained image categorization” at the 34th AAAI Conference on Artificial Intelligence in 2020. They proposed a self-channel interaction (SCI) module to establish a correlation model between different channels in an image frame, and a contrast channel interaction (CCI) module to model the cross-sample channel interaction relationship between two image frames. In contrast, the channel reorganization module proposed in the present invention first separates and reorganizes the feature sequences of all image frames extracted by 2D CNN to obtain sub-channel groups containing the channel feature sequences of all adjacent frames, and then uses BERT's self-attention mechanism to perform self-attention calculations on these sub-channel feature sequences, which not only improves the channel interaction ability in an image frame or between two image frames, but also enhances the channel interaction relationship between all input image frames and sub-channel groups, obtaining a more complete channel semantic representation, so that the feature channels of all adjacent time frames of the video, not just the two image frames, can learn the discriminative semantic features in the video, thereby improving the classification accuracy.

[0007] Chinese patent CN111597929A discloses a group behavior recognition method based on channel information fusion and spatial structured modeling of group relationships. This invention addresses the accuracy issues of collective behavior recognition by using the CSTM and CMM modules of a 2D network STM to extract per-frame spatiotemporal information and inter-frame motion information, respectively. A channel selection module is proposed to fuse the spatiotemporal and motion information of each frame. A graph convolution-LSTM network is then used to extract a collective relationship evolution model. To capture long video sequences, the recognition video is sparsely sampled. The present invention inputs the per-frame spatial information extracted by the 2D CNN into the BERT self-attention mechanism to obtain the temporal relationship between inter-frame spatial and channel changes over a period of time. This method of learning sequence features through a multi-head self-attention mechanism not only captures key image frame and key channel semantic features but also improves the long-term temporal modeling capabilities of the behavior recognition model. Furthermore, the present invention eliminates the need to extract a collective relationship evolution model.

[0008] Chinese patent CN113591774A discloses a Transformer-based behavior recognition algorithm. In the pose estimation phase, this invention uses a convolutional layer to simultaneously extract temporal and spatial information of skeleton points from the original video. This fused temporal and spatial information is then directly input into the Transformer self-attention network to capture the changing relationships between human skeleton nodes. This algorithm first uses a 2D CNN to extract spatial information from each frame, and then utilizes the BERT self-attention mechanism to extract spatial and temporal changes for behavior recognition. This approach leverages the advantages of 2D convolution for extracting spatial features and the BERT self-attention mechanism for processing time series to enhance the neural network's behavior modeling capabilities.

[0009] Chinese patent CN113673489A discloses a video group behavior recognition method based on cascade Transformer, which performs human target detection through the first Transformer module to extract individual features of key frames, and models the hierarchical relationship between individuals and groups through the second Transformer module to complete the group behavior recognition task. The BERT module adopted in the present invention is a self-attention model based on the bidirectional Transformer module. The BERT module is used to learn the similarities between sub-channel groups and time frames in branches 1 and 2 respectively to complete the individual behavior recognition task, without the need to extract the hierarchical relationship between individuals and groups. In addition, unlike its focus on individual features of key frames, the present invention adds a weight sharing strategy to the parameter matrix of BERT in the joint-BERT model training, jointly learns the key channel and time features of the human body, thereby fusing information of different dimensions to improve the integrity of the description of human motion features, and further focusing on the key channel semantic features in key frames. Summary of the Invention

[0010] The purpose of the present invention is to address the above-mentioned deficiencies in the existing technology and provide a method for behavior recognition based on the temporal modeling of channel feature interaction using the BERT model, which can improve the accuracy of behavior classification. Combining a 2D convolutional network with the BERT self-attention mechanism, the method specifically studies the extraction of correlations between the temporal and channel dimensions in frame image features. Furthermore, the fusion mechanism of the BERT self-attention model is used to learn key channel semantic features in key frames, optimizing the fusion of image frames in the temporal and channel dimensions to achieve the goal of improving video classification accuracy.

[0011] Based on a two-dimensional convolutional neural network, the present invention proposes a joint-BERT self-attention model that simultaneously extracts key channel and temporal features. The model consists of a first branch that extracts the correlation between channels and a second branch that extracts the temporal correlation between image frames. The two-dimensional convolutional neural network consists of multiple two-dimensional convolutional layers, which perform convolution in the spatial dimension to obtain the spatial features of video image frames. The first branch is used to establish interactions between sub-channels from the semantic features of the reorganized sub-channel group using a multi-head self-attention mechanism to obtain key semantic features and their correlations for distinguishing action categories. The second branch is used to extract key image frames based on the similarity calculation between frames when the image frames at different times are fused using the self-attention mechanism.

[0012] The specific steps of the present invention are as follows:

[0013] 1) Decompose the action video into corresponding RGB image sequences (10), input the sequences into a two-dimensional convolutional neural network module (20) for feature extraction, and obtain feature maps (30) corresponding to B, T, C, H, and W dimensions, where B represents the batch number of input video frames during batch training, C represents the number of channels, T represents T consecutive frames of images, and H and W represent the height and width of the input image;

[0014] 2) Inputting the extracted feature map into the pooling module (40) to perform spatial average pooling operation to obtain a feature sequence F (50) of B, T, and C dimensions;

[0015] 3) The feature sequence F is input into the two branches of the joint-BERT self-attention model to extract channel and time features respectively: In the first branch, the extracted feature sequence F (50) is input into the channel reorganization module (60), and the recombined sub-channel feature sequence X is output. C After weighted processing by the channel BERT module (70), prediction is performed through the output of the fully connected layer to obtain the first prediction matrix (80) for behavior recognition; in the second branch, the feature sequence F (50) is input into the temporal BERT module (71), and the second prediction matrix (81) for behavior recognition is obtained through the output of the fully connected layer; the channel BERT module (70) and the temporal BERT module (71) in branches 1 and 2 share parameters;

[0016] 4) The first prediction matrix (80) and the second prediction matrix (81) are weighted and fused and then input into the classification module (90) to obtain the classification result of behavior recognition.

[0017] Furthermore, in step 3) of the above technical solution, the joint-BERT self-attention model includes two branches, a first branch for extracting inter-channel correlation and a second branch for extracting temporal correlation between image frames; the first branch is used to establish interactions between sub-channels from the semantic features of the reorganized sub-channel group using a multi-head self-attention mechanism to obtain key semantic features and their correlations for distinguishing action categories; the second branch is used to extract key image frames based on the similarity calculation between frames when the image frames at different times are fused using the self-attention mechanism.

[0018] As described in step 3) of the above technical solution, the specific steps of performing channel and time feature extraction in branch 1 and branch 2 respectively include:

[0019] (1) In the first branch, the key channel semantic features and their correlations for distinguishing action categories are obtained based on the channel reorganization module (60) and the channel BERT module (70). The feature sequence F (50) extracted by the two-dimensional convolutional neural network is input into the channel reorganization module (60). In the channel reorganization module, the sub-channel features corresponding to adjacent frames are reorganized and spliced ​​along the time dimension to form a sub-channel feature time series X containing the time change relationship. C ; Output recombinant channel characteristic sequence X C Self-attention calculation is performed through the channel BERT module (70) to extract key sub-channel features with large time-varying differences and their interaction correlations; weighted processing is performed based on the correlations between sub-channel feature sequences, and the first prediction matrix (80) for behavior recognition is obtained through the output of the fully connected layer, thereby realizing the modeling of channel feature interactions in the time dimension.

[0020] (2) In the second branch, the pooled feature sequence F (50) is directly input into the temporal BERT module (71), the similarity between video frames is calculated, and the second prediction matrix (81) for action recognition is obtained through the fully connected layer output.

[0021] The channel reorganization module (60) in step 3) of the above technical solution is as follows Figure 2 As shown, it is characterized by comprising a channel separation module (601) and a sub-channel feature sequence (602). The feature sequence F is input into the channel separation module (601), and is evenly divided into N sub-channels along the channel dimension. Each sub-channel contains C / N channel features, that is, F = [F'(1), F'(2), ..., F'(N)], where The sub-channel features F corresponding to adjacent frames are spliced ​​along the time dimension to obtain a sub-channel feature sequence (602). where N C =T×C / N; Specifically, for the nth (1≤n≤N) subchannel group have Indicates that each sub-channel group contains T-frame image feature sequence information.

[0022] The channel BERT module (70) in step 3) of the above technical solution is as follows Figure 3 As shown, it is characterized by comprising a position encoding layer (701), a multi-head self-attention mechanism module (702), a channel connection module (703) and a fully connected layer (704). The feature sequence X output by the channel reorganization module C The position encoding layer (701) input into the channel BERT module (70) encodes the position information to obtain the position embedding feature Embed positions into features Input the multi-head attention mechanism and the position feedforward network PFFN(·) layer shown in the multi-head self-attention mechanism module (702), and obtain a matrix that can highlight channel differences and sub-channel group interactions through self-attention calculation and nonlinear mapping of the PFFN(·) layer. The output Y of all sub-channel groups C The input channel connection module (703) performs splicing along the channel dimension to obtain a matrix y with the same channel dimension as the feature F C ; The matrix y C The fully connected layer (704) is input to obtain the first prediction matrix (80) for behavior recognition.

[0023] The present invention proposes a channel feature interaction time modeling behavior recognition method based on the BERT model, which is used to jointly learn the temporal change relationship of video frames and the interactive correlation between channels. Among them, the first branch designs a channel reorganization module and a channel BERT module to extract key sub-channel features with large differences in time changes and the interactive correlation between them, which not only enhances the channel interaction relationship between image frames and sub-channel groups, but also obtains key semantic features and their correlations for distinguishing action categories, thereby improving the accuracy of behavior classification. Joint-BERT combines the first branch for extracting inter-channel correlations with the second branch for extracting temporal correlations, and shares weights on the learnable parameters of the multi-head self-attention layer and feedforward neural network layer of the channel BERT and temporal BERT. It not only extracts key channel semantic features and their correlations in the channel BERT, but also obtains key action image frames through temporal BERT, which can further focus on the key channel semantic features in the key frames, thereby obtaining higher behavior recognition accuracy and achieving leading action recognition accuracy at home and abroad.

[0024] Compared with the existing technology, the present invention innovatively proposes a channel feature interaction time modeling behavior recognition method based on the BERT model. The outstanding technical effects of the present invention are:

[0025] 1. Based on the channel reorganization module and the channel BERT module, key sub-channel features with large time-varying differences and their interactive correlations are extracted. Channel aggregation is performed based on the correlation between sub-channel feature sequences to obtain key semantic features and their correlations for distinguishing action categories, thereby improving the accuracy of behavior classification.

[0026] 2. By combining channel-BERT and temporal BERT, not only can key channel semantic features and their correlations be extracted from channel-BERT, but key action image frames can also be obtained through temporal BERT. The fusion of the two can further focus on the key channel semantic features in key frames, further improving the accuracy of action classification, thereby achieving leading action recognition accuracy at home and abroad.

[0027] 3. This method uses a 2D CNN to extract semantic features from each image frame, and then leverages the BERT self-attention mechanism to extract spatial and temporal changes for action recognition. Compared to traditional action recognition methods based on 3D CNN, 2D CNN has fewer parameters and lower computational cost. Furthermore, by sharing weights between temporal BERT and channel BERT, the joint BERT network model does not add additional parameters.

[0028] 4. The behavior recognition accuracy of the present invention on the Something-Something (Sth-Sth) V1, V2 and HMDB-51 general public datasets is improved to 57.1%, 68.2% and 83.8% respectively, achieving leading behavior recognition accuracy at home and abroad. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the framework of the method of the present invention.

[0030] Figure 2 This is a schematic diagram of the channel recombination module structure design of the present invention.

[0031] Figure 3 This is a schematic diagram of the channel BERT module structure design of branch 1 of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0033] The present invention proposes a channel feature interaction time modeling behavior recognition method based on the BERT model, which consists of a first branch for extracting the correlation between channels and a second branch for extracting the temporal correlation between image frames; Figure 1 As shown, this embodiment specifically includes the following steps:

[0034] 1) Decompose the action video into corresponding RGB image sequences (10), which are input into a two-dimensional convolutional neural network module (20) for feature extraction to obtain feature maps (30) corresponding to B, T, C, H, and W dimensions; where B represents the batch number of input video frames during batch training, C represents the number of channels, T represents T consecutive frames, and H and W represent the height and width of the input image. In this embodiment, the TDN network in the existing method 1 is selected as the 2D ResNet50 residual network for extracting spatial features of the RGB sequence.

[0035] 2) The extracted feature map is input into the pooling module (40) for spatial average pooling operation to obtain a feature sequence F (50) of B, T, and C dimensions.

[0036] 3) The extracted feature sequence F(50) is input into the two branches of Joint-BERT respectively: the first branch uses the channel BERT model to extract the correlation between channels, and the second branch uses the temporal BERT model to extract the temporal correlation between image frames; when the first and second branches are jointly trained, the parameters of the channel BERT module (70) and the temporal BERT module (71) are shared.

[0037] 4) The first branch first inputs the feature sequence F (50) into the channel reorganization module (60), which is specifically expanded as follows Figure 2 As shown, the recombinant sub-channel characteristic sequence X is obtained through the channel separation operation module (601) C (602); Recombinant subchannel characteristic sequence X C enter Figure 1 The middle channel BERT module (70) is specifically expanded as follows Figure 3 As shown, through the position encoding layer (701) in the channel BERT module (70), the multi-head attention mechanism of the multi-head self-attention mechanism module (702) and the position feedforward network PFFN(·), the matrix Y that can highlight the interaction between the sub-channel groups and the channel differences is obtained. C ; Output matrix Y of all sub-channel groupsC The input channel connection module (703) performs splicing along the channel dimension to obtain an output vector y with the same channel dimension as the feature map C , and input into the fully connected layer (704) to obtain the first prediction matrix (80) for behavior recognition.

[0038] 5) The second branch directly inputs the spatial average pooled feature sequence F (50) into the temporal BERT module (71) and obtains the second prediction matrix (81) for action recognition by calculating the similarity between video frames.

[0039] 6) The first prediction matrix (80) and the second prediction matrix (81) are input into the classification module (90) to obtain the classification result of behavior recognition.

[0040] In step 4), the channel reorganization module (60) is as follows Figure 2 As shown in Figure 2, the feature sequence F extracted by the 2D ResNet-50 network is used as the input of the channel reorganization module, where F[F(1),F(2),...,F(T)], First, the feature sequence F is evenly divided into N sub-channels along the channel dimension by the channel separation operation module (601), which is expressed as F′=[F′(1), F′(2), ..., F′(N)]. Then the obtained sub-channel feature sequence F is spliced ​​along the time dimension to obtain the recombined sub-channel feature sequence (602) containing the channel information of adjacent frames. where N C =T×C / N; Specifically, for the nth (1≤n≤N)) subchannel group

[0041]

[0042] Among them, F′ T (n) represents the Tth frame in the nth segmented subchannel (1≤n≤N). Therefore, N recombined subchannel feature sequences are generated, namely Each channel group contains the medium- and long-term change relationships between T-frame image features.

[0043] In step 4), the channel BERT module (70) is as follows Figure 3 As shown, the characteristic sequence X output by the channel reorganization module C As the input of the channel BERT module, the interaction between the sub-channel group and the adjacent video frames is enhanced through the BERT self-attention mechanism. The position encoding layer (701) in the channel BERT self-attention module is used to obtain the learnable position embedding feature.

[0044]

[0045] Among them, P C Indicates that X C Learnable positional parameters of the same dimension.

[0046] Embed positions into features Input the multi-head self-attention mechanism module (702) to do Attention(·) calculation, usually given a query key Sum Yes Perform linear transformation mapping to obtain the corresponding query matrix, key matrix and value matrix, set d q =d k =d v =N C / h; Therefore, the calculation of Attention(·) of the i-th self-attention head can be expressed as:

[0047]

[0048] in, is the scale factor, describe and The similarities between Attention(·) calculation and The channel interactions between all subchannel groups in .

[0049] The outputs of the h heads of the self-attention mechanism are concatenated and linearly transformed to obtain the output of the multi-head self-attention mechanism:

[0050]

[0051] in,

[0052] The features learned by the multi-head attention mechanism are input into the position feedforward network PFFN(·) to obtain the matrix of sub-channel group interactions

[0053]

[0054] Where PFFN(x)=W2GELU(W1x+b1)+b2, GELU(·) is the activation function.

[0055] The output matrix Y of all recombinant sub-channel sequences C Input channel connection module (703), Y CMapped to the C / N dimension through a trainable linear projection and then concatenated along the channel dimension:

[0056]

[0057] in,

[0058] Finally, y c Input the fully connected layer to make the final action prediction. C Mapped to the same prediction matrix as the number of action recognition video classifications: y cb =y c W1. At the same time, in order to better aggregate the information of the video frame, the time dimension of the feature sequence F is averaged pooled: y avg =AvgPool(FW2). N FC is the number of action recognition video classifications. Therefore, the loss function of channel-BERT for end-to-end training is:

[0059]

[0060] Where α is a hyperparameter, 0<α<1; L(·) is the cross entropy loss function; To output the true label.

[0061] In step 5), branch 2 directly inputs the pooled feature sequence F into the temporal BERT module (71). The feature sequence F passes through the position encoding layer in the temporal BERT module to obtain a learnable position embedding feature.

[0062]

[0063] in, P T Represents During training, the channel BERT module (70) and the temporal BERT module (71) share the parameter matrix W Q 、W K and W V .

[0064] To use the temporal BERT module for classification, add additional classification embedding vectors and the corresponding classification vector

[0065]

[0066] in, describe and The similarities between Attention(·) calculation and The importance of all image frames in .

[0067] Therefore, the total loss function when Joint-BERT is trained end-to-end is:

[0068]

[0069] in, is the predicted class vector of the temporal BERT module, given by Mapped to action category generation through a fully connected layer W3∈. jb =βy cb +γy tb It is the weighted fusion of the outputs of branch 1 and branch 2. β and γ are hyperparameters, 0<β<1, 0<γ<1.

[0070] To verify the present invention, the validity was verified using the publicly available Something-Something(Sth-Sth)V1, V2, and HMDB-51 datasets commonly used in behavior recognition. The comparison results of the behavior recognition accuracy with other known advanced methods are shown in Table 1. The behavior recognition accuracy of the method of the present invention on the Something-Something(Sth-Sth)V1, V2, and HMDB-51 datasets was improved to 57.1%, 68.2%, and 83.8%, respectively. The comparison results show that the joint-BERT model proposed in the present invention achieves more effective modeling of channel feature interactions in the time dimension, thereby achieving better behavior recognition accuracy, reaching the current leading level at home and abroad.

[0071] Table 1. Comparison of behavior recognition accuracy

[0072] method Sth-Sth V1 Sth-Sth V2 HMDB51 Existing method 1 47.2％ 63.4％ 73.5％ Existing method 2 51.0％ 62.9％ 75.7％ Existing method three 53.9％ 65.3％ 76.3％ Embodiments of the present invention 57.1％ 68.2％ 83.8％

[0073] Existing method 1: The method proposed by Lin, J. et al. in "TSM: Temporal shift module for efficient video understanding" published at ICCV (IEEE International Conference on Computer Vision) in 2019.

[0074] Existing method 2: The method proposed by Wu, W. et al. in the 2021 AAAI (Association for the Advance of Artificial Intelligence) conference article "MVFNet: Multi-View Fusion Network for Efficient Video Recognition".

[0075] Existing method three: The method proposed by Wang, L. et al. in "TDN: Temporal Difference Networks for Efficient Action Recognition" published at the 2021 CVPR (IEEE Conference on Computer Vision and Pattern Recognition).

[0076] This paper proposes an action recognition method that uses temporal modeling of channel-feature interactions, combining a 2D convolutional network and the BERT self-attention mechanism. This method extracts key sub-channel features that vary significantly over time, as well as their interactive correlations. This method obtains key semantic features that distinguish action categories and their correlations, thereby improving action classification accuracy. By combining channel-BERT with temporal BERT, we can further focus on key channel semantic features in key frames, achieving even higher action recognition accuracy.

Claims

1. A channel feature interaction time modeling behavior recognition method based on the BERT model, characterized by The specific steps are as follows: 1) Decompose the action video into corresponding RGB image sequences and input them into a two-dimensional convolutional neural network to extract feature maps with B, T, C, H, and W dimensions. Where B represents the batch size of input video frames during batch training, C represents the number of channels, T represents T consecutive frames, and H and W represent the height and width of the input image. 2) Input the extracted feature map into the pooling module for spatial average pooling operation to obtain the feature sequence F of B, T, and C dimensions; 3) The feature sequence F is input into the two branches of the joint-BERT self-attention model to extract channel and time features respectively: In the first branch, the extracted feature sequence F is input into the channel reorganization module, and the recombined sub-channel feature sequence X is output. C After weighted processing by the channel BERT module, prediction is performed through the fully connected layer output to obtain the first prediction matrix for behavior recognition. In the second branch, the feature sequence F is input into the temporal BERT module, and the second prediction matrix for behavior recognition is obtained through the fully connected layer output. The channel BERT module and the temporal BERT module in the first and second branches share parameters. The joint-BERT self-attention model includes two branches: a first branch for extracting inter-channel correlations and a second branch for temporal correlations between image frames. The first branch is used to establish interactions between sub-channels from the semantic features of the recombined sub-channel group using a multi-head self-attention mechanism to obtain key semantic features and their correlations for distinguishing action categories. The second branch is used to extract key image frames based on similarity calculations between frames when the self-attention mechanism is used to fuse image frames at different times. The specific steps of inputting the feature sequence F into the two branches of the joint-BERT self-attention model and extracting channel and time features respectively include: (1) In the first branch, the key channel semantic features and their correlations for distinguishing action categories are obtained based on the channel reorganization module and the channel BERT module. The feature sequence F extracted by the two-dimensional convolutional neural network is input into the channel reorganization module. In the channel reorganization module, the sub-channel features corresponding to adjacent frames are reorganized and spliced ​​along the time dimension to form a sub-channel feature time series containing time-varying relationships; the reorganized sub-channel feature sequence X is output. C The channel BERT module performs self-attention calculations to extract key sub-channel features that vary significantly over time and their interaction correlations. Weighted processing is performed based on the correlations between sub-channel feature sequences, and the first prediction matrix for behavior recognition is output through the fully connected layer, achieving modeling of channel feature interactions in the temporal dimension. (2) In the second branch, the pooled feature sequence F is directly input into the temporal BERT module to calculate the similarity between video frames, and the second prediction matrix for action recognition is obtained through the output of the fully connected layer; 4) The first prediction matrix and the second prediction matrix are weighted and fused and then input into the classification module to obtain the classification result of behavior recognition.

2. The channel feature interaction time modeling behavior recognition method based on the BERT model as claimed in claim 1 is characterized by In step 3), the channel recombining module comprises a channel separation module and a sub-channel characteristic sequence; The feature sequence F is input into the channel separation module and divided into N sub-channels along the channel dimension. Each sub-channel contains C / N channel features, that is, F' = [F'(1), F'(2), ..., F'(N)], where The sub-channel features F' corresponding to adjacent frames are spliced ​​along the time dimension to obtain the sub-channel feature sequence. where N C =T×C / N; for the nth (1≤n≤N)) subchannel group have Indicates that each sub-channel group contains T-frame image feature sequence information.

3. The channel feature interaction time modeling behavior recognition method based on the BERT model as claimed in claim 1 is characterized by In step 3), the channel BERT module includes a position encoding layer, a multi-head self-attention mechanism module, a channel connection module and a fully connected layer; the feature sequence X output by the channel reorganization module C The position encoding layer of the input channel BERT module encodes the position information to obtain the position embedding feature Embed positions into features The multi-head attention mechanism and position feedforward network PFFN(·) layer of the multi-head self-attention mechanism module are input. Through the self-attention calculation and the nonlinear mapping of the PFFN(·) layer, a matrix that can highlight channel differences and sub-channel group interactions is obtained. The output Y of all sub-channel groups C The input channel connection module is spliced ​​along the channel dimension to obtain a matrix y with the same channel dimension as the feature F C ; The matrix y C Input the fully connected layer to obtain the first prediction matrix for behavior recognition.

Images