A behavior recognition method and device based on multi-level feature fusion

Abstract

The present invention discloses a behavior recognition method and device based on multi-level feature fusion, comprising the following steps: Step S1: Collection of multi-level features; Step S2: Significant expression of temporal features; Step S3: Multi-level feature fusion method; Step S4: Outputting the target task result. The present invention simultaneously considers for the first time the problem of the temporal receptive field and the problem of weak ability to distinguish different motion rates in three-dimensional convolutional neural networks, so that the three-dimensional convolutional neural network can effectively improve the accuracy of behavior recognition while maintaining little change in computational overhead. The present invention designs a significant expression mechanism for temporal features, obtains global spatiotemporal information by performing significant expression processing on features, and enhances the long-term temporal modeling capability of the model. A multi-level feature fusion module is designed, which utilizes the perception of motion rates by features at different levels to enhance the resolution of the model and realize the ability to recognize behaviors at different motion rates under a single input source.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer neural networks, and in particular relates to a behavior recognition method and device based on multi-level feature fusion. Background Art

[0002] In recent years, thanks to the widespread use of surveillance cameras and the rapid development of internet companies, behavior recognition methods have been widely applied in daily life, such as in smart transportation, virtual reality technology, smart cities, security monitoring, and video recommendations. Video-based behavior recognition technology uses computer vision techniques to identify and understand human behavior in videos. It is widely used in video recommendations and intelligent security systems. Compared to image classification tasks, video data adds a temporal dimension to the input, so how to mine temporal features in videos has become a key focus of behavior recognition research.

[0003] Currently, effective action recognition methods based on convolutional neural networks can be roughly divided into two categories: two-stream convolutional neural networks (CNNs) and three-dimensional convolutional neural networks (3D convolutional neural networks). Both network structures can effectively capture spatiotemporal information from video data and accomplish action recognition tasks. The core idea of ​​the two-stream convolutional network is to design two independent CNNs, taking image data and optical flow maps as input, respectively, to extract spatial and temporal features from the video. Ultimately, different fusion strategies are used to effectively fuse these two features, resulting in highly accurate action classification. 3D convolutional neural networks, by expanding the convolution kernel along the temporal dimension to obtain a three-dimensional convolution kernel for feature extraction, can directly mine spatiotemporal features from video data. Compared to traditional machine learning methods, action recognition methods based on 3D convolutional neural networks do not require optical flow or hand-crafted features in the input data, reducing the complexity of data preprocessing and achieving an end-to-end deep learning network architecture. With the continuous increase in computing resources and continuous algorithm optimization, 3D convolutional neural networks are playing an important role in the field of action recognition.

[0004] Three-dimensional convolutional neural networks can simultaneously extract temporal and spatial features from videos. However, due to the limited receptive field of the convolution kernel, their ability to model long-term temporal sequences is limited. The size of the 3D convolution kernel is typically set at the outset of network design, and the kernel cannot perceive video data outside its receptive field. If an action occurs over a long period of time, the convolutional neural network will lose its ability to perceive the global spatiotemporal information, leading to misidentification of the action. Furthermore, visually similar actions can generate different labels depending on their speed, resulting in reduced accuracy in action recognition. For example, walking, jogging, and running are very similar in morphology, but their movement rates vary significantly. Convolutional neural networks can recognize actions of varying speeds by adding input sources with different sampling rates. However, this requires a separate network branch for each input source, exponentially increasing the computational resources required by the model.

[0005] In summary, although effective end-to-end convolutional neural network methods have been invented in the field of action recognition, the problems of long-term video modeling and the resolution of different motion rates have not yet been well solved.

[0006] The main problems with current mainstream behavior recognition technology are:

[0007] First, 3D convolutional neural networks can achieve high-precision behavior recognition. However, these methods, primarily due to the size of the convolution kernel, fail to fully consider the long-term temporal modeling issues involved in behavior recognition. Consequently, they often misidentify samples with long behavior cycles. Specifically, the convolution kernels in convolutional neural networks cannot adaptively adjust based on the input data, and are unable to process video data outside of their receptive field. Furthermore, simply increasing the receptive field of the convolution kernel not only increases computing resource consumption but also reduces behavior recognition accuracy due to the collection of excessive redundant information. Therefore, the long-term temporal modeling issues of 3D convolutional neural networks remain to be addressed.

[0008] Secondly, 3D convolutional networks easily confuse action labels with similar visual appearance but different motion speeds. This is because the predefined network structure can only process actions at the sampling rate of the input data. However, increasing the input source—sampling video data at different sampling rates and feeding it into the convolutional neural network for processing—can improve action recognition accuracy, but the repeated input of multiple data types leads to an exponential increase in resource consumption.

[0009] The implementation scheme closest to the present invention is the Chinese invention disclosure patent entitled "A method for human behavior recognition based on multi-scale spatiotemporal graph convolutional neural network", application date 2021.09.01, application number 202111020275.8. In this method, a deep neural network model is created that includes a multi-scale graph convolution module and a multi-time feature fusion module. The deep neural network model used uses the ST-GCN neural network model as the basic structure of the backbone network, converting the graph convolution network in ST-GCN into a multi-scale graph convolutional network module, and the temporal convolution network into a multi-time feature fusion module, so that the network can extract the spatiotemporal features of the skeletal sequence to be identified.

[0010] The multi-temporal feature fusion module is used to extract temporal features from human skeleton sequences. It consists of four parallel temporal convolutional network branches with different convolution kernel sizes and one attention module branch. The convolution kernel sizes of the four temporal convolutional network branches are 3x1, 5x1, 7x1, and 9x1, respectively, with different receptive fields corresponding to features of different temporal lengths. Another attention module branch uses a Squared-Excitation Network (SEN) that performs global average pooling on the output of the spatial multi-scale module, passes it through a fully connected layer, and finally applies the probabilities output by the softmax function to each of the four temporal convolutional network branches. The feature maps output by the four branches are then concatenated. The created dataset is used to train and test a deep convolutional neural network model; the trained model is then used to classify the video images to be recognized.

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention is proposed. Summary of the Invention

[0012] The purpose of the present invention is to address the deficiencies in the existing technology. The present invention innovatively proposes a behavior recognition method and device based on multi-level feature fusion, redesigns the three-dimensional neural network structure, designs a temporal feature compression excitation mechanism and a multi-level feature fusion method, and improves the model's long-term temporal modeling capability and the ability to distinguish different motion rates. First, the features of different levels are extracted, and the global spatiotemporal information is extracted through the compression excitation module, and the important time segments are targetedly enhanced, thereby expanding the receptive field of the three-dimensional convolutional neural network. Secondly, the perception of motion rate of features at different levels is utilized to fuse multi-level features, so as to enable the model to obtain more accurate behavior recognition capabilities. Behavior recognition of different motion rates under a single input source is realized.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] A behavior recognition method based on multi-level feature fusion includes the following steps:

[0015] Step S1: Collection of multi-level features;

[0016] Construct a three-dimensional convolutional neural network model to extract features at different levels in the behavior recognition network and realize the collection of multi-level features;

[0017] Step S2: significant expression of temporal features;

[0018] After the operation of step S1, a three-dimensional feature map containing spatiotemporal information is obtained;

[0019] Through the time series feature saliency expression method, global information is obtained. By compressing and stimulating the time series dimension features, feature saliency expression is completed and long time series modeling is realized. Specifically, it includes:

[0020] Step S2-1: compression;

[0021] Compress the overall feature map to the time series dimension before performing the excitation operation, so that the time series information does not interfere with each other before interacting at this layer;

[0022] Characteristics of this layer Perform average pooling operation to obtain independent feature expression at the time series level, and the generated features Then proceed with feature excitation to achieve global temporal feature modeling;

[0023]

[0024] In the formula, Γ is the reshape operation, which performs an average pooling operation after swapping the time dimension and channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, i and j are the spatial positions of the feature points, and the compressed time feature expression of the frame is obtained by swapping the channel and time dimension of the sampled features and then pooling them.

[0025] Step S2-2: incentive;

[0026] The time series excitation operation uses the time series features obtained through compression to control the global weight and learn the contextual relationship of the time series dimension, which helps to better mine representative time series feature expressions during the final prediction.

[0027] The excitation operation is achieved by using ReLU, Sigmoid and fully connected layers:

[0028] M=f ex (F Squeeze )=σ(W2δ(W1F Squeeze ))

[0029] Where σ is the Sigmoid function, σ is the ReLU function,

[0030] In order to obtain the saliency map M, the excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map

[0031] Step S2-3: global timing mapping;

[0032] Add the mapping to the entire feature and broadcast the mapping to obtain the stimulated feature.

[0033] Mapping after broadcast The feature map is recalculated by element-wise multiplication, completing the temporal excitation operation of the feature:

[0034] In the formula, Fan broadcasts the input mapping map M and expands it into a mapping map of the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication. base Get the feature F after excitation ex .

[0035] F ex =Fan(M)×F base

[0036] Step S3: multi-level feature fusion method;

[0037] After obtaining the global features after excitation, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is based on the comprehensive features at different levels and different motion speeds. The specific steps are as follows:

[0038] In step S3.1, average pooling is performed on different features to adjust the feature space size of different layers to the same as the feature size of the last layer;

[0039] In step S3.2, after performing the average pooling operation, the features at different levels are concatenated in the channel dimension to aggregate the features of all levels in the pyramid structure.

[0040] Step S3.3, fusion;

[0041] Using three-dimensional convolution kernel, we can mine spatiotemporal features of features after splicing at different levels and correct the fusion feature F. fusion The number of channels;

[0042] The fusion feature size and number of channels are aligned with the output feature size of the last layer of the backbone network;

[0043] The fusion method is calculated using the following formula;

[0044] in The feature after the n-th level feature excitation, there are 4 layers of features in total, n is 4;

[0045] Conv is the convolution operation;

[0046] Pool is an average pooling operation, which aligns the features of different levels in spatial dimensions;

[0047] Concat performs channel-level splicing operations to splice features at different levels, and obtains the feature F after multi-level feature fusion. mff ;

[0048]

[0049] Step S4: Output the target task result.

[0050] The preferred technical solution provided by the present invention is:

[0051] The three-dimensional convolutional neural network model includes a three-dimensional convolutional layer module, a batch normalization module, a ReLU activation layer module, a direct mapping module, and a skip mapping module, wherein:

[0052] The three-dimensional convolutional layer module can simultaneously extract spatial and temporal information from video data, laying a good foundation for subsequent global processing and multi-level feature fusion;

[0053] Batch normalization modules and ReLU activation layer modules can improve the learning ability of the network and prevent gradient explosion and overfitting;

[0054] The direct mapping module and the skip mapping module can ensure that the generalization ability of the three-dimensional convolutional neural network model is maintained when the model depth increases, and increase the number of parameters of the three-dimensional convolutional neural network model, thereby adapting to more complex tasks.

[0055] Beneficial effects:

[0056] This invention is the first to simultaneously consider the problem of temporal receptive field priority and the weak ability to distinguish different motion rates in three-dimensional convolutional neural networks, enabling the three-dimensional convolutional neural network to effectively improve the accuracy of behavior recognition while maintaining little change in computational overhead.

[0057] This paper designs a mechanism for expressing temporal feature saliency. By processing the features for saliency expression, it captures global spatiotemporal information and enhances the model's long-term temporal modeling capabilities. First, features at different levels of a three-dimensional convolutional neural network are extracted, and temporal self-attention maps are calculated and mapped onto the original features to complete the excitation operation. A multi-level feature fusion module is designed, leveraging the motion rate perception capabilities of features at different levels to enhance the model's resolution and enable behavioral recognition of different motion rates from a single input source.

[0058] This paper designs a behavior recognition method and device based on multi-level feature fusion. By extracting global spatiotemporal information and integrating multi-level features, the model's long-term temporal modeling capabilities and its ability to distinguish between different motion rates are enhanced. While maintaining a single input source, the computational overhead remains minimal and the model's behavior recognition accuracy is improved.

[0059] Explanation of terms:

[0060] Deep learning network: A deep learning network is a machine learning model that consists of multiple layers of neurons. Each layer is connected to the neurons in the previous and next layers to learn the mapping relationship from input to output.

[0061] Multi-layer features: The "depth" in deep learning networks refers to the number of layers within the network. Generally, networks with more layers can learn more complex features and patterns. Multi-layer features refer to the features of varying depths extracted during the information mining process of a neural network.

[0062] Behavior recognition: Behavior recognition refers to the use of data and machine learning techniques to identify and classify behavioral patterns or actions of humans or other entities. This typically involves collecting data using sensors (such as cameras, accelerometers, gyroscopes, etc.) and inputting it into a deep learning model for analysis and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 A processing flow chart of a behavior recognition method and device based on multi-level feature fusion provided by an embodiment of the present invention;

[0065] Figure 2 This is a specific instantiation structure diagram of the multi-level feature fusion network model in the present invention;

[0066] Figure 3 This is a specific instantiation structure diagram of the timing feature compression excitation module in the present invention;

[0067] Figure 4 This is a specific instantiation structure diagram of unified excitation and group excitation in the temporal feature compression excitation module of the present invention;

[0068] Figure 5This is a specific instantiation structure diagram of multi-level feature fusion. DETAILED DESCRIPTION

[0069] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the following detailed description is illustrative rather than restrictive and should not be used to limit the scope of protection of the present invention.

[0070] The present invention proposes a behavior recognition method and device based on multi-level feature fusion. The processing flow chart of the method is as follows: Figure 1 As shown, it specifically includes the following steps:

[0071] Step S1: Multi-level feature collection.

[0072] While ensuring the model's learning capabilities, it also reduces the cost of computing and storage resources as much as possible. The specific structure is as follows Figure 2 shown.

[0073] Taking the 3D-ResNet network as an example, the multi-level feature collection method is based on the output of the residual block. A set of feature fusion modules with four levels of feature layering are designed. These features are arranged from shallow to deep, with the temporal and spatial receptive fields gradually increasing. The length of the temporal dimension does not change, while the size of the spatial dimension gradually decreases.

[0074] To preserve spatial semantics as much as possible, the space is not downsampled before feature fusion. Therefore, the collected 4-level features will consist of features {F1, F2, ..., F4} of size {C1 × T × H1 × W1, ..., C4 × T × H4 × W4}. Specifically, T is set to 8, that is, the length of the temporal dimension in the feature is 8; H1 and W1 are set to 28, and the width and height are reduced to half of the original depth each time the depth increases. That is, the width and height of each subsequent feature layer are 14, 7, and 4 respectively.

[0075] C1 is set to 256. The number of channels doubles with each additional layer of depth, so the number of channels per layer is 256, 1024, and 2048, respectively. The multi-level pyramid features collected in this way contain richer semantic expressions in the spatial dimension, but attention should be paid to the subsequent feature fusion to ensure that the information flow during fusion is correctly transmitted.

[0076] Step S2: Time series feature significance expression method

[0077] After the operation of step S1, the present invention can obtain a three-dimensional feature map containing rich spatiotemporal information. However, the model is limited by the three-dimensional convolution kernel and its long-term time series modeling capability is lacking. Therefore, the present invention designs a temporal feature saliency expression (STE) method to obtain global information, and completes the feature saliency expression and realizes long-term time series modeling by compressing and exciting the temporal dimension features. The specific structure is as follows: Figure 3 shown.

[0078] Step S2-1: Compression

[0079] The present invention proposes to compress the overall feature map into the time series dimension before performing the excitation operation, so that the time series information will not interfere with each other before the interaction of the current layer. Perform average pooling operation to obtain independent feature expression at the time series level, and the generated features Then proceed with feature excitation to achieve global timing feature modeling.

[0080]

[0081] In the formula, Γ is a reshape operation, which performs an average pooling operation after swapping the temporal dimension and channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, i and j are the spatial positions of the feature points, and the compressed temporal feature expression of the frame is obtained by swapping the channel and temporal dimensions of the sampled features and then pooling them.

[0082] Step S2-2: Incentive

[0083] The time series excitation operation uses the time series features obtained by compression to control the global weight and learn the contextual relationship of the time series dimension. This can better help to mine representative time series feature expressions during the final prediction.

[0084] To achieve these goals, the present invention implements the excitation operation by using ReLU, Sigmoid and fully connected layers:

[0085] M=f ex (F Squeeze )=σ(W2δ(W1F Squeeze ))

[0086] Where σ is the Sigmoid function, σ is the ReLU function, In order to obtain the saliency map M. The excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map

[0087] Step S2-3: Global Timing Mapping

[0088] In order to apply the mapping to the entire feature, the mapping is broadcasted to obtain the stimulated feature.

[0089] Because the mapping in the saliency expression module is calculated for the temporal dimension, the spatial information is compressed into the temporal dimension during the calculation process. Similarly, during the excitation phase, this temporal mapping can also be applied to the entire spatial dimension and channel dimension.

[0090] Mapping after broadcast The feature map is recalculated through element-level multiplication to complete the temporal excitation operation of the feature.

[0091] F ex =Fan(M)×F base

[0092] In the formula, Fan broadcasts the input mapping map M and expands it into a mapping map of the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication. base Get the feature F after excitation ex .

[0093] Depending on the complexity of the scene and action classification, the temporal mapping methods under different actions may be completely different.

[0094] To make the model more robust, a grouping operation was added to the STE module to enhance the adaptability of the temporal saliency expression module to video data with different actions and backgrounds. The research hopes to enable the model to adaptively stimulate different actions through multiple groupings.

[0095] Specifically, when the feature is input into the STE module, the STE module will split the feature map into 4 groups along the channel dimension. The group features obtained by segmentation will each enter their own independent stimulation layer for temporal significance stimulation, such as Figure 4 shown.

[0096] Step S3: Multi-level feature fusion method

[0097] After obtaining the global features after excitation, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is based on the comprehensive features of different levels and different motion speeds. The fusion stage process is as follows: Figure 5 shown.

[0098] In order to complete comprehensive feature fusion, we first perform average pooling on different features to adjust the feature space sizes of different levels to the same as the feature size of the last layer. This is beneficial to the calculation of the fully connected layer, and since the feature size of the last layer is the smallest in the entire backbone network, it can reduce computational overhead.

[0099] There are two reasons for choosing pooling operation instead of convolution operation:

[0100] Firstly, the feature sizes at different levels are different. Through the pooling operation, the morphological differences between features at different levels can be eliminated, preparing for subsequent concatenation along the channel dimension.

[0101] Secondly, the pooling operation can save computing resources while maintaining the model performance.

[0102] The features output by the saliency expression module are {C1×T×H1×W1,...,C4×T×H4×W4}, and {Ci < Cj, Hi > Hj, Wi > Wj; i < j}. Average pooling is performed on the features before the 4th layer to align the spatial size of the features with the feature size of the last layer, that is, the feature sizes of all levels are controlled to be 8×4×4.

[0103] Through the pooling operation on the feature map, the resources consumed by subsequent calculations can be reduced, and the global information obtained after activation can be retained.

[0104] After the average pooling operation, concatenation operations are performed on the features at different levels in the channel dimension to summarize the features of all levels in the pyramid structure.

[0105] The features obtained after summarization Although the differences in spatio-temporal dimensions of the fused features are eliminated, there is a significant gap in the channel dimension compared with the output of the last layer of the backbone network. Through experiments and observations, when the unbalanced backbone network features and the fused features are input into the fully connected layer together, the spatio-temporal information therein will interfere with each other, increasing the computational cost and reducing the accuracy of action recognition.

[0106] To address this problem, the present invention uses a three-dimensional convolutional kernel to perform spatio-temporal feature mining on the concatenated features at different levels to correct the fused feature F fusion The number of channels to maintain the balance between the backbone network and the feature fusion branch. After the above calculations, the fused feature size and the number of channels are aligned with the output feature size of the last layer of the backbone network

[0107] The fusion method is calculated according to the following formula.

[0108]

[0109] The present invention first considers both the problem of the temporal receptive field in the three-dimensional convolutional neural network and the problem of weak ability to distinguish different motion rates for the first time, enabling the three-dimensional convolutional neural network to effectively improve the accuracy of action recognition while keeping the computational cost change不大.

[0110] This paper designs a mechanism for expressing temporal feature saliency. By processing the features for saliency expression, it captures global spatiotemporal information and enhances the model's long-term temporal modeling capabilities. First, features at different levels of a three-dimensional convolutional neural network are extracted, and temporal self-attention maps are calculated and mapped onto the original features to complete the excitation operation. A multi-level feature fusion module is designed, leveraging the motion rate perception capabilities of features at different levels to enhance the model's resolution and enable behavioral recognition of different motion rates from a single input source.

[0111] This paper designs a behavior recognition method and device based on multi-level feature fusion. By extracting global spatiotemporal information and integrating multi-level features, the model's long-term temporal modeling capabilities and its ability to distinguish between different motion rates are enhanced. While maintaining a single input source, the computational overhead remains minimal and the model's behavior recognition accuracy is improved.

[0112] The terminal proposed in this embodiment and the behavior recognition method proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to any of the above embodiments, and this embodiment has the same beneficial effects as executing the behavior recognition method.

[0113] In addition, an embodiment of the present application also proposes a computer-readable storage medium, which can be a non-volatile computer-readable storage medium. A behavior recognition program is stored on the computer-readable storage medium, and when the behavior recognition program is executed by the processor, the behavior recognition method of the present application as described above is implemented.

[0114] The various embodiments of the electronic device and computer-readable storage medium of the present application can refer to the various embodiments of the behavior recognition method of the present application, and will not be repeated here.

[0115] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0116] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling an electronic device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0118] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method based on multi-level feature fusion, characterized in that: The following steps are involved: Step S1: Collection of multi-level features; Construct a three-dimensional convolutional neural network model to extract features at different levels in the behavior recognition network and realize the collection of multi-level features; Step S2: significant expression of temporal features; After the operation of step S1, a three-dimensional feature map containing spatiotemporal information is obtained; Through the time series feature saliency expression method, global information is obtained. By compressing and stimulating the time series dimension features, feature saliency expression is completed and long time series modeling is realized. Specifically, it includes: Step S2-1: compression; Compress the overall feature map to the time series dimension before performing the excitation operation, so that the time series information does not interfere with each other before interacting at this layer; Characteristics of this layer Perform average pooling operation to obtain independent feature expression at the time series level, and the generated features Then proceed with feature excitation to achieve global temporal feature modeling; In the formula, Γ is the reshape operation, which performs an average pooling operation after swapping the time dimension and channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, i and j are the spatial positions of the feature points, and the compressed time feature expression of the frame is obtained by swapping the channel and time dimension of the sampled features and then pooling them. Step S2-2: incentive; The time series excitation operation uses the time series features obtained through compression to control the global weight and learn the contextual relationship of the time series dimension, which helps to better mine representative time series feature expressions during the final prediction. The excitation operation is achieved by using ReLU, Sigmoid and fully connected layers: M=f ex (F Squeeze )=σ(W2δ(W1F Squeeze )) where σ is the Sigmoid function, σ is the ReLU function, In order to obtain the saliency map M, the excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map Step S2-3: global timing mapping; Add the mapping to the entire feature and broadcast the mapping to obtain the stimulated feature. Mapping after broadcast The feature map is recalculated by element-wise multiplication, completing the temporal excitation operation of the feature: F ex =Fan(M)×F base The specific calculation method is as follows: Fan broadcasts the input mapping map M to expand it into a mapping map with the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication. base Get the feature F after excitation ex； Step S3: multi-level feature fusion method; After obtaining the global features after excitation, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is based on the comprehensive features at different levels and different motion speeds. The specific steps are as follows: In step S3.1, average pooling is performed on different features to adjust the feature space size of different layers to the same as the feature size of the last layer; In step S3.2, after performing the average pooling operation, the features at different levels are concatenated in the channel dimension to aggregate the features of all levels in the pyramid structure. Step S3.3, fusion; Using three-dimensional convolution kernel, we can mine spatiotemporal features of features after splicing at different levels and correct the fusion feature F. fusion The number of channels; The fusion feature size and number of channels are aligned with the output feature size of the last layer of the backbone network; The fusion method is calculated using the following formula; in The feature after the n-th level feature excitation, there are 4 layers of features in total, n is 4; Conv is the convolution operation; Pool is an average pooling operation, which aligns the features of different levels in spatial dimensions; Concat performs channel-level splicing operations to splice features at different levels, and obtains the feature F after multi-level feature fusion. mff ; Step S4: Output the target task result.

2. The behavior recognition method based on multi-level feature fusion according to claim 1 is characterized in that: The three-dimensional convolutional neural network model includes a three-dimensional convolutional layer module, a batch normalization module, a ReLU activation layer module, a direct mapping module, and a skip mapping module, wherein: The three-dimensional convolutional layer module can simultaneously extract spatial and temporal information from video data, laying a good foundation for subsequent global processing and multi-level feature fusion; Batch normalization modules and ReLU activation layer modules can improve the learning ability of the network and prevent gradient explosion and overfitting; The direct mapping module and the skip mapping module can ensure that the generalization ability of the three-dimensional convolutional neural network model is maintained when the model depth increases, and increase the number of parameters of the three-dimensional convolutional neural network model, thereby adapting to more complex tasks.

3. An electronic device, characterized in that: include: A memory, a processor, and a behavior recognition program stored in the memory and executable on the processor, wherein the behavior recognition program, when executed by the processor, implements the behavior recognition method according to any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a behavior recognition program, which, when executed by a processor, implements the behavior recognition method according to any one of claims 1 to 2.

Images