Behavior recognition method and device based on multi-level feature fusion

Abstract

The invention discloses a behavior recognition method and device based on multi-level feature fusion. The method comprises the steps of S1, collecting multi-level features; s2, performing significant expression on the time sequence characteristics; s3, a multi-level feature fusion method is adopted; step S4; and outputting a target task result. According to the method, the first problem of the time sequence receptive field in the three-dimensional convolutional neural network and the problem of weak resolution capability of different movement rates are considered at the same time for the first time, so that the accuracy of behavior recognition is effectively improved under the condition that the three-dimensional convolutional neural network maintains small change of calculation overhead. According to the method, a time sequence feature saliency expression mechanism is designed, global spatio-temporal information is obtained by performing saliency expression processing on features, and the long time sequence modeling capability of the model is enhanced. A multi-level feature fusion module is designed, the perception ability of different level features to the motion rate is utilized, the resolution of the model is enhanced, and the behavior recognition ability of different motion rates under a single input source is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer neural network, and in particular relates to a behavior recognition method and device based on multi-level feature fusion. Background Art

[0002] In recent years, benefiting from the popularity of surveillance cameras and the rapid development of Internet companies, behavior recognition methods have been widely used in people's daily lives, such as smart transportation, virtual reality technology, smart cities, security monitoring, video recommendation, etc. Video-based behavior recognition technology is a task that uses computer vision technology to identify and understand human behavior in videos. It is widely used in video recommendation and intelligent security systems. Compared with image classification tasks, video data adds a time series dimension to the input, so how to mine the time series features in videos has become the focus of behavior recognition research.

[0003] At present, effective behavior recognition methods based on convolutional neural networks can be roughly divided into two categories, namely two-stream convolutional neural networks and three-dimensional convolutional neural networks. Both network structures can effectively obtain spatiotemporal information in video data and complete behavior recognition tasks. The core idea of ​​the two-stream convolutional network is to design two independent convolutional neural networks, input image data and optical flow maps respectively to extract spatial features and temporal features in the video, and finally use different fusion strategies to effectively fuse the two features to perform high-precision classification of behaviors in the video. The three-dimensional convolutional neural network expands the convolution kernel along the time dimension to obtain a three-dimensional convolution kernel for feature extraction, and can directly mine spatiotemporal features from video data. Compared with traditional machine learning methods, the behavior recognition method based on the three-dimensional convolutional neural network does not need to use optical flow or manually designed features in the input data, which reduces the complexity of data preprocessing and realizes an end-to-end deep learning network architecture. With the continuous enhancement of computing resources and the continuous optimization of algorithms, the three-dimensional convolutional neural network plays an important role in the field of behavior recognition.

[0004] Three-dimensional convolutional neural networks can extract temporal and spatial features from videos at the same time, but due to the limited receptive field of the convolution kernel, the long-term modeling capability of three-dimensional convolutional neural networks is slightly insufficient. The size of the three-dimensional convolution kernel is generally set at the beginning of network design. The convolution kernel cannot perceive video data outside its receptive field. If the behavior occurs over a long period of time, the convolutional neural network will not be able to perceive the global spatiotemporal information and misidentify the behavior. In addition, actions with similar visual appearance will produce different labels due to the speed of the action, resulting in a decrease in the accuracy of the algorithm in the behavior recognition task. For example, the three behaviors of walking, jogging and running are very similar in morphology, but the speed of the action is very different. Convolutional neural networks can complete the recognition of behaviors with different movement speeds by adding input sources with different sampling rates, but at the same time, each data input source needs to correspond to a network branch, so the computing resources required for the model are also doubled.

[0005] In summary, although an effective end-to-end convolutional neural network method has been invented in the field of behavior recognition, the problems of long-term video modeling and the resolution of different motion rates have not yet been well solved.

[0006] The main problems with current mainstream behavior recognition technology are:

[0007] First, 3D convolutional neural networks can accomplish high-precision behavior recognition tasks. However, this type of method is limited by the size of the convolution kernel and does not fully consider the long-term temporal modeling problem in behavior recognition. Therefore, it often produces misidentification results for samples with long behavior cycles. Specifically, the convolution kernel in the convolutional neural network cannot be adaptively adjusted according to the input data, and the convolution kernel cannot process video data outside the receptive field. In addition, simply increasing the receptive field of the convolution kernel will not only increase the consumption of computing resources, but also reduce the accuracy of behavior recognition due to the collection of too much redundant information. Therefore, the long-term modeling problem of 3D convolutional neural networks needs to be solved.

[0008] Secondly, the 3D convolutional network is easily confused for behavior labels with similar visual appearance but different movement speeds. The reason is that for the pre-defined network structure, the network can only process actions at the sampling rate in the input data. However, if the input source is increased, that is, the video data is sampled at different sampling rates and input into the convolutional neural network for processing, although the accuracy of behavior recognition can be improved, the repeated input of multiple data will lead to an exponential increase in resource overhead.

[0009] The closest implementation scheme to the present invention is the Chinese invention patent entitled "A method for human behavior recognition based on multi-scale spatiotemporal graph convolutional neural network", application date 2021.09.01, application number 202111020275.8. In this method, a deep neural network model including a multi-scale graph convolution module and a multi-time feature fusion module is created. The deep neural network model used is the basic structure of the ST-GCN neural network model as the backbone network, and the graph convolution network in ST-GCN is transformed into a multi-scale graph convolutional network module, and the temporal convolution network is transformed into a multi-time feature fusion module, so that the network can extract the spatiotemporal features of the skeletal sequence to be identified.

[0010] The multi-time feature fusion module is used to extract the temporal features of the human skeleton sequence. It is a module composed of four parallel branches of temporal convolutional networks with different convolution kernel sizes and one attention module branch. The sizes of the convolution kernels of the four temporal convolutional network branches with different receptive fields are 3x1, 5x1, 7x1, and 9x1 respectively. Different receptive fields correspond to features of different time lengths. Another attention module branch uses a compression and excitation network (Square Excitation Networks). The output of the spatial multi-scale module is globally averaged pooled, passed through a fully connected layer, and finally the probability of the output of the Softmax function is applied to the four temporal convolutional network branches respectively, and the feature maps output by the four branches are spliced. Use the created dataset to train and test the deep convolutional neural network model; use the trained model to classify the video images to be recognized.

[0011] In view of the above-mentioned deficiencies in the prior art, the present invention is proposed. Summary of the invention

[0012] The purpose of the present invention is to address the deficiencies in the prior art. The present invention innovatively proposes a behavior recognition method and device based on multi-level feature fusion, redesigns the three-dimensional neural network structure, designs a time series feature compression excitation mechanism and a multi-level feature fusion method, and improves the model's long-term time series modeling capability and the ability to distinguish different motion rates. First, the features of different levels are extracted, and the global spatiotemporal information is extracted through a compression excitation module, and important time series segments are enhanced in a targeted manner, thereby expanding the receptive field of the three-dimensional convolutional neural network. Secondly, the perception of motion rate by features at different levels is utilized to fuse multi-level features, enabling the model to obtain more accurate behavior recognition capabilities. Behavior recognition of different motion rates under a single input source is achieved.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] A behavior recognition method based on multi-level feature fusion includes the following steps:

[0015] Step S1: Collection of multi-level features;

[0016] Construct a three-dimensional convolutional neural network model to extract features at different levels in the behavior recognition network and collect multi-level features;

[0017] Step S2: significant expression of temporal features;

[0018] After the operation of step S1, a three-dimensional feature map containing spatiotemporal information is obtained;

[0019] Through the time series feature saliency expression method, global information is obtained, and the feature saliency expression is completed and long time series modeling is realized by compressing and exciting the time series dimension features, including:

[0020] Step S2-1: compression;

[0021] Compress the overall feature map to the time series dimension before performing the excitation operation, so that the time series information does not interfere with each other before the interaction of this layer;

[0022] Characteristics of this layer Perform average pooling operation to obtain independent feature expressions at the time series level, and the generated features Then proceed with feature excitation to achieve global timing feature modeling;

[0023]

[0024] In the formula, Γ is a reshape operation, and an average pooling operation is performed after swapping the time dimension and channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, i and j are the spatial positions of the feature points, and the compressed time feature expression of the frame is obtained by swapping the channel and time dimension of the sampled features and then pooling them;

[0025] Step S2-2: incentive;

[0026] The time series excitation operation uses the time series features obtained by compression to control the global weight and learn the contextual relationship of the time series dimension, which can better help to mine representative time series feature expressions in the final prediction;

[0027] The activation operation is achieved by using ReLU, Sigmoid and fully connected layers:

[0028] M=f ex (F Squeeze )=σ(W 2 δ(W 1 F Squeeze ))

[0029] Where σ is the Sigmoid function, σ is the ReLU function,

[0030] In order to obtain the saliency map M, the excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map

[0031] Step S2-3: global timing mapping;

[0032] Add the mapping to the entire feature, and broadcast the mapping to obtain the stimulated feature;

[0033] Map after broadcast The feature map is recalculated through element-level multiplication to complete the temporal excitation operation of the feature:

[0034] In the formula, Fan broadcasts the input mapping map M to expand it into a mapping map of the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication. base Get the feature F after excitation ex .

[0035] F ex =Fan(M)×F base

[0036] Step S3: multi-level feature fusion method;

[0037] After obtaining the global features after excitation, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is based on the comprehensive features at different levels and different motion speeds. The specific steps are as follows:

[0038] Step S3.1, perform average pooling operation on different features to adjust the feature space size of different levels to the same as the feature size of the last layer;

[0039] Step S3.2, after the average pooling operation, the features at different levels are concatenated in the channel dimension to summarize the features of all levels in the pyramid structure;

[0040] Step S3.3, fusion;

[0041] Using the three-dimensional convolution kernel, the spatiotemporal features of the features after splicing at different levels are mined, and the fusion feature F is corrected. fusion The number of channels;

[0042] The fusion feature size and number of channels are aligned with the output feature size of the last layer of the backbone network;

[0043] The fusion method is calculated using the following formula;

[0044] in It is the feature after the n-th level feature is stimulated. There are 4 layers of features in total, and n is 4;

[0045] Conv is the convolution operation;

[0046] Pool is an average pooling operation, which aligns the features of different levels in spatial dimensions;

[0047] Concat performs channel-level concatenation operations to concatenate features at different levels, and obtains the feature F after concatenation, which is a fusion of multiple levels of features. mff ;

[0048]

[0049] Step S4: Output the target task result.

[0050] The preferred technical solution provided by the present invention is:

[0051] The three-dimensional convolutional neural network model includes a three-dimensional convolutional layer module, a batch normalization module, a ReLU activation layer module, a direct mapping module, and a skip mapping module, wherein:

[0052] The three-dimensional convolutional layer module can simultaneously extract spatial information and temporal information from video data, laying a good foundation for subsequent global processing and multi-level feature fusion;

[0053] The batch normalization module and the ReLU activation layer module can improve the learning ability of the network and prevent gradient explosion and overfitting;

[0054] The direct mapping module and the skip mapping module can ensure that the generalization ability of the three-dimensional convolutional neural network model is maintained when the model depth increases, and increase the number of parameters of the three-dimensional convolutional neural network model, thereby adapting to more complex tasks.

[0055] Beneficial effects:

[0056] The present invention simultaneously considers for the first time the problem of temporal receptive field priority and the problem of weak ability to distinguish different motion rates in three-dimensional convolutional neural networks, so that the three-dimensional convolutional neural network can effectively improve the accuracy of behavior recognition while maintaining little change in computational overhead.

[0057] The present invention designs a significant expression mechanism for temporal features. By performing significant expression processing on features, global spatiotemporal information is obtained to enhance the long-term temporal modeling capability of the model. First, the features of different levels in the three-dimensional convolutional neural network are extracted, and the temporal self-attention maps are calculated respectively and mapped to the original features to complete the excitation operation. A multi-level feature fusion module is designed to utilize the perception of motion rate by features of different levels to enhance the resolution of the model and realize the behavior recognition capability of different motion rates under a single input source.

[0058] The present invention designs a behavior recognition method and device based on multi-level feature fusion, extracts global spatiotemporal information and multi-level feature fusion, improves the long-term modeling capability of the entire model and the ability to distinguish different action rates. When the model is maintained with a single input source, the computational overhead is maintained without a significant increase, and the behavior recognition accuracy of the model is improved.

[0059] Terminology explanation:

[0060] Deep learning network: A deep learning network is a machine learning model that consists of multiple layers of neurons. Each layer is connected to the neurons in the previous and next layers to learn the mapping relationship from input to output.

[0061] Multi-level features: The "depth" in deep learning networks refers to the number of levels in the network. Generally speaking, networks with more levels can learn more complex features and patterns. Multi-level features refer to the features of different depths extracted during the process of neural network mining information.

[0062] Behavior recognition: Behavior recognition refers to the use of data and machine learning techniques to identify and classify behavioral patterns or actions of humans or other entities. This usually involves collecting data using sensors (such as cameras, accelerometers, gyroscopes, etc.) and inputting it into a deep learning model for analysis and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0064] Figure 1 A processing flow chart of a behavior recognition method and device based on multi-level feature fusion provided by an embodiment of the present invention;

[0065] Figure 2 It is a specific instantiation structure diagram of the multi-level feature fusion network model in the present invention;

[0066] Figure 3 It is a specific instantiation structure diagram of the timing feature compression excitation module in the present invention;

[0067] Figure 4 It is a specific instantiation structure diagram of unified excitation and group excitation in the time series feature compression excitation module of the present invention;

[0068] Figure 5 This is a specific instantiation structure diagram of multi-level feature fusion. DETAILED DESCRIPTION

[0069] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0070] The present invention proposes a behavior recognition method and device based on multi-level feature fusion. The processing flow chart of the method is as follows: Figure 1 As shown, it specifically includes the following steps:

[0071] Step S1: Multi-level feature collection.

[0072] While ensuring the model's learning ability, the cost of computing and storage resources is reduced as much as possible. The specific structure is as follows Figure 2 shown.

[0073] Taking the 3d-ResNet network as an example, the multi-level feature collection method is based on the residual block output, and a set of feature fusion modules with 4-level feature layers are designed. These features are from shallow to deep, and the temporal and spatial receptive fields gradually increase. The length of the temporal dimension does not change, and the size of the spatial dimension will gradually decrease.

[0074] In order to keep the spatial semantic expression as much as possible, the space will not be downsampled in the step before feature fusion, so the collected 4-level features will be composed of {C 1 ×T×H 1 ×W 1 ,...,C 4 ×T×H 4 ×W 4 Features of {F 1 ,F 2 ,...,F 4 Specifically, T is set to 8, that is, the length of the time series dimension in the feature is 8; H 1 , W 1Set it to 28, and each time the depth increases by one layer, the width and height are reduced to half of the original, that is, the width and height of each layer of features thereafter are 14, 7, and 4 respectively;

[0075] C 1 Set to 256, the number of channels doubles with each additional layer of depth, that is, the number of channels in each layer is 256, 1024, and 2048. The multi-level pyramid features collected in this way contain richer semantic expressions in the spatial dimension, but attention should be paid to the subsequent feature fusion to ensure that the information flow during fusion can be correctly transmitted.

[0076] Step S2: Method for expressing the significance of temporal features

[0077] After the operation of step S1, the present invention can obtain a three-dimensional feature map containing rich spatiotemporal information. However, the model is limited by the three-dimensional convolution kernel, and its long-term time series modeling capability is lacking. Therefore, the present invention designs a temporal feature saliency expression (STE) method to obtain global information, and completes feature saliency expression and realizes long-term time series modeling by compressing and exciting the temporal dimension features. The specific structure is as follows Figure 3 shown.

[0078] Step S2-1: Compression

[0079] The present invention proposes to compress the overall feature map into the time series dimension before performing the excitation operation, so that the time series information will not interfere with each other before the interaction of the current layer. Perform average pooling operation to obtain independent feature expressions at the time series level, and the generated features Then proceed with feature excitation to achieve global timing feature modeling.

[0080]

[0081] In the formula, Γ is a reshape operation, and an average pooling operation is performed after swapping the temporal dimension and the channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, and i and j are the spatial positions of the feature points. The sampled features are swapped in the channel and temporal dimensions and then pooled to obtain the compressed temporal feature expression of the frame.

[0082] Step S2-2: Incentive

[0083] The time series excitation operation controls the global weight and learns the contextual relationship of the time series dimension by utilizing the time series level features obtained by compression, which can better help to mine representative time series feature expressions in the final prediction.

[0084] To achieve these goals, the present invention implements the excitation operation by using ReLU, Sigmoid and fully connected layers:

[0085] M=f ex (F Squeeze )=σ(W 2 δ(W 1 F Squeeze ))

[0086] Where σ is the Sigmoid function, σ is the ReLU function, In order to obtain the saliency map M. The excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map

[0087] Step S2-3: Global Timing Mapping

[0088] In order to add the mapping to the entire feature, the mapping is broadcasted to obtain the stimulated feature.

[0089] Because the mapping in the saliency expression module is calculated for the temporal dimension, the spatial information is compressed into the temporal dimension during the calculation process. Similarly, during the excitation stage, this temporal mapping can also be applied to the entire spatial dimension and channel dimension.

[0090] Map after broadcast The feature map is recalculated through element-level multiplication to complete the temporal excitation operation of the feature.

[0091] F ex =Fan(M)×F base

[0092] In the formula, Fan broadcasts the input mapping map M to expand it into a mapping map of the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication. base Get the feature F after excitation ex .

[0093] Depending on the complexity of the scene and action classification, the timing mapping methods under different actions may be completely different.

[0094] In order to make the model more robust, a grouping operation is added to the STE module to enhance the adaptability of the temporal saliency expression module to video data with different actions and backgrounds. The study hopes to enable the model to adaptively stimulate different actions through multiple groupings.

[0095] Specifically, when the features are input into the STE module, the STE module will split the feature map into 4 groups along the channel dimension, and the split group features will each enter their respective independent excitation layers for saliency excitation at the temporal level, as Figure 4 shown.

[0096] Step S3: Multi-level feature fusion method

[0097] After obtaining the excited global features, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is comprehensively obtained based on the features at different levels and different motion speeds. The fusion stage process is as Figure 5 shown.

[0098] To complete comprehensive feature fusion, first perform average pooling operations on different features to adjust the feature space sizes at different levels to be the same as the size of the features in the last layer. This can facilitate the calculation of the fully connected layer, and since the size of the features in the last layer is the smallest in the overall backbone network, it can reduce the computational overhead.

[0099] There are two reasons for choosing pooling operations instead of convolutional operations:

[0100] One is that the feature sizes at different levels are different, and through pooling operations, the morphological differences between features at different levels can be eliminated, preparing for subsequent concatenation along the channel dimension;

[0101] The second is that pooling operations can save computational resources while maintaining the model performance.

[0102] The features output by the saliency expression module are where {C 1 ×T×H 1 ×W 1 ,..., C 4 ×T×H 4 ×W 4} and {Ci < Cj, Hi > Hj, Wi > Wj; i < j}. Average pooling is performed on the features before the 4th layer to align the spatial size of the features with the size of the features in the last layer, that is, the feature sizes of all levels are controlled to be 8×4×4.

[0103] Through the pooling operation on the feature map, the resources consumed by subsequent calculations can be reduced, and the global information obtained after excitation can be retained.

[0104] After performing the average pooling operation, concatenation operations are performed on the features at different levels along the channel dimension to summarize the features of all levels in the pyramid structure.

[0105] The features obtained after summarization Although the fused features eliminate the differences in spatiotemporal dimensions, the channel dimension is far behind the output of the last layer of the backbone network. Through experiments and observations, when the unbalanced backbone network features and the fused features are input into the fully connected layer together, the spatiotemporal information therein will interfere with each other, increasing the computational overhead and reducing the accuracy of behavior recognition.

[0106] To solve this problem, the present invention uses a three-dimensional convolution kernel to mine the spatiotemporal features of the features after splicing at different levels, and corrects the fusion feature F fusion The number of channels is used to maintain the balance of the backbone network as the feature fusion branch. After the above calculations, the fusion feature size and the number of channels are aligned with the last layer output feature size of the backbone network.

[0107] The fusion method is calculated using the following formula.

[0108]

[0109] The present invention simultaneously considers for the first time the problem of temporal receptive field priority and the problem of weak ability to distinguish different motion rates in three-dimensional convolutional neural networks, so that the three-dimensional convolutional neural network can effectively improve the accuracy of behavior recognition while maintaining little change in computational overhead.

[0110] The present invention designs a significant expression mechanism for temporal features. By performing significant expression processing on features, global spatiotemporal information is obtained to enhance the long-term temporal modeling capability of the model. First, the features of different levels in the three-dimensional convolutional neural network are extracted, and the temporal self-attention maps are calculated respectively and mapped to the original features to complete the excitation operation. A multi-level feature fusion module is designed to utilize the perception of motion rate by features of different levels to enhance the resolution of the model and realize the behavior recognition capability of different motion rates under a single input source.

[0111] The present invention designs a behavior recognition method and device based on multi-level feature fusion, extracts global spatiotemporal information and multi-level feature fusion, improves the long-term modeling capability of the entire model and the ability to distinguish different action rates. When the model is maintained with a single input source, the computational overhead is maintained without a significant increase, and the behavior recognition accuracy of the model is improved.

[0112] The terminal proposed in this embodiment and the method for behavior recognition proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to any of the above embodiments, and this embodiment has the same beneficial effects as executing the behavior recognition method.

[0113] In addition, an embodiment of the present application also proposes a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, on which a behavior recognition program is stored, and when the behavior recognition program is executed by a processor, the behavior recognition method of the present application as described above is implemented.

[0114] The various embodiments of the electronic device and computer-readable storage medium of the present application can all refer to the various embodiments of the behavior recognition method of the present application, which will not be repeated here.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0116] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling an electronic device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0118] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A behavior recognition method based on multi-level feature fusion, characterized in that: The following steps are involved: Step S1: Collection of multi-level features; Construct a three-dimensional convolutional neural network model to extract features at different levels in the behavior recognition network and collect multi-level features; Step S2: significant expression of temporal features; After the operation of step S1, a three-dimensional feature map containing spatiotemporal information is obtained; Through the time series feature saliency expression method, global information is obtained, and the feature saliency expression is completed and long time series modeling is realized by compressing and exciting the time series dimension features, including: Step S2-1: compression; Compress the overall feature map to the time series dimension before performing the excitation operation, so that the time series information does not interfere with each other before the interaction of this layer; Characteristics of this layer Perform average pooling operation to obtain independent feature expressions at the time series level, and the generated features Then proceed with feature excitation to achieve global timing feature modeling; In the formula, Γ is a reshape operation, and an average pooling operation is performed after swapping the time dimension and channel dimension in the feature map, where t is the feature of the tth frame in the entire sampling, i and j are the spatial positions of the feature points, and the compressed time feature expression of the frame is obtained by swapping the channel and time dimension of the sampled features and then pooling them; Step S2-2: incentive; The time series excitation operation uses the time series features obtained by compression to control the global weight and learn the contextual relationship of the time series dimension, which can better help to mine representative time series feature expressions in the final prediction; The activation operation is achieved by using ReLU, Sigmoid and fully connected layers: M=f ex (F Squeeze )=σ(W2δ(W1F Squeeze )) where σ is the Sigmoid function, σ is the ReLU function, In order to obtain the saliency map M, the excitation operation uses less computation to infer the salient action area of ​​the video segment, and finally outputs the map Step S2-3: global timing mapping; Add the mapping to the entire feature, and broadcast the mapping to obtain the stimulated feature; Map after broadcast The feature map is recalculated through element-level multiplication to complete the temporal excitation operation of the feature: F ex =Fan(M)×F base The specific calculation method is as follows: Fan broadcasts the input mapping map M to expand it into a mapping map of the same size as the feature map, and multiplies it with the original feature map F through element-wise multiplication base Get the feature F after excitation ex . Step S3: multi-level feature fusion method; After obtaining the global features after excitation, it is necessary to perform a fusion operation on the features of the pyramid structure to ensure that the final prediction result is based on the comprehensive features at different levels and different motion speeds. The specific steps are as follows: Step S3.1, perform average pooling operation on different features to adjust the feature space size of different levels to the same as the feature size of the last layer; Step S3.2, after the average pooling operation, the features at different levels are concatenated in the channel dimension to summarize the features of all levels in the pyramid structure; Step S3.3, fusion; Using the three-dimensional convolution kernel, the spatiotemporal features of the features after splicing at different levels are mined, and the fusion feature F is corrected. fusion The number of channels; The fusion feature size and number of channels are aligned with the output feature size of the last layer of the backbone network; The fusion method is calculated using the following formula; in It is the feature after the n-th level feature is stimulated. There are 4 layers of features in total, and n is 4; Conv is the convolution operation; Pool is an average pooling operation, which aligns the features of different levels in spatial dimensions; Concat performs channel-level concatenation operations to concatenate features at different levels, and obtains the feature F after concatenation, which is a fusion of multiple levels of features. mff ; Step S4: Output the target task result.

2. The behavior recognition method based on multi-level feature fusion according to claim 1 is characterized in that: The three-dimensional convolutional neural network model includes a three-dimensional convolutional layer module, a batch normalization module, a ReLU activation layer module, a direct mapping module, and a skip mapping module, wherein: The three-dimensional convolutional layer module can simultaneously extract spatial information and temporal information from video data, laying a good foundation for subsequent global processing and multi-level feature fusion; The batch normalization module and the ReLU activation layer module can improve the learning ability of the network and prevent gradient explosion and overfitting; The direct mapping module and the skip mapping module can ensure that the generalization ability of the three-dimensional convolutional neural network model is maintained when the model depth increases, and increase the number of parameters of the three-dimensional convolutional neural network model, thereby adapting to more complex tasks.

3. An electronic device, characterized in that: include: A memory, a processor, and a behavior recognition program stored in the memory and executable on the processor, wherein the behavior recognition program, when executed by the processor, implements the behavior recognition method as claimed in any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a behavior recognition program, and when the behavior recognition program is executed by a processor, the behavior recognition method according to any one of claims 1 to 2 is implemented.

Images