A method for animal behavior recognition based on multi-cue fusion

Abstract

The present invention provides an animal behavior recognition method based on multi-cue fusion, including: constructing a behavior data set; extracting an optical flow image of animal movement from an RGB frame; inputting the RGB frame and the optical flow frame into an animal behavior recognition model constructed by a multi-cue fusion network to recognize the animal behavior, including: extracting features from the RGB frame and the optical flow frame using a dual-stream network to obtain the apparent features and motion features of the animal behavior; performing multi-cue feature exchange on the above features, and discarding feature maps that have no influence on the final result; inputting the dual-stream features of the MFSM module into the AFFM module, performing adaptive feature fusion on the apparent features and the optical flow features, and obtaining apparent features enhanced by motion information; inputting the fused features into a backbone network, and obtaining the final behavior prediction result through a classifier. The method can reliably, automatically, and efficiently complete the recognition of animal behavior, and largely solve the problems caused by manual labeling.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an animal behavior recognition method based on multi-cue fusion. Background Art

[0002] Animal behavior recognition is a technology that identifies and analyzes animal behavior by analyzing video and other data. It is widely used in intelligent monitoring, behavior warning, medical health and other fields.

[0003] At present, the public behavior recognition methods can be divided into two categories: methods based on hand-designed features and methods based on deep learning. Methods based on hand-designed features often use template matching to convert a motion image sequence into one or a group of static templates, and obtain recognition results by matching the template of the sample to be recognized with the known template. Its main methods include: motion energy image (MED) and motion history image (MHI), mean motion shape (MMS) based on contour and mean motion energy (AME) based on motion foreground. The deep learning-based methods generally extract the behavior appearance features of each frame through convolutional neural network, and then fuse the multi-frame information to obtain the recognition result. For example, based on the two-stream convolutional neural network, the static branch is used to extract the behavior appearance features from the RGB video frame, and the dynamic branch is used to extract the dynamic information from the video optical flow frame. Then, the behavior predictions of the two branches are fused to obtain the classification result. For example, the convolutional neural network is cascaded with the long short-term memory network to form a recurrent convolution structure. At each step, the convolutional neural network extracts the features of each frame and then sends them to the long short-term memory network to model the temporal dynamic information of the behavior. For example, the method based on 3D convolutional neural network directly uses a 3D convolution kernel to perform convolution operations along the spatiotemporal dimension of the video to obtain the spatiotemporal information of the behavior. For example, the SMART Block proposed by ARTNet extracts the behavior appearance features based on the two-dimensional convolution operation, and extracts the temporal features of the behavior in combination with the two-dimensional convolution operation to obtain the temporal information of the behavior.

[0004] Generally, hand-designed feature algorithms are greatly affected by complex backgrounds and work well on limited types, restricted scenes, and small-scale data sets, but cannot process large-scale real-world scene data sets. At the same time, hand-designed features have high computational complexity, which is not conducive to real-time processing. It is difficult to effectively capture the dynamic information of behavior based on deep learning methods, and it is impossible to use effective methods to guide convolutional neural networks to extract areas related to behavior. The above problems can be solved to a certain extent by extracting the convolutional layer features of RGB frames through neural networks, iteratively optimizing the behavioral features to obtain optical flow features, and inputting the optical flow features into the classifier for classification to identify animal behavior. However, when the dynamic features are complex due to rapid movement, occlusion, blur, etc., the accuracy of the recognition results of this method will drop significantly.

[0005] The basic principle of the two-stream model structure is to calculate the dense optical flow of every two frames in the video sequence to obtain time information. Then the CNN model is trained on video images, space and time, and the two branches of the network are used to judge the action category respectively. Finally, the training results of the two networks are directly fused to obtain the final classification result. The advantage of the two-stream convolutional network structure is high accuracy, but slow speed.

[0006] Feichtenhofer et al. follow the two-stream convolutional network structure. In order to better utilize the spatiotemporal information in the two-stream model, the author improved the spatiotemporal network fusion strategy. He proposed five different spatiotemporal network fusion schemes and three temporal network fusion methods. Wang et al. listed the accuracy of the two-stream network under several latest CNN network structures. Xiong Jun et al. found that previous research results can only handle short-term actions, lack of understanding of the temporal structure of long-term actions, and small training samples. To this end, the authors adopted a sparse temporal sampling strategy and a video monitoring strategy. The video is segmented in the time domain, and the first deficiency is compensated by randomly selected segments, and the second deficiency is compensated by cross-training, regularization and data expansion. The network structure is named Temporal Segmented Network (TSN). Due to the recent successful application of residual networks (ResNet) in deep learning, Feichtenhofer et al. proposed a new spatiotemporal residual network model that combines ResNet and the two-stream model. The temporal and spatial characteristics of the behavior are learned hierarchically through the residual connection between the spatial and temporal streams.

[0007] In the early stage of feature extraction, the present invention uses a two-stream network, combined with RGB images and optical flow images. The purpose is to use the complementary advantages of the two clues to identify mouse behavior and further improve the accuracy of prediction. After feature fusion, the two-stream network is converted into a single-stream network, which can effectively control the computational overhead and the number of parameters and improve the recognition performance.

[0008] In many works, the fusion of different modal features is an important way to improve accuracy. Combining the features of different modalities and taking advantage of their complementarity can achieve better recognition results. Shalawi et al. proposed a method to combine 2D shape human posture estimation with skeleton features. Fusion of effective 2D contour and 3D skeleton features can obtain visual features with high discriminative value, and use the additional discriminative data provided by the contour to improve the robustness of human action recognition errors. Sanchez-riera et al. combined RGB features with depth features for gesture recognition and general object recognition, and evaluated two fusion schemes, early and late. Li et al. proposed a multi-feature sparse fusion model to extract multiple features of human body parts from skeleton and depth data respectively. When sparse regularization technology is used to automatically identify the feature structure of key parts, the learned weighted features are more discriminative for multi-task classification. Chen et al. extracted deep image features and RGB video features of human actions based on depth cameras and inertial body sensors, respectively, and evaluated two recognition frameworks: feature-level fusion and decision-level fusion.

[0009] Mice are widely present in biomedical sciences, and their response to diseases or treatments is often detected by measuring their behavioral patterns. In most cases, videos are recorded and labeled manually. Since manually labeling animal behavior is a difficult task, there is a need for a reliable and automated behavior recognition system that completes the task through computers. Through a high-performance system, the problem of manual labeling can be solved and the labeling efficiency can be improved. In many studies on animal behavior recognition, some animal behavior recognition systems have been proposed. These systems are mainly divided into two types: traditional methods based on manual feature extraction and deep learning methods using neural networks. The method of building complex models based on manual feature expression can no longer meet the requirements of high precision and high speed, so deep learning technology has brought new development directions for animal behavior recognition.

[0010] In view of the problem that the existing technology has insufficient accuracy in animal behavior recognition and cannot effectively and automatically classify animal behavior under conditions such as rapid animal movement, occlusion, and blur, the present invention proposes an animal behavior recognition method based on multi-cue fusion. Summary of the invention

[0011] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an animal behavior recognition method based on multi-cue fusion to solve the problem that the existing recognition method has inaccurate recognition results when the animal moves quickly, is blocked or the imaging is blurred, so that the network can directly learn how to filter the features of different modalities, thereby retaining only the information that is useful for the combination. At each spatial position, the features of different modalities are adaptively fused, and some features may be filtered out because of the conflicting information at that position, while other features may dominate.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] An animal behavior recognition method based on multi-cue fusion includes the following steps:

[0014] Obtain animal behavior video data and extract RGB frames from the video;

[0015] Perform optical flow extraction on the RGB frame to obtain the optical flow frame of animal motion;

[0016] The RGB frame and the optical flow frame are input into the animal behavior recognition model constructed by the multi-cue fusion network to recognize the animal behavior. The recognition steps include:

[0017] Use a dual-stream feature extraction network to extract features from RGB frames and optical flow frames to obtain the apparent features and motion features of animal behavior;

[0018] The appearance features and motion features are input into the multi-cue feature exchange module MFSM for multi-cue feature exchange. If the scale factor of the feature map is less than the set threshold, the feature map will be replaced by other cues at the same position.

[0019] The output result of the multi-cue feature exchange module is input into the adaptive feature fusion module AFFM, and the apparent features after feature exchange and the optical flow features are adaptively fused to obtain the apparent features enhanced by motion information;

[0020] The output results of the adaptive feature fusion module are input into the backbone network, and the final behavior prediction results are obtained through the classifier;

[0021] According to the behavior prediction results and behavior categories, the loss value is calculated through the cross entropy loss function, the loss is back-propagated, and the network parameters are iteratively updated until the animal behavior recognition model converges. Behavior recognition is performed based on the converged animal behavior recognition model to obtain the recognition result.

[0022] Preferably, the multi-cue feature exchange module MFSM introduces a batch normalization layer, using x m,c represents the cth feature map in the mth feature network. In the batch normalization layer, x m,c Perform an affine transformation as follows:

[0023]

[0024] Among them, μ m,c and σ m,c They represent the mean and mean standard deviation of all pixel positions of feature map c in the mth feature network respectively; m,c and β m,care trainable scale factors and offsets; ε is a constant; x' m,c Represents x m,c The value obtained after performing an affine transformation;

[0025] Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps;

[0026] The appearance features and motion features are input into the multi-cue feature exchange module MFSM, and batch normalization is performed on them respectively to obtain the different γ m,c , γ m,c The role of x is to evaluate m,c and x' m,c and then determine whether to perform feature exchange.

[0027] Preferably, formula (2) is used to determine whether feature exchange is to be performed:

[0028]

[0029] Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps, m' represents another input feature, and M represents the number of input features;

[0030] If the scale factor γ of the feature map c m,c is greater than the threshold θ, then x' m,c is x m,c The value obtained after performing the affine transformation; if the scale factor γ of the feature map c m,c If it is less than the threshold θ, the current feature map c is replaced by the average value of the feature maps c of other feature networks. Since the input features only have appearance features and motion features, it is only necessary to use the feature maps of other feature networks for replacement to realize the exchange of multi-cue features.

[0031] Preferably, the fusion method of the adaptive feature fusion module AFFM is as follows:

[0032]

[0033] Among them, y ij is the tensor of the input feature at position (i, j) after weight calculation; α ij and β ij are two spatial importance weights for different cues obtained through network adaptive learning; α ij and βij Shared between feature maps; define α ij +β ij =1, and according to the following formula α ij ,β ij ∈[0,1],

[0034]

[0035] Among them, α ij and β ij Use and As a control parameter of the SoftMax function;

[0036] Use a 1×1 convolutional layer and a SoftMax function to calculate the 1 、x 2 The weight map α in ij , β ij , adaptive learning through the network;

[0037] In the network, and The inputs are the appearance features and optical flow features after feature exchange;

[0038] and Calculated by 1×1 convolution and Will and After splicing, the number of channels is changed again through 1×1 convolution calculation, and then the weight mapping parameter α is calculated using the SoftMax function. ij and β ij ;

[0039] The feature maps of appearance features and optical flow features are mapped to their respective weight parameters α at each position (i, j). ij and β ij The (i, j) of is multiplied and then added to get the final fusion result.

[0040] Preferably, the backbone network is InceptionV3, which includes:

[0041] The first part is the input layer, which has two input streams, namely RGB frame and optical flow frame. The input layer consists of six 2D convolution blocks, two maximum pooling layers and an adaptive feature fusion module. The 2D convolution block consists of a convolution layer, an activation layer, a batch normalization layer and a multi-cue feature exchange layer. The batch normalization layer obtains the parameters required by the multi-cue feature exchange layer and then performs multi-cue exchange. The output of the multi-cue exchange will be used as the input of the next layer.

[0042] The second part is the fusion layer, which consists of three InceptionA blocks, one InceptionC block, four InceptionB blocks, one InceptionD block and two InceptionE blocks. The input of these Inception blocks is the output of the adaptive fusion module. At the end of the fusion layer, these Inception blocks will be spliced ​​and input to the next layer.

[0043] The third part is the output layer, which consists of an average pooling layer and a fully connected layer to obtain the final classification result.

[0044] The animal behavior recognition method based on multi-cue fusion provided by the present invention has the following beneficial effects:

[0045] (1) The present invention greatly improves the recognition accuracy while slightly increasing the computational cost and the number of parameters, and can effectively and automatically complete the classification of animal behaviors.

[0046] (2) The new multi-cue exchange module proposed in the present invention can replace the feature graphs that have no influence on the final result with the feature graphs of other clues.

[0047] (3) The adaptive feature fusion module proposed in the present invention can adaptively fuse the features after feature exchange.

[0048] (4) The present invention proposes a multi-cue fusion network for animal behavior recognition, which uses a multi-cue exchange module and an adaptive feature fusion module.

[0049] (5) The present invention uses RGB and optical flow images as input, uses a dual-stream network and performs early fusion, which can effectively improve the recognition accuracy under the premise of slightly increasing the amount of calculation. The model of the present invention can reliably, automatically and efficiently complete the recognition of animal behavior, which can largely solve the problems caused by manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 It is a flow chart of the animal behavior recognition method based on multi-cue fusion of the present invention;

[0052] Figure 2 It is a schematic diagram of multi-cue fusion strategy;

[0053] Figure 3This is the overall architecture diagram of the multi-cue fusion network based on InceptionV3;

[0054] Figure 4 It is an experimental diagram of an embodiment of the present invention;

[0055] Figure 5 This is the distribution diagram of the number of frames in each behavior of the Jiang dataset;

[0056] Figure 6 This is the distribution diagram of the number of frames in each line of the Jhuang dataset. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0058] Example 1

[0059] The present invention provides an animal behavior recognition method based on multi-cue fusion, specifically Figure 1 As shown, including:

[0060] Step 1: Obtain animal behavior video data and extract RGB frames from the video;

[0061] Step 2: Use the optical flow extraction method to extract the optical flow of the RGB frame to obtain the optical flow frame of the animal movement;

[0062] Step 3: Input the RGB frame and the optical flow frame into the animal behavior recognition model constructed by the multi-cue fusion network to recognize the animal behavior. The recognition steps include:

[0063] Step 3.1, use a dual-stream feature extraction network to extract features from the RGB frame and the optical flow frame to obtain the apparent features and motion features of the animal behavior;

[0064] Step 3.2: Input the appearance features and motion features into the multi-cue feature exchange module MFSM to perform multi-cue feature exchange. If the scale factor of the feature map is less than the set threshold, the feature map will be replaced by other clues at the same position. The purpose of the multi-cue feature exchange module is to discard the feature map that has no influence on the final result.

[0065] Step 3.3, the output result of the multi-cue feature exchange module is input into the adaptive feature fusion module AFFM, and the apparent features after feature exchange and the optical flow features are adaptively fused to obtain the apparent features enhanced by motion information;

[0066] Step 3.4: Input the output result of the adaptive feature fusion module into the backbone network, and obtain the final behavior prediction result through the classifier;

[0067] Step 3.5: According to the behavior prediction results and behavior categories, the loss value is calculated through the cross entropy loss function, the loss is back-propagated, and the network parameters are iteratively updated until the animal behavior recognition model converges. Behavior recognition is performed based on the converged animal behavior recognition model to obtain the recognition result.

[0068] Since the multi-cue feature exchange module needs to use the BN layer, the present invention first introduces the BN layer. The function of the BN layer is to enhance the generalization ability and speed up the network training and convergence speed.

[0069] Specifically, in this embodiment, the multi-cue feature exchange module MFSM introduces a batch normalization layer and uses x m,c represents the cth feature map in the mth feature network. In the batch normalization layer, x m,c Perform an affine transformation as follows:

[0070]

[0071] Among them, μ m,c and σ m,c They represent the mean and mean standard deviation of all pixel positions of feature map c in the mth feature network respectively; m,c and β m,c are trainable scale factors and offsets; ε is a constant that prevents zero division; x' m,c Represents x m,c The value after performing an affine transformation.

[0072] Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps.

[0073] The appearance features and motion features are input into the multi-cue feature exchange module MFSM, and batch normalization is performed on them to obtain the different γ m,c , γ m,c The role of x is to evaluate m,c and x' m,c and then determine whether to perform feature exchange.

[0074] Factor γ m,c The role of x′ is to evaluate the training m,c With x m,c If γ m,c →0, loss gradient x m,cwill approach zero, which means that x m,c will lose its influence on the result, so x m,c will become redundant.

[0075] Therefore, this prompts the present invention to m,c Smaller feature maps are replaced with feature maps of other feature networks, because these feature maps will lose their influence on the results and become redundant feature maps. The present invention proposes the following formula (2) to solve this problem:

[0076] Formula (2) is used to determine whether feature exchange is to be performed:

[0077]

[0078] Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps, m' represents another input feature, and M represents the number of input features;

[0079] If the scale factor γ of the feature map c m,c is greater than the threshold θ, then x' m,c is x m,c The value obtained after performing the affine transformation; if the scale factor γ of the feature map c m,c If it is less than the threshold θ, the current feature map c is replaced by the average value of the feature maps c of other feature networks. Since the input features only have appearance features and motion features, it is only necessary to use the feature maps of other feature networks for replacement to realize the exchange of multi-cue features.

[0080] In formula (2)(2), if the scaling factor γ of the feature map c is m,c If the feature map c is less than the threshold θ, the current feature map c is replaced by the average of the feature maps c of other feature networks. That is, if a feature map of a clue loses its influence on the result, it is replaced by the average of the feature maps of other feature networks. In the implementation of the present invention, the present invention applies the above formula to the feature extraction process and switches the feature map for each clue after convolution and nonlinear activation. The present invention represents the scale factor to be switched as γ′ m,c , and for γ′ m,c Sparse constraints are imposed to avoid unnecessary switching. This not only replaces useless feature maps, but also avoids the occurrence of useless switching. Multi-cue fusion strategies such as Figure 2 As shown in Figure 2. The sparsity constraint on the scaling factor is applied to the disjoint regions of different cues. If the scaling factor of a feature map is less than a certain threshold, the feature map will be replaced by other cues at the same position.

[0081] The adaptive feature fusion module (AFFM) proposed in this invention is different from the previous multi-level feature fusion methods based on element summation or splicing. The core idea of ​​this invention is to adaptively learn the spatial weights of feature map fusion at each scale. is the feature vector at position (i, j) in the mth feature network. The feature fusion method proposed in the present invention is as follows:

[0082] The fusion method of the adaptive feature fusion module AFFM is as follows:

[0083]

[0084] Among them, y ij is the tensor of the input feature at position (i, j) after weight calculation; α ij and β ij are two spatial importance weights for different cues obtained through network adaptive learning; α ij and β ij are shared between feature maps, they are not simple scalar variables. This invention forces the definition of α ij +β ij =1, and according to the following formula α ij ,β ij ∈[0,1],

[0085]

[0086] Among them, α ij and β ij Use and As a control parameter of the SoftMax function.

[0087] The present invention uses a 1×1 convolutional layer and a SoftMax function to calculate the 1 、x 2 The weight map α in ij , β ij , so that adaptive learning can be performed through the network.

[0088] In the network, and The inputs are the appearance features and optical flow features after feature exchange;

[0089] and Calculated by 1×1 convolution and Will and After splicing, the number of channels is changed again through 1×1 convolution calculation, and then the weight mapping parameter α is calculated using the SoftMax function.ij and β ij ;

[0090] The feature maps of appearance features and optical flow features are mapped to their respective weight parameters α at each position (i, j). ij and β ij The (i, j) of is multiplied and then added to get the final fusion result.

[0091] The multi-cue fusion network uses MFSM and AFFM to exchange feature maps and adaptively fuses the features of the two cues to give full play to the complementary advantages of the two cues. The network is used to identify the output features. In this way, the output of GSM can be viewed from the perspective of time and space, channel interdependence, and motion perception information. Figure 2 Shows the multi-cue fusion network structure used for InceptionV3.

[0092] In this embodiment, InceptionV3 is used as the backbone network, which consists of the following parts:

[0093] The first part is the input layer, which has two input streams, namely RGB frames and optical flow frames; the input layer consists of six 2D convolution blocks, two maximum pooling layers and an adaptive feature fusion module; the 2D convolution block consists of a convolution layer, an activation layer, a batch normalization layer and a multi-cue feature exchange layer. The batch normalization layer obtains the parameters required by the multi-cue feature exchange layer and then performs multi-cue exchange. The specific order can be seen from Table 2. The output of the multi-cue exchange will be used as the input of the next layer.

[0094] The second part is the fusion layer, which consists of three InceptionA blocks, one InceptionC block, four InceptionB blocks, one InceptionD block and two InceptionE blocks. The specific order of these Inception blocks can be seen in Table 2. The input of these Inception blocks is the output of the adaptive fusion module. At the end of the fusion layer, these Inception blocks will be spliced ​​and input to the next layer;

[0095] The third part is the output layer, which consists of an average pooling layer and a fully connected layer to obtain the final classification result. The specific order can be seen in Table 2.

[0096] The convolution kernel number, interval and output tensor size of the convolutional layer are detailed in Table 2.

[0097] Based on the above convolutional neural network structure, training it to obtain a recognition model can improve the recognition ability of the model and make it more robust. At the same time, it can reduce the requirements for calculation and storage space.

[0098] For example, two animal behavior datasets, Jiang Dataset and Jhuang Dataset, are used. The specific frame number distribution is given by Figure 5 and Figure 6 visible.

[0099] The normalization method used in this example is batch normalization.

[0100] The present invention uses TSN as the reference architecture for behavior recognition. TSN uses the C2D backbone to perform time pooling of frame-level features. The present invention chooses to use BN-Inception and InceptionV3 as the backbone options for TSN, but the present invention makes some modifications to the feature extraction part in front of the backbone. The present invention changes the input to two streams of RGB and optical flow input, and plugs in MFSM and AFFM. After that, the present invention plugs GSM into the backbone network. In all experiments, the present invention uses 16 frames as input. All programs are implemented in pytorch and run on a PC equipped with two NVIDIA Tesla T4 GPUs.

[0101] like Figure 4 As shown, this embodiment uses mice as the object and uses two data sets, namely the "Clip Database" of the Jhuang Dataset and the Jiang Dataset. The "Clip Database" includes 8 behavior categories: drink (drinking from the water supply), eat (getting food), groom (the mouse combs its fur), hang (the mouse hangs on the top of the cage), head (slight movement of the limbs or head), rear (standing with the forelimbs off the ground), rest (the mouse remains stable or sleeps), and walk (the mouse walks or runs in the cage). In addition to the above-mentioned public datasets, the Ji ang dataset is also used, which includes 6 behavior categories: dig (lifting sawdust with the forelimbs or head), eat (the mouse gets food from the food box), groom (the forelimbs sweep across the face or torso), rear (standing with the forelimbs off the ground), head (slight movement of the limbs or head), and walk (moving). The sample video frames of the Jiang dataset and the Jhuang database are shown as follows. Figure 5 and Figure 6 shown.

[0102] Table 1 Comparison of the accuracy of the proposed method on the Jiang dataset with other state-of-the-art methods

[0103]

[0104]

[0105] Table 2: Multi-cue fusion network implemented using Inceptionv3 architecture

[0106]

[0107] Table 1 shows the performance comparison of the multi-cue fusion network and the most advanced methods on the Jiang dataset. The best accuracy is the MGSN (Ours) method using InceptionV3 as the backbone network, followed by the MGSN (Ours) method using BN-Inception as the backbone network, and the accuracy of TDN (CVPR2021) ranks third. The present invention used 16 frames as input in the experiment. The table lists the comparison between the most advanced methods and the method of the present invention, and lists the accuracy using different backbones. Table 2 is the implementation of the multi-cue fusion network using the Inception v3 architecture. All convolution blocks are convolutional layers, ReLU nonlinearities, BN layers, and multi-cue feature exchange layers in sequence. C is the number of classes in the dataset. Table 2 shows the InceptionV3 architecture. Before using GSM, the present invention inserts MFSM in each Conv and performs feature adaptive fusion on RGB and optical flow patterns.

[0108] As can be seen from Table 1, the maximum absolute gain of the multi-cue fusion network over the baseline GSM is 5.41% (83.11% vs 77.70%). Under the same circumstances, the backbone network has different degrees of gain. The top three recognition accuracy in the table are multi-cue fusion networks using different modules. The experiment also shows that the recognition accuracy of InceptionV3, which is larger than BN-Inception, reaches 83.11%. The methods using the backbone as the ResNet50 network are TDN, PAN, ACTION-Net and TAM, and TDN has achieved good recognition accuracy. TDN, which can be used by ResNet-101, has achieved the highest accuracy before, reaching 81.31%, which is higher than the accuracy of GSM based on BN-Inception, 80.79%. However, the accuracy of MGSN (of the present invention) based on BN-Inception exceeds all previous methods, with an accuracy of 82.70%. The recognition accuracy of MGSN (of the present invention) based on InceptionV3 is further improved, exceeding 83.11% of all current methods.

[0109] The embodiments described above are only preferred specific implementation modes of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. An animal behavior recognition method based on multi-cue fusion, characterized in that: include: Obtain animal behavior video data and extract RGB frames from the video; Perform optical flow extraction on the RGB frame to obtain the optical flow frame of animal motion; The RGB frame and the optical flow frame are input into the animal behavior recognition model constructed by the multi-cue fusion network to recognize the animal behavior. The recognition steps include: Use a dual-stream feature extraction network to extract features from RGB frames and optical flow frames to obtain the apparent features and motion features of animal behavior; The appearance features and motion features are input into the multi-cue feature exchange module MFSM for multi-cue feature exchange. If the scale factor of the feature map is less than the set threshold, the feature map will be replaced by other cues at the same position. The output result of the multi-cue feature exchange module is input into the adaptive feature fusion module AFFM, and the apparent features after feature exchange and the optical flow features are adaptively fused to obtain the apparent features enhanced by motion information; The output results of the adaptive feature fusion module are input into the backbone network, and the final behavior prediction results are obtained through the classifier; According to the behavior prediction results and behavior categories, the loss value is calculated through the cross entropy loss function, the loss is back-propagated, and the network parameters are iteratively updated until the animal behavior recognition model converges. Behavior recognition is performed based on the converged animal behavior recognition model to obtain the recognition result.

2. The animal behavior recognition method based on multi-cue fusion according to claim 1, characterized in that: The multi-cue feature exchange module MFSM introduces a batch normalization layer and uses x m,c represents the cth feature map in the mth feature network. In the batch normalization layer, x m,c Perform an affine transformation as follows: Among them, μ m,c and σ m,c They represent the mean and standard deviation of all pixel positions of feature map c in the mth feature network respectively; m,c and β m,c are trainable scale factors and offsets; ε is a constant; x' m,c Represents x m,c The value obtained after performing an affine transformation; Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps; The appearance features and motion features are input into the multi-cue feature exchange module MFSM, and batch normalization is performed on them respectively to obtain the different γ m,c , γ m,c The role of x is to evaluate m,c and x' m,c and then determine whether to perform feature exchange.

3. The animal behavior recognition method based on multi-cue fusion according to claim 2 is characterized in that: Formula (2) is used to determine whether feature exchange is to be performed: Among them, x m,c The input is the tensor value of different feature maps of appearance features and motion features, m represents different input features, including appearance features and motion features; c represents different feature maps, m' represents another input feature, and M represents the number of input features; If the scale factor γ of the feature map c m,c is greater than the threshold θ, then x' m,c is x m,c The value obtained after performing the affine transformation; if the scale factor γ of the feature map c m,c If it is less than the threshold θ, the current feature map c is replaced by the average value of the feature maps c of other feature networks. Since the input features only have appearance features and motion features, it is only necessary to use the feature maps of other feature networks for replacement to realize the exchange of multi-cue features.

4. The animal behavior recognition method based on multi-cue fusion according to claim 3 is characterized in that: The fusion method of the adaptive feature fusion module AFFM is as follows: Among them, y ij is the tensor of the input feature at position (i, j) after weight calculation; α ij and β ij are two spatial importance weights for different cues obtained through network adaptive learning; α ij and β ij Shared between feature maps; define α ij +β ij =1, and according to the following formula α ij ,β ij ∈[0,1], Among them, α ij and β ij Use and As a control parameter of the SoftMax function; Use a 1×1 convolutional layer and a SoftMax function to calculate the 1 、x 2 The weight map α in ij , β ij , adaptive learning through the network; In the network, and The inputs are the appearance features and optical flow features after feature exchange; and Calculated by 1×1 convolution and λ αij and After splicing, the number of channels is changed again through 1×1 convolution calculation, and then the weight mapping parameter α is calculated using the SoftMax function. ij and β ij ; The feature maps of appearance features and optical flow features are mapped to their respective weight parameters α at each position (i, j). ij and β ij The (i, j) of is multiplied and then added to get the final fusion result.

5. The animal behavior recognition method based on multi-cue fusion according to claim 1, characterized in that: The backbone network is InceptionV3, which includes: The first part is the input layer, which has two input streams, namely RGB frame and optical flow frame. The input layer consists of six 2D convolution blocks, two maximum pooling layers and an adaptive feature fusion module. The 2D convolution block consists of a convolution layer, an activation layer, a batch normalization layer and a multi-cue feature exchange layer. The batch normalization layer obtains the parameters required by the multi-cue feature exchange layer and then performs multi-cue exchange. The output of the multi-cue exchange will be used as the input of the next layer. The second part is the fusion layer, which consists of three InceptionA blocks, one InceptionC block, four InceptionB blocks, one InceptionD block and two InceptionE blocks. The input of these Inception blocks is the output of the adaptive fusion module. At the end of the fusion layer, these Inception blocks will be spliced ​​and input to the next layer. The third part is the output layer, which consists of an average pooling layer and a fully connected layer to obtain the final classification result.

Images