A method and system for giant panda behavior recognition based on deep learning

Through the iterative fusion of features and scene information perceptron of deep learning methods, the problem of insufficient accuracy of giant panda behavior recognition in the existing technology is solved, and a more efficient behavior recognition effect is achieved.

CN118570877BActive Publication Date: 2024-11-26GUANGDONG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410884953.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-03
Publication Date
2024-11-26
Estimated Expiration
2044-07-03

AI Technical Summary

Technical Problem

The existing giant panda behavior recognition methods lack accuracy in background extraction and behavior recognition, making it difficult to effectively utilize scene information, resulting in insufficient recognition effect.

Method used

Using a deep learning-based method, the action and background features are extracted through the iterative fusion of the action model and the background model, combined with the scene information perceptron and Attention mechanism, and the action and background features are extracted, and the action and background features are weighted fusions are combined through multiple iterations to achieve the accuracy of giant panda behavior. Classification.

Benefits of technology

It improves the accuracy of giant panda behavior recognition, can better integrate background and action characteristics, and enhances the accuracy of behavior recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570877B_ABST
    Figure CN118570877B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of deep learning, and more specifically, to a method and system for giant panda behavior recognition based on deep learning. The method includes: obtaining a video to be detected, cutting frames of the video to be detected, and resampling; obtaining an action image to be recognized according to a frame difference method from the image to be detected as action understanding input information; setting the current iteration number i and the iteration standard, wherein the current iteration number i=1; constructing an action model and a background model; obtaining a bounding box indicating the position information of the giant panda; obtaining action features through the action model; obtaining background features through the background model: performing weighted fusion after iteration to obtain the final fusion features; and classifying the giant panda behavior according to the final fusion features. This method can better fuse background features and action features through multiple iterations and weighted fusion, so that the final giant panda behavior recognition effect is more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and more specifically, to a method and system for identifying giant panda behaviors based on deep learning. Background Art

[0002] Studying the behavior of giant pandas can help us better understand their living habits, social structure, and strategies for adapting to the environment, thereby providing a scientific basis and effective measures for their protection and survival. However, in actual conservation work, there are still some challenges in how to monitor and understand the behavior of giant pandas more accurately and efficiently. Traditional behavior monitoring methods rely on manual 24-hour recording, which has low work efficiency and high personnel training costs. Video-based automatic behavior recognition can achieve full-time, automated monitoring and statistics of giant panda behavior, improve work efficiency, help staff better understand the ecological habits and behavioral characteristics of giant pandas, and provide data support for the formulation of more scientific conservation strategies. At present, most behavior recognition methods only focus on motion features or appearance features, but ignore the important role of scene information in behavior recognition tasks.

[0003] The prior art discloses a method for tracking and analyzing the giant panda's pacing, bamboo eating, and estrus behaviors. The main purpose is to solve the problem that the giant panda's movement background is relatively complex and the traditional background extraction algorithm is difficult to achieve the ideal foreground target extraction effect. The main scheme includes step 1, inputting a giant panda video image, and extracting the foreground target from the video frame using the improved vibe method; step 2, performing morphological corrosion and expansion on the extracted foreground template; step 3, taking the minimum circumscribed rectangle of the contour with the largest area of ​​the connected area as the target area, and taking the centroid of the target area as the target location; step 4, performing the same operations as steps 1-3 on each frame of the image, outputting the giant panda's movement trajectory and movement speed, and analyzing the behavior. However, this method simply extracts the background and then further identifies the giant panda's behavior, and the identification of the giant panda's behavior is still not accurate enough. Summary of the invention

[0004] The purpose of the present invention is to disclose a giant panda behavior recognition method and system based on deep learning with better recognition effect.

[0005] In order to achieve the above object, the present invention provides a giant panda behavior recognition method based on deep learning, comprising:

[0006] S1: Obtain a video to be detected, cut frames of the video to be detected to obtain an image to be detected; resample the image to be detected to obtain a resampled image, and use the resampled image as target object guidance input information and background understanding input information; obtain an action image to be identified from the image to be detected according to a frame difference method as action understanding input information; set the current iteration number i and the iteration standard, where the current iteration number i=1; construct an action model and a background model;

[0007] S2: Obtaining a bounding box indicating the location information of the giant panda according to the target object guidance input information;

[0008] S3: The action understanding input information is passed through the action model to obtain action features; the background understanding input information is passed through the background model to obtain background features;

[0009] S4: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5;

[0010] S5: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and weightedly fusing the background features and the action features in the bounding box to obtain the final fused features;

[0011] S6: Classify giant panda behaviors based on the final fused features.

[0012] Furthermore, in step S1, after obtaining the video to be detected, data enhancement is first performed: the video to be detected is flipped horizontally and vertically, and flipped at random angles to obtain the video to be detected after data enhancement; then the video to be detected after data enhancement is cut into frames to obtain the image to be detected.

[0013] Furthermore, in step S2, there are a preliminary feature extraction stage, a candidate region extraction stage, a size fixing stage and a classification stage:

[0014] S2.1 Preliminary feature extraction stage: extract preliminary features based on the target object guided input information to obtain a preliminary feature map;

[0015] S2.2 Candidate region extraction stage: Generate multiple candidate regions based on the feature map;

[0016] S2.3 Fixed size stage: traverse each candidate region, keep the floating point number boundary unquantized; then divide each candidate region into 7×7 units, and the boundary of each unit is not quantized; in each unit, bilinear interpolation and maximum pooling are performed; finally, a candidate region of fixed size 7×7 is obtained;

[0017] S2.4 Classification stage: Classify the candidate area of ​​fixed size 7×7 into specific categories, and then perform bounding box regression on the candidate area of ​​fixed size 7×7 to obtain a bounding box indicating the location information of the giant panda.

[0018] Further, in step S3, the action understanding input information is passed through the action model to obtain action features, including: the action model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; the action understanding input information is passed through a Conv layer with a convolution kernel of 5×7×7 to extract preliminary action features, and the preliminary action features are passed through the scene information sensor to obtain action features; the action model includes a scene information sensor, a semantic feature extractor and an Attention mechanism; the semantic feature extractor is stacked with four layers, the first layer structure is composed of 3 3d residual blocks, the second layer structure is composed of 4 3d residual blocks, the third layer structure is composed of 6 3d residual blocks, and the fourth layer structure is composed of 3 3d residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3d residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: .

[0019] Further, in step S3, background features are obtained by passing the background understanding input information through the background model, including: the background model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; background features are extracted from the background understanding input information through a 1×7×7 Conv layer to obtain preliminary background features, and the preliminary background features are passed through the scene information sensor to obtain background features; the semantic feature extractor is a four-layer structure stacked, the first layer structure is composed of 3 3d residual blocks, the second layer structure is composed of 4 3d residual blocks, the third layer structure is composed of 6 3d residual blocks, and the fourth layer structure is composed of 3 3d residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3d residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: .

[0020] Furthermore, the scene information sensor is a Channel Context Perception Block structure: comprising a first branch and a second branch;

[0021] The first branch emphasizes the importance of different features in the receptive field. It first performs a 3×3×3 maximum pooling operation. The group convolution layer performs a group convolution operation on the input feature map, and then performs a 1×1×1 group convolution. Then, softmax is used to emphasize the importance of each feature in the receptive field feature. The second branch learns the spatial feature information in the receptive field, uses a 3×3×3 group convolution operation, and then performs a dot product operation on the first branch and the second branch to re-weight the feature map {30, h, w, 27c} ({time, height, width, channel}), which is then reorganized into {90, 3h, 3w, c}. Then, channel attention is used to focus on the key channels of the action features in the image sequence. Finally, a 3×3×3 convolution is performed to obtain the feature map {30, h, w, c}.

[0022] Furthermore, the scene information sensor is a Spatial Context Perception Block structure: comprising a first branch and a second branch;

[0023] The first branch first uses the average pooling operation to retain the background information, then uses 1×1×1 grouped convolution for information interaction, and uses softmax to emphasize the importance of each feature in the receptive field feature. The second branch uses 1×3×3 grouped convolution to learn the spatial feature information in the convolution kernel and generate {1,h,w,27c}, which is also transformed into {3,3h,3w,c} through dot product and reorganization operations. Then, the spatial attention mechanism is used to learn the weights of different positions, and finally 3×3×3 convolution is performed to obtain {1,h,w,c}.

[0024] Furthermore, the attention mechanism is used to learn the importance of features in different channels and positions;

[0025] The Group Convolution layer performs group convolution operations on the input information. The T×K×K group convolution divides the input channels into several groups, and then performs convolution calculations on the channels in each group.

[0026] Further, in step S5, it includes: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, obtaining the fusion weights through the background features and the action features in the bounding box, selectively suppressing or enhancing the features before fusion: interpolating the background features, and then concatenating the background features and the action features in the bounding box to obtain the features {30,1,1,4096}, then learn the features {30,1,1,4096} to get the fusion weight; perform softmax operation on the fusion weight to get the normalized fusion weight [ , ], according to the normalized fusion weight [ , ] Linearly weighted fusion of action features and background features within the bounding box is performed to obtain the final fused features.

[0027] In addition, the present invention also provides a giant panda behavior recognition system based on deep learning, comprising:

[0028] Acquisition module: acquire the video to be detected, cut frames of the video to be detected to obtain the image to be detected; resample the image to be detected to obtain the resampled image, and use the resampled image as the target object guidance input information and background understanding input information; obtain the action image to be identified according to the frame difference method of the image to be detected as the action understanding input information; set the current iteration number i and the iteration standard, where the current iteration number i=1; build the action model and the background model;

[0029] Bounding box module: obtains a bounding box indicating the location of the giant panda based on the target object guidance input information;

[0030] Feature module: the action understanding input information is passed through the action model to obtain action features; the background understanding input information is passed through the background model to obtain background features;

[0031] Iteration module: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5;

[0032] Feature fusion module: obtain the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and perform weighted fusion of the background features and the action features in the bounding box to obtain the final fusion features;

[0033] Classification module: classify the giant panda behavior based on the final fusion features.

[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0035] This method determines whether the current iteration number reaches the iteration standard. If not, the background features and action features are fused to obtain fused features, and the fused features are used as input information for the action understanding module; the background features are used as background understanding input information, and the current iteration number is updated and iterated again; if it is reached, the background features and action features are weightedly fused to obtain the final fused features; through multiple iterations and weighted fusion, the background features and action features can be better fused, making the final giant panda behavior recognition effect more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a method for identifying giant panda behavior based on deep learning as described in Example 1;

[0037] Figure 2 A block diagram of a giant panda behavior recognition system based on deep learning as described in Example 3;

[0038] Figure 3 This is a structural diagram of the Channel Context Perception Block described in Example 3;

[0039] Figure 4 This is a structural diagram of the Spatial Context Perception Block described in Example 3; DETAILED DESCRIPTION

[0040] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0041] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0042] Embodiment 1:

[0043] The present invention provides Figure 1 A giant panda behavior recognition method based on deep learning is shown, comprising:

[0044] S1: Obtain a video to be detected, cut frames of the video to be detected to obtain an image to be detected; resample the image to be detected to obtain a resampled image, and use the resampled image as target object guidance input information and background understanding input information; obtain an action image to be identified from the image to be detected according to a frame difference method as action understanding input information; set the current iteration number i and the iteration standard, where the current iteration number i=1; construct an action model and a background model;

[0045] S2: Obtaining a bounding box indicating the location information of the giant panda according to the target object guidance input information;

[0046] S3: The action understanding input information is passed through the action model to obtain action features; the background understanding input information is passed through the background model to obtain background features;

[0047] S4: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5;

[0048] S5: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and weightedly fusing the background features and the action features in the bounding box to obtain the final fused features;

[0049] S6: Classify giant panda behaviors based on the final fused features.

[0050] This embodiment determines whether the current iteration number reaches the iteration standard. If not, the background features and action features are fused to obtain fused features, and the fused features are used as input information for the action understanding module; the background features are used as background understanding input information, and the current iteration number is updated and iterated again; if it is reached, the background features and action features are weightedly fused to obtain the final fused features; through multiple iterations and weighted fusion, the background features and action features can be better fused, so that the final giant panda behavior recognition effect is more accurate.

[0051] Embodiment 2:

[0052] This embodiment further discloses on the basis of the first embodiment:

[0053] In step S1, after obtaining the video to be detected, data enhancement is first performed: the video to be detected is flipped horizontally and vertically, and flipped at random angles to obtain the video to be detected after data enhancement; then the video to be detected after data enhancement is cut into frames to obtain the image to be detected.

[0054] In step S2, there are preliminary feature extraction stage, candidate region extraction stage, size fixing stage and classification stage:

[0055] S2.1 Preliminary feature extraction stage: extract preliminary features based on the target object guided input information to obtain a preliminary feature map;

[0056] S2.2 Candidate region extraction stage: Generate multiple candidate regions based on the feature map;

[0057] S2.3 Fixed size stage: traverse each candidate region, keep the floating point number boundary unquantized; then divide each candidate region into 7×7 units, and the boundary of each unit is not quantized; in each unit, bilinear interpolation and maximum pooling are performed; finally, a candidate region of fixed size 7×7 is obtained;

[0058] S2.4 Classification stage: Classify the candidate area of ​​fixed size 7×7 into specific categories, and then perform bounding box regression on the candidate area of ​​fixed size 7×7 to obtain a bounding box indicating the location information of the giant panda.

[0059] In step S3, the action understanding input information is passed through the action model to obtain action features, including: the action model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; the action understanding input information is passed through a Conv layer with a convolution kernel of 5×7×7 to extract preliminary action features, and the preliminary action features are passed through the scene information sensor to obtain action features; the action model includes a scene information sensor, a semantic feature extractor and an Attention mechanism; the semantic feature extractor is stacked with four layers, the first layer structure is composed of 3 3d residual blocks, the second layer structure is composed of 4 3d residual blocks, the third layer structure is composed of 6 3d residual blocks, and the fourth layer structure is composed of 3 3d residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3d residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: .

[0060] In step S3, background features are obtained by passing background understanding input information through a background model, including: the background model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; background features are extracted from the background understanding input information through a 1×7×7 Conv layer to obtain preliminary background features, and the preliminary background features are passed through the scene information sensor to obtain background features; the semantic feature extractor is stacked with four layers, the first layer structure is composed of 3 3D residual blocks, the second layer structure is composed of 4 3D residual blocks, the third layer structure is composed of 6 3D residual blocks, and the fourth layer structure is composed of 3 3D residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3D residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: .

[0061] The scene information sensor is a Channel Context Perception Block structure: including the first branch and the second branch;

[0062] The first branch emphasizes the importance of different features in the receptive field. It first performs a 3×3×3 maximum pooling operation. The group convolution layer performs a group convolution operation on the input feature map, and then performs a 1×1×1 group convolution. Then, softmax is used to emphasize the importance of each feature in the receptive field feature. The second branch learns the spatial feature information in the receptive field, uses a 3×3×3 group convolution operation, and then performs a dot product operation on the first branch and the second branch to re-weight the feature map {30, h, w, 27c} ({time, height, width, channel}), which is then reorganized into {90, 3h, 3w, c}. Then, channel attention is used to focus on the key channels of the action features in the image sequence. Finally, a 3×3×3 convolution is performed to obtain the feature map {30, h, w, c}.

[0063] The scene information sensor is a Spatial Context Perception Block structure: including the first branch and the second branch;

[0064] The first branch first uses the average pooling operation to retain the background information, then uses 1×1×1 grouped convolution for information interaction, and uses softmax to emphasize the importance of each feature in the receptive field feature. The second branch uses 1×3×3 grouped convolution to learn the spatial feature information in the convolution kernel and generate {1,h,w,27c}, which is also transformed into {3,3h,3w,c} through dot product and reorganization operations. Then, the spatial attention mechanism is used to learn the weights of different positions, and finally 3×3×3 convolution is performed to obtain {1,h,w,c}.

[0065] Attention Mechanism is used to learn the importance of features in different channels and positions;

[0066] The Group Convolution layer performs group convolution operations on the input information. The T×K×K group convolution divides the input channels into several groups, and then performs convolution calculations on the channels in each group.

[0067] In step S5, it includes: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, obtaining the fusion weights through the background features and the action features in the bounding box, selectively suppressing or enhancing the features before fusion: interpolating the background features, and then concatenating the background features and the action features in the bounding box to obtain the features {30,1,1,4096}, then learn the features {30,1,1,4096} to get the fusion weight; perform softmax operation on the fusion weight to get the normalized fusion weight [ , ], according to the normalized fusion weight [ , ] Linearly weighted fusion of action features and background features within the bounding box is performed to obtain the final fused features.

[0068] This embodiment determines whether the current iteration number reaches the iteration standard. If not, the background features and action features are fused to obtain fused features, and the fused features are used as input information for the action understanding module; the background features are used as background understanding input information, and the current iteration number is updated and iterated again; if it is reached, the background features and action features are weightedly fused to obtain the final fused features; through multiple iterations and weighted fusion, the background features and action features can be better fused, so that the final giant panda behavior recognition effect is more accurate.

[0069] Embodiment three:

[0070] In addition, the present invention also provides Figure 2 A giant panda behavior recognition system based on deep learning is shown, comprising:

[0071] Acquisition module: acquire the video to be detected, cut frames of the video to be detected to obtain the image to be detected; resample the image to be detected to obtain the resampled image, and use the resampled image as the target object guidance input information and background understanding input information; obtain the action image to be identified according to the frame difference method of the image to be detected as the action understanding input information; set the current iteration number i and the iteration standard, where the current iteration number i=1; build the action model and the background model;

[0072] Bounding box module: obtains a bounding box indicating the location of the giant panda based on the target object guidance input information;

[0073] Feature module: the action understanding input information is passed through the action model to obtain action features; the background understanding input information is passed through the background model to obtain background features;

[0074] Iteration module: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5;

[0075] Feature fusion module: obtain the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and perform weighted fusion of the background features and the action features in the bounding box to obtain the final fusion features;

[0076] Classification module: classify the giant panda behavior based on the final fusion features.

[0077] The process of obtaining a bounding box indicating the location information of the giant panda according to the target object guided input information includes: feature extraction stage, RPN (Region Proposal Network) stage, RoI Align stage and Classifier stage.

[0078] The input is a single frame image. In the feature extraction stage, the backbone of ResNet50 is used to extract the Feature Map of the input image. ResNet adopts a residual structure. In each residual block, the residual connection allows the input to be directly added to the output of the convolutional layer, thus forming a residual connection. The introduction of this structure greatly accelerates the training process of the network, while also allowing the construction of a deeper network and solving the gradient vanishing problem.

[0079] The RPN network is mainly used to generate Region Proposals. First, × Each pixel on the Feature Map is equipped with 9 rectangular anchors with 3 aspect ratios of [1:1, 1:2, 2:1] as the initial detection box. Then it is divided into two branches. One branch obtains positive and negative classifications through Softmax classification of anchors to generate { ,18} vector. The other branch is used to calculate the dimension of anchors is { ,36} bounding box regression offset to obtain accurate proposals. The Proposal layer is responsible for synthesizing the positive anchors and the corresponding boundingbox regression offset to obtain the corrected proposals. It uses the offset of the regression branch to correct the original anchors, and uses the softmax scores of the classification branch to sort the positive scores from large to small, taking the top N anchors, and removing proposals that are too small and out of bounds. Finally, NMS (non-maximum suppression) is performed on the remaining positive anchors, and the upper left corner coordinates and lower right corner coordinates corresponding to the original image size are output. , , , ].

[0080] Roi Align uses the proposals generated by RPN and the Feature Map obtained by the last layer of ResNet50 to traverse each candidate region, keeping the floating point boundaries unquantized. Then the candidate region is divided into 7×7 units, and the boundaries of each unit are not quantized. In each unit, the values ​​of these four positions are calculated using the bilinear interpolation method, and then the maximum pooling operation is performed. Finally, a fixed size 7×7 region proposals are obtained for subsequent fully connected layer processing.

[0081] In the classifier stage, the region proposals are classified into specific categories through the FC layer and Softmax, and the output dimension is {1,1,4}, corresponding to the probabilities of the four behaviors. Then the region proposals are regressed by boundingbox, and the anchors offset {1,1,4} is output to correct the anchors to obtain a more accurate predicted box.

[0082] The action model and the background model each include a scene perception module;

[0083] The scene perception module CPB consists of two key components: (1) Information Extraction Unit (IEU), which is used to capture scene context information and consider the different importance of each feature in different contexts; (2) Semantic Feature Extractor (SFE), which is used to extract rich semantic feature information from image sequences. CPB uses Channel Context Perception Block (CCPB) in the action model, and its structure is as follows: Figure 3 The Spatial Context Perception Block (SCPB) is used in the background model, and its structure is as follows: Figure 4 .

[0084] The Context Information Perceptron (CIP) designed in this paper considers the contextual scene information of each feature to further improve the efficiency of feature extraction, and uses the attention mechanism to learn the importance of features in different channels and different positions. The IEU in the action understanding branch extracts action features that consider the scene context information between consecutive frames. The previous branch emphasizes the importance of different features in the receptive field. Since the focus is on the significant features in the image sequence, a 3×3×3 (time×height×width) MaxPool operation is performed first. The Group Convolution layer performs a group convolution operation on the input feature map. The T×K×K group convolution divides the input channels into several groups, and then the channels in each group are convolved. This operation can reduce the number of parameters and enhance the nonlinear expression ability of the network to a certain extent. Then a 1×1×1 group convolution is performed to interact with information. Then softmax is used to emphasize the importance of each feature in the receptive field feature. The next branch learns the spatial feature information in the receptive field, using a 3×3×3 group convolution operation. Then the two branches are re-weighted to obtain the feature map {30, h, w, 27c} ({time, height, width, channel}), which is reshaped into {90, 3h, 3w, c}, and then the channel attention is used to focus on the key channels of the action features in the image sequence. Finally, a 3×3×3 convolution is performed to obtain the feature map {30, h, w, c}. In the background understanding branch, the IEU only processes a single frame of image compared to the action understanding branch. First, the Avgpool operation is used to retain the background information, and then a 1×1×1 grouped convolution is used for information interaction, and softmax is used to emphasize the importance of each feature in the receptive field feature. The other branch uses a 1×3×3 grouped convolution operation to learn the spatial feature information within the convolution kernel and generate {1, h, w, 27c}. It is also re-weighted and reshaped to {3, 3h, 3w, c}, and then the spatial attention mechanism is used to learn the weights of different positions, and finally a 3×3×3 convolution is performed to obtain {1, h, w, c}.

[0085] Use Conv3d Block to gradually extract richer semantic feature representation. In this embodiment, the iteration standard is 4, and the four iterations call Stage1, Stage2, Stage3, and Stage4 respectively; when i=1, Stage1 is called, when i=2, Stage2 is called, when i=3, Stage3 is called, and when i=4, Stage4 is called; the blocks of Stage1, Stage2, Stage3, and Stage4 are respectively composed of 3, 4, 6, and 3 stacked 3d residual blocks.

[0086]

[0087] The dimensions of the convolution kernels in the table are represented by {T,H,W,C}, which represent time, height, width, and channel size. The first convolution layer of each block is used to reduce the dimension of the feature map, thereby reducing the number of parameters and the amount of computation. The second convolution kernel is usually used to extract spatial information in the feature map. The third convolution kernel acts similarly to the first convolution kernel and is used to further adjust the dimension of the feature map for residual connection. In the path to action understanding, the first residual block of each block uses a convolution kernel with a time dimension of 3 to extract action feature information between consecutive time frames.

[0088] The action features obtained by applying action understanding input information to the action model include:

[0089] The action understanding input information is passed through a Conv layer with a convolution kernel of 5×7×7 to extract action features. The extracted features are input into CCPB of Stage 1 to obtain action features containing contextual scene information. CCPB not only uses the channel attention mechanism to enhance the importance of the extracted action features, enabling the network to focus on key feature channels more effectively, but also emphasizes the importance of different features within the receptive field. This is because the action information is different and interrelated between each channel and each feature map. At the same time, the extracted features are downsampled to obtain high-level semantic features. , further extracting deep action information to capture more complex behavior patterns.

[0090] Next, the action features are fused with the background features extracted at the same stage to further extract richer action features. The action characteristics in the current stage Perform element-wise summation to obtain ', to introduce more information about the environment and context.

[0091] The output of each stage will be used as the input of the next stage. Through gradual iteration, the network can gradually extract richer and more abstract feature representations, thereby achieving accurate recognition and detection of complex behaviors. Next, we use the bounding box generated by the target guidance branch to indicate the location of the giant panda and extract the action features of the area. The bounding box coordinates are mapped to the feature map generated by Stage4, and the feature map of {30,7,7,2048} is obtained after the RoI Align operation to obtain the action features in the detection box. The {30,1,1,2048} features are obtained through 1×7×7MaxPool.

[0092] This embodiment determines whether the current iteration number reaches the iteration standard. If not, the background features and action features are fused to obtain fused features, and the fused features are used as input information for the action understanding module; the background features are used as background understanding input information, and the current iteration number is updated and iterated again; if it is reached, the background features and action features are weightedly fused to obtain the final fused features; through multiple iterations and weighted fusion, the background features and action features can be better fused, so that the final giant panda behavior recognition effect is more accurate.

[0093] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for identifying giant panda behavior based on deep learning, characterized in that: The following steps are involved: S1: Obtain a video to be detected, cut frames of the video to be detected to obtain an image to be detected; resample the image to be detected to obtain a resampled image, and use the resampled image as target object guidance input information and background understanding input information; The image to be detected is used to obtain the action image to be identified according to the frame difference method as the action understanding input information; the current iteration number i and the iteration standard are set, where the current iteration number i=1; the action model and the background model are constructed; S2: Obtaining a bounding box indicating the location information of the giant panda according to the target object guidance input information; S3: The action understanding input information is passed through the action model to obtain the action features; The background understanding input information is passed through the background model to obtain the background features; The step of obtaining action features from action understanding input information through an action model includes: the action model includes a scene information sensor, a semantic feature extractor, a Conv layer, and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; extracting preliminary action features from action understanding input information through a Conv layer with a convolution kernel of 5×7×7, and obtaining action features from the preliminary action features through a scene information sensor; the action model includes a scene information sensor, a semantic feature extractor, and an Attention mechanism; the semantic feature extractor is stacked with four layers, the first layer structure is composed of three 3D residual blocks, the second layer structure is composed of four 3D residual blocks, the third layer structure is composed of six 3D residual blocks, and the fourth layer structure is composed of three 3D residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3D residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: ; The scene information sensor is a Channel Context Perception Block structure: comprising a first branch and a second branch; The first branch emphasizes the importance of different features in the receptive field. It first performs a 3×3×3 maximum pooling operation. The group convolution layer performs a group convolution operation on the input feature map, and then performs a 1×1×1 group convolution. Then, softmax is used to emphasize the importance of each feature in the receptive field feature. The second branch learns the spatial feature information in the receptive field, using a 3×3×3 group convolution operation. Then, the first branch and the second branch are re-weighted by a dot product operation to obtain a feature map {30, h, w, 27c} ({time, height, width, channel}), which is then reorganized into {90, 3h, 3w, c}. Then, channel attention is used to focus on the key channels of the action features in the image sequence. Finally, a 3×3×3 convolution is performed to obtain the feature map {30, h, w, c}. The background understanding input information is passed through the background model to obtain background features, which includes: the background model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; the background understanding input information is passed through a 1×7×7 Conv layer to extract background features to obtain preliminary background features, and the preliminary background features are passed through the scene information sensor to obtain background features; the semantic feature extractor is a four-layer structure stacked, the first layer structure is composed of 3 3d residual blocks, the second layer structure is composed of 4 3d residual blocks, the third layer structure is composed of 6 3d residual blocks, and the fourth layer structure is composed of 3 3d residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3d residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: ; The scene information sensor is a Spatial Context Perception Block structure: comprising a first branch and a second branch; The first branch first uses the average pooling operation to retain the background information, then uses 1×1×1 grouped convolution for information interaction, and uses softmax to emphasize the importance of each feature in the receptive field feature. The second branch uses 1×3×3 grouped convolution to learn the spatial feature information in the convolution kernel and generate {1,h,w,27c}, which is also transformed into {3,3h,3w,c} through dot product and reorganization operations. Then, the spatial attention mechanism is used to learn the weights of different positions, and finally 3×3×3 convolution is performed to obtain {1,h,w,c}. S4: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5; S5: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and weightedly fusing the background features and the action features in the bounding box to obtain the final fused features; S6: Classify giant panda behaviors based on the final fused features.

2. A giant panda behavior recognition method based on deep learning according to claim 1, characterized in that: In step S1, after obtaining the video to be detected, data enhancement is first performed: the video to be detected is flipped horizontally and vertically, and flipped at random angles to obtain the video to be detected after data enhancement; then the video to be detected after data enhancement is cut into frames to obtain the image to be detected.

3. A giant panda behavior recognition method based on deep learning according to claim 1, characterized in that: In step S2, there are preliminary feature extraction stage, candidate region extraction stage, size fixing stage and classification stage: S2.1 Preliminary feature extraction stage: extract preliminary features based on the target object guided input information to obtain a preliminary feature map; S2.2 Candidate region extraction stage: Generate multiple candidate regions based on the feature map; S2.3 Fixed size stage: traverse each candidate region, keep the floating point number boundary unquantized; then divide each candidate region into 7×7 units, and the boundary of each unit is not quantized; in each unit, bilinear interpolation and maximum pooling are performed; finally, a candidate region of fixed size 7×7 is obtained; S2.4 Classification stage: Classify the candidate area of ​​fixed size 7×7 into specific categories, and then perform bounding box regression on the candidate area of ​​fixed size 7×7 to obtain a bounding box indicating the location information of the giant panda.

4. A giant panda behavior recognition method based on deep learning according to claim 1, characterized in that: Attention Mechanism is used to learn the importance of features in different channels and positions; The Group Convolution layer performs group convolution operations on the input information. The T×K×K group convolution divides the input channels into several groups, and then performs convolution calculations on the channels in each group.

5. A giant panda behavior recognition method based on deep learning according to claim 1, characterized in that: In step S5, it includes: obtaining the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, obtaining the fusion weights through the background features and the action features in the bounding box, selectively suppressing or enhancing the features before fusion: interpolating the background features, and then concatenating the background features and the action features in the bounding box to obtain the features {30,1,1,4096}, then learn the features {30,1,1,4096} to get the fusion weight; perform softmax operation on the fusion weight to get the normalized fusion weight [ , ], according to the normalized fusion weight [ , ] Linearly weighted fusion of action features and background features within the bounding box is performed to obtain the final fused features.

6. A giant panda behavior recognition system based on deep learning, characterized in that: include: Acquisition module: acquire the video to be detected, cut frames of the video to be detected to obtain the image to be detected; Resampling the image to be detected to obtain a resampled image, and using the resampled image as target object guidance input information and background understanding input information; The image to be detected is used to obtain the action image to be identified according to the frame difference method as the action understanding input information; the current iteration number i and the iteration standard are set, where the current iteration number i=1; the action model and the background model are constructed; Bounding box module: obtains a bounding box indicating the location of the giant panda based on the target object guidance input information; Feature module: The action understanding input information is passed through the action model to obtain the action features; The background understanding input information is passed through the background model to obtain the background features; The step of obtaining action features from action understanding input information through an action model includes: the action model includes a scene information sensor, a semantic feature extractor, a Conv layer, and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; extracting preliminary action features from action understanding input information through a Conv layer with a convolution kernel of 5×7×7, and obtaining action features from the preliminary action features through a scene information sensor; the action model includes a scene information sensor, a semantic feature extractor, and an Attention mechanism; the semantic feature extractor is stacked with four layers, the first layer structure is composed of three 3D residual blocks, the second layer structure is composed of four 3D residual blocks, the third layer structure is composed of six 3D residual blocks, and the fourth layer structure is composed of three 3D residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3D residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: ; The scene information sensor is a Channel Context Perception Block structure: comprising a first branch and a second branch; The first branch emphasizes the importance of different features in the receptive field. It first performs a 3×3×3 maximum pooling operation. The group convolution layer performs a group convolution operation on the input feature map, and then performs a 1×1×1 group convolution. Then, softmax is used to emphasize the importance of each feature in the receptive field feature. The second branch learns the spatial feature information in the receptive field, using a 3×3×3 group convolution operation. Then, the first branch and the second branch are re-weighted by a dot product operation to obtain a feature map {30, h, w, 27c} ({time, height, width, channel}), which is then reorganized into {90, 3h, 3w, c}. Then, channel attention is used to focus on the key channels of the action features in the image sequence. Finally, a 3×3×3 convolution is performed to obtain the feature map {30, h, w, c}. The background understanding input information is passed through the background model to obtain background features, which includes: the background model includes a scene information sensor, a semantic feature extractor, a Conv layer and an Attention mechanism, wherein the scene information sensor includes a Group Convolution layer; the background understanding input information is passed through a 1×7×7 Conv layer to extract background features to obtain preliminary background features, and the preliminary background features are passed through the scene information sensor to obtain background features; the semantic feature extractor is a four-layer structure stacked, the first layer structure is composed of 3 3d residual blocks, the second layer structure is composed of 4 3d residual blocks, the third layer structure is composed of 6 3d residual blocks, and the fourth layer structure is composed of 3 3d residual blocks. When i=1, the first layer structure is called, when i=2, the second layer structure is called, when i=3, the third layer structure is called, and when i=4, the fourth layer structure is called. The 3d residual block of the first layer structure is: , the second layer structure 3d residual block is: , the third layer structure 3d residual block is: , the fourth layer structure 3d residual block is: ; The scene information sensor is a Spatial Context Perception Block structure: comprising a first branch and a second branch; The first branch first uses the average pooling operation to retain the background information, then uses 1×1×1 grouped convolution for information interaction, and uses softmax to emphasize the importance of each feature in the receptive field feature. The second branch uses 1×3×3 grouped convolution to learn the spatial feature information in the convolution kernel and generate {1,h,w,27c}, which is also transformed into {3,3h,3w,c} through dot product and reorganization operations. Then, the spatial attention mechanism is used to learn the weights of different positions, and finally 3×3×3 convolution is performed to obtain {1,h,w,c}. Iteration module: Determine whether the current iteration number reaches the iteration standard. If not, fuse the background features and action features to obtain fused features, use the fused features as action understanding input information, use the background features as background understanding input information, and update the current iteration number i, i=i+1; then return to execute step S3; if it reaches, execute step S5; Feature fusion module: obtain the action features in the bounding box according to the bounding box indicating the position information of the giant panda and the action features, and perform weighted fusion of the background features and the action features in the bounding box to obtain the final fusion features; Classification module: classify the giant panda behavior based on the final fusion features.