Cow behavior recognition method based on multi-scale attention double-flow 3D convolutional network

Abstract

The invention relates to a cattle behavior recognition method based on a multi-scale attention double-flow 3D convolutional network. The method comprises the following steps: step 1, performing data set construction on collected videos of different cattle daily behaviors; 2, intercepting an RGB image, and calculating an optical flow image corresponding to the RGB image; step 3, constructing an EMA module; step 4, the EMA module constructed in the step 3 is embedded into InflatedI3D, and a Non-Local operation is integrated into an I3D network, so that M3D is constructed; step 5, constructing a two-stream M3D (Media Three Dimensional) network; and step 6, obtaining a final cattle behavior classification result based on the two-stream M3D network. According to the method, the cattle behavior recognition precision and robustness are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a cattle behavior recognition method, in particular to a cattle behavior recognition method based on a multi-scale attention dual-stream 3D convolutional network. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and big data technology, deep learning technology has achieved breakthrough progress in many fields, especially in the field of computer vision. The application of deep learning models has greatly improved the accuracy and efficiency of video and image recognition. These advances not only save a lot of time in manual feature extraction, but also significantly promote the rapid development of computer vision technology. In animal husbandry, especially in the field of cattle behavior recognition, the application of this technology provides new possibilities for improving breeding efficiency and animal welfare. With the rapid development of modern animal husbandry, the demand for cattle health management and behavior recognition is increasing.

[0003] As a major livestock farming country, China's total livestock production has long been among the highest in the world, and the proportion of the livestock industry is also increasing. Against this background, ranch modernization has become an inevitable choice for the development of animal husbandry. Especially in the field of cattle farming, due to the complexity of dairy cow behavior data in a group farming environment, such as complex scenes, large changes in target scales, and behavioral diversity, behavior recognition poses a considerable challenge. For economic animals such as cattle, accurately identifying their behaviors such as grazing, drinking, resting, excreting, and socializing can not only improve the welfare and production efficiency of the herd, but also detect diseases early and prevent potential health problems, which is crucial in large-scale intensive farming systems. For example, cattle with hoof disease will show signs of lameness, reduced exercise or increased rest time, while cattle will show behaviors such as climbing during estrus, and estrus behavior directly affects conception rate and reproductive efficiency.

[0004] Traditionally, the monitoring of cattle behavior mainly relies on manual observation. This method is not only time-consuming and inefficient, but also highly subjective. It is difficult to achieve continuous and large-scale monitoring around the clock. It is easily affected by the subjective judgment of observers, which limits the efficiency and accuracy of cattle management. With the introduction of computer vision technology, especially the application of deep learning technology, it has become possible to use RGB images obtained by cameras and other devices to identify animal behavior. This method can provide a contactless, automated and cost-effective way to monitor cattle behavior, greatly reducing the manual burden and improving the accuracy and real-time nature of monitoring.

[0005] Although current technology has made significant progress, the field of cattle behavior recognition still faces many challenges: (1) Intra-class and inter-class differences: cattle react differently to changes in internal conditions or external environmental stimuli. A cow may show different postures and speeds when performing the same action. Even the same action may present multiple different styles. In addition, the subtle differences in action and color similarity between cattle pose a challenge to deep learning systems for behavior recognition. Since cattle keep their heads down to the ground most of the time, their actions are visually very similar. Therefore, in addition to relying on visual features, the analysis of action features becomes particularly critical in the recognition process. (2) Occlusion and data imbalance problems: In a herd-living pasture environment, occlusion of cattle by other individuals or scene components often makes it difficult to fully visualize the action area. In addition, our dataset videos are collected from multiple days of video, including daytime and nighttime scenes. The frequency of some behaviors in the dataset (such as eating grass and drinking water) is much lower than that of other behaviors, resulting in significant imbalance in the training data. (3) Changing illumination and background as well as scale changes: The changes in light intensity within a day range from day to night. At night or under low light conditions, the degradation of video quality seriously affects the accuracy of behavior recognition. Although the video is shot in the background of a specific farm, the dynamic background containing moving objects may interfere with the behavior recognition process. In addition, the size of the cows captured by cameras at different distances varies. Cows far away appear smaller than those close. This scale change may also interfere with the accurate recognition of cow behavior, posing another challenge.

[0006] How to deal with intra-class and inter-class differences of cattle, how to deal with illumination changes, occlusion problems and scale changes, and how to improve the accuracy and robustness of the algorithm, etc. Therefore, continuing to explore and develop more advanced deep learning models and algorithms to improve the accuracy and efficiency of cattle behavior recognition has important research and application value for promoting the modernization and intelligence of animal husbandry.

[0007] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a cattle behavior recognition method based on a multi-scale attention dual-stream 3D convolutional network, which significantly improves the recognition accuracy and robustness of cattle behavior.

[0009] The present invention solves the practical problem by adopting the following technical solutions:

[0010] A cattle behavior recognition method based on a multi-scale attention dual-stream 3D convolutional network includes the following steps:

[0011] Step 1: Construct a data set and preprocess the collected videos of different daily behaviors of cattle;

[0012] Step 2: Based on the data set constructed in step 1, capture the RGB image and calculate the optical flow image corresponding to the RGB image;

[0013] Step 3: Construct EMA module;

[0014] Step 4: Embed the EMA module constructed in step 3 into InflatedI3D, and integrate the Non-Local operation into the I3D network to construct M3D;

[0015] Step 5: Build a two-stream M3D network to obtain the output of the two branches: RGB M3D and Optical Flow M3D.

[0016] Step 6: Spatial flow features F output by the two branches of RGB M3D and Optical Flow M3D based on the two-stream M3D network space and the time flow feature F time , the output features of the two branches are fused by feature concatenation, the spatial channel dependency is established, and the fused feature F combined After inputting the constructed efficient cattle behavior classifier Classifier, the final classification result of the cattle behavior is obtained.

[0017] Moreover, the specific steps of step 1 include:

[0018] (1) Preliminary preprocessing was performed on the videos of different daily behaviors of cattle collected in the natural environment. The video clips of each behavior of cattle were extracted into 256×256 pixel frames. The peripheral areas of the input images were zero-filled to obtain the Kinectics400 format cattle behavior recognition dataset, which was named CSIROV1.

[0019] (2) Create a new cattle behavior dataset named CSIROV2. Clip videos longer than 5 seconds in the CSIROV1 dataset into 5-second segments, and filter out clips shorter than 3 seconds. Each clipped video contains an independent behavior.

[0020] (3) The videos in the CSIROV2 dataset are classified and uniformly named in the format of “action name + number + splits number”, and the training and validation datasets are divided.

[0021] Moreover, the specific method of calculating the optical flow image corresponding to the RGB image in step 2 is:

[0022] For each behavior video in the CSIROV2 dataset constructed in step 1, capture an RGB image every 10 frames, then use the TVL1 optical flow algorithm implemented in opencv to calculate the captured spatial flow RGB image sequence, initialize the DualTVL1OpticalFlow object, and then load the spatial flow image sequence. For the adjacent image frame pairs frame-t and frame-t+1 in the sequence, call the calc method to estimate the horizontal flow-x and vertical flow-y optical flow components;

[0023] Among them, the calculation of optical flow is based on the equation:

[0024] and are the gradients of the horizontal and vertical optical flow components respectively, and each optical flow image sample E cattle is an optical flow vector, which is used to measure the overall error of optical flow estimation. Each optical flow vector consists of a horizontal optical flow component u and a vertical optical flow component v, which represent the pixel displacement in the horizontal and vertical directions, respectively. cattle (x, y, t) represents the brightness of the pixel at time t and position (x, y), I cattle (x+u,y+v,t+1) represents the brightness of the pixel position (x+u,y+v) after the estimated displacement (u,v) in the next time frame t+1. λ is a weight parameter that balances the data term and the smoothness term, and adjusts the accuracy and smoothness of the optical flow estimation.

[0025] Moreover, the structure of the EMA module constructed in step 3 is:

[0026] The EMA module design adopts feature grouping and parallel processing strategies. The EMA module accepts an input feature map with C channels and H×W×T spatial dimensions, and divides the input feature map into g groups along the channel dimension, each group contains C / / g channels;

[0027] The EMA module consists of three parallel processing branches, which are used for global information encoding and local information enhancement to extract the attention weight descriptor of the grouped feature map:

[0028] Among them, two parallel branches are 1x1x1 branches, and the third branch is a 3x3x3 branch;

[0029] The 1x1x1 branch encodes the channels along two spatial directions through 3D global average pooling, then connects the encoded features in the image height direction and shares the same 1x1x1 convolution. After decomposing the output of the 1x1x1 convolution into two vectors, two nonlinear Sigmoid functions are used to fit the 3D bilinear distribution on the linear convolution.

[0030] The 3x3x3 branch directly acts on the original feature map through a 3x3x3 kernel, captures local cross-channel interactions to expand the feature space, and outputs it through a Sigmoid activation function to provide attention weights in the spatial dimension.

[0031] Finally, the output feature maps of the 1x1x1 convolution branch and the 3x3x3 convolution branch are directly added and fused in the corresponding dimensions, and the fused feature maps are input into another Sigmoid function for the final attention modulation.

[0032] Moreover, the specific steps of constructing M3D in step 4 include:

[0033] (1) First, the front end of M3D is constructed, which consists of multiple 3D convolution and 3D maximum pooling layers. These layers use three-dimensional convolution modules for downsampling and feature extraction, and are responsible for extracting the basic spatial features of video frames; the first convolution layer: uses a 7x7x7 convolution kernel with a step size of 2x2x2. The second maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2. The third convolution layer: uses a 1x1x1 convolution kernel with a step size of 1x1x1; the fourth convolution layer: uses a 3x3x3 convolution kernel with a step size of 1x1x1. The fifth maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2.

[0034] (2) The output of the fifth maximum pooling layer is then input into the EMA module to filter out irrelevant background information of the video. The EMA module divides the input feature map into multiple groups along the channel dimension, each group contains the same number of channels; it contains three parallel processing branches, two of which use 1x1x1 convolution kernels to perform feature weighting after global average pooling, and the other 3x3x3 branch is used to capture local features; the parallel branches merge their outputs through the Sigmoid activation function to obtain the adjusted feature map. The final output of EMA has the same size as its input;

[0035] (3) The features output by the EMA module are further refined through a series of Inception modules. Each Inception module combines convolution kernels of different sizes: 1x1x1 and 3x3x3 to meet the needs of capturing different spatial details.

[0036] (4) In the later stage of building the M3D network, the high-level features processed and fused by the Inception module at different scales will be input into the Non-Local module; the Non-Local module captures long-distance dependency information by calculating the pairwise relationships between all positions in the input feature map, and performs a global analysis of the input feature map through the self-attention mechanism. Its calculation is not limited to neighboring pixels, but involves all positions in the entire feature map.

[0037] (5) After multiple rounds of experimental verification, it was chosen to introduce EMA before the Inc-M-3c module, and the Non-Local module was placed before Inc-M-4f.

[0038] (6) In the final stage of the M3D network, the output of the NonLocal module is first processed by the Inc-M-4f module, and then passed to a maximum pooling layer with a 2x2x2 convolution kernel and a stride of 2x2x2; the processed features flow into the Inc-M-5b and Inc-M-5c modules, and then enter a global average pooling layer with a 2x7x7 convolution kernel and a stride of 1x1x1; thereafter, the features are sent to the Logits fully connected layer to map the complex three-dimensional features into a one-dimensional feature vector, which is then input into a Softmax classifier. The Softmax classifier maps each feature vector to a predefined cattle behavior category based on the weights obtained by the network through learning, thus successfully completing the construction of the M3D network.

[0039] Moreover, the specific structure of the two-streamM3D in step 5 is:

[0040] Two-streamM3D consists of two parallel branches. The upper branch is the RGB M3D network, which processes the original RGB video frame sequence and extracts spatial features related to visual appearance. The lower branch is the Optical FlowM3D network, which processes the optical flow image frame sequence generated by calculating the difference of superimposed video frames to capture the dynamic movement information of the cow; the features of the RGB image flow and the optical flow image frame sequence are extracted through two independent network branches, thereby obtaining the outputs of the two branches, RGB M3D and OpticalFlowM3D.

[0041] Moreover, the specific method of step 6 is:

[0042] The outputs of the RGB M3D and OpticalFlowM3D branches of the two-streamM3D network are the spatial flow features F space and the time flow feature F time , where the spatial flow feature F space =

[0043] GCInception s (D space ), used to capture the details and texture information of the static object cow in the video frame, the temporal flow feature F time =GCInception t (D time ), used to capture the fast-changing motion information of cattle in the video frame, D space is the spatial stream image sequence, D time is a temporal stream image sequence.

[0044] The output spatial flow features F of the two branches of RGB M3D and OpticalFlowM3D are space and the time flow feature F time The information from the RGB image stream and the optical flow image stream is integrated into a high-level fusion model, which uses a consensus fusion mechanism to fuse the information from the two branches of the RGB image stream and the optical flow image stream to establish spatial channel dependencies. Feature fusion equation: F combined =σ(W f2 ·σ(W f1 Concat(F space ,F time )+b f1 )+b f2 ), Concat(·) is feature concatenation, W f1 , W f2 , b f1 , b f2 are the weight and bias parameters of the fusion network, and σ is the nonlinear activation function.

[0045] Then, the fused feature F combined Input the constructed efficient cattle behavior classifier Classifier. The classification process uses the Softmax function to output the predicted probability of each behavior state. Finally, the Classifier Classifier obtains the final classification result of the cattle behavior state based on the output predicted probability.

[0046] Advantages and beneficial effects of the present invention:

[0047] 1. The present invention proposes a cattle behavior recognition method based on a multi-scale attention two-stream 3D convolutional network. The two-streamM3D consists of two parallel branches. The upper branch is an RGB M3D network, which processes the original RGB video frame sequence and extracts spatial features related to the visual appearance. Static images can recognize most of the spatial features in the cattle behavior video. The lower branch is an OpticalFlowM3D network, which processes an optical flow image frame sequence generated by calculating the difference of superimposed video frames. By learning the optical flow image, Optical FlowM3D can learn the time changes that are not touched in static images, learn the connection between action changes, and capture the deeper motion feature information of the cattle action in the video in the time dimension. The spatial flow feature F output by the two branches of RGB M3D and OpticalFlowM3D based on the two-streamM3D network space and the time flow feature F time , the output features of the two branches are fused by feature concatenation, the spatial channel dependency is established, and the fused feature F combined After inputting the constructed efficient cattle behavior classifier Classifier, the final cattle behavior classification result is obtained, which can accurately identify the cattle's behavioral status, such as drinking water, eating grass, and other behaviors, and provide accurate cattle behavior data support for pasture management. It has not only achieved remarkable results in the field of cattle behavior recognition, but also provided a new perspective and method for real-time monitoring of agricultural automation and animal welfare.

[0048] 2. The method for recognizing cattle behavior based on efficient multi-scale attention two-stream 3D convolutional network (two-stream M3D) proposed in the present invention realizes the global integration of long-distance feature relationships by introducing EMA and Non-Local modules in EMAInception3D ConvNet network (M3D). At the same time, through the integration of global context, the distribution differences between features are reduced, and the behavior recognition of cattle in complex video sequences is effectively processed, especially the long-term dependencies and subtle changes of continuous behaviors are captured. By integrating EMA and Non-Local technologies, two-stream M3D can effectively utilize the spatiotemporal information of behavioral videos, accurately capture key behavioral features in complex backgrounds, solve the differences within and between cattle classes, and realize the interaction of more fine-grained class information. Two-stream M3D performs well in dealing with lighting changes, occlusion problems, and scale changes, and significantly improves the recognition accuracy and robustness of cattle behavior.

[0049] 3. The present invention processes spatial flow image sequence and temporal flow image sequence data in parallel, effectively integrates and utilizes optical flow features and original video information, and retains spatial details while extracting the temporal information between video frames, optimizes the feature extraction process, and thus achieves comprehensive capture of subtle feature information and temporal information. By stacking optical flow images and accurately matching them with corresponding video frames, optical flow features are used to capture subtle changes in cattle movements and temporal information in video sequences, improving the model's ability to recognize cattle behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 These are sample images of three behaviors in the dataset of the present invention: two consecutive video frames, horizontal optical flow component x and vertical optical flow component y;

[0051] Figure 2 A comparison diagram of the two-dimensional and three-dimensional convolution kernel operations and feature mapping in feature extraction of the present invention;

[0052] Figure 3 Schematic diagram of the efficient multi-scale attention ECA structure of the present invention;

[0053] Figure 4 Schematic diagram of the structure of the EMAInception3D ConvNet network (M3D) of the present invention;

[0054] Figure 5 A schematic diagram of a method for recognizing cattle behavior based on an efficient multi-scale attention two-stream 3D convolutional network (two-stream M3D) of the present invention;

[0055] Figure 6 This is a schematic diagram of the main interface of the cattle behavior detection system of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0057] A cattle behavior recognition method based on a multi-scale attention dual-stream 3D convolutional network includes the following steps:

[0058] Step 1: Construct a data set and preprocess the collected videos of different daily behaviors of cattle;

[0059] The specific steps of step 1 include:

[0060] (1) Preliminary preprocessing was performed on the videos of different daily behaviors of cattle collected in the natural environment. The video clips of each behavior of cattle were extracted into 256×256 pixel frames. The peripheral areas of the input images were zero-filled to obtain the Kinectics400 format cattle behavior recognition dataset, which was named CSIRO V1.

[0061] (2) Create a new cattle behavior dataset named CSIRO V2. Clip videos longer than 5 seconds in the CSIRO V1 dataset into 5-second segments, and filter out clips shorter than 3 seconds. Each clipped video contains an independent behavior.

[0062] (3) The videos in the CSIRO V2 dataset are classified and uniformly named in the format of “action name + number + splits number”, and the training and validation datasets are divided.

[0063] In this embodiment, three GoPro 5Black cameras with built-in GPS receivers were installed at the site, recording for 8 hours a day, and successfully recording the drinking, grazing and other behaviors of cattle in the natural environment without human intervention. The videos of different daily behaviors of cattle in the natural environment collected for 8 days were preliminarily preprocessed, and the video clips of each behavior of the cattle were extracted as 256×256 pixel frames. In addition, in order to make the cattle mainly located in the center of the frame, the peripheral area of ​​the input image was zero-filled, and the Kinectics400 format cattle behavior recognition dataset was obtained, which contained 1715 videos and was named CSIRO V1. CSIRO V1 includes 360 "drinking" videos, 413 "grazing" videos and 942 "other" videos. Since the duration of individual videos in the CSIRO V1 dataset is different, the short video duration is about 1 second, and the long video duration is about 4 minutes and 47 seconds. Considering the short-term nature of some cattle behaviors and the long duration of other behaviors, in order to improve the accuracy of recognition. We created a new cattle behavior dataset named CSIRO V2. We clipped videos longer than 5 seconds in the CSIRO V1 dataset into 5-second segments, and filtered out clips shorter than 3 seconds. The total clipping time was 529.5 minutes, the frame size was 256×256 pixels, and the frame rate was 30fps. Each clipped video contains an independent behavior. This method prevents videos that are too long from containing too much irrelevant information, and videos that are too short from missing key behavior information, which affects the recognition results. The videos in the CSIRO V2 dataset were classified and uniformly named in the format of "action name + number + splits number". A total of 6619 videos were finally exported. Among them, the "drinking water" class contained 1608 videos, the "grazing" class contained 2718 videos, and the "other" class contained 2293 videos. The training and validation datasets were re-divided in a ratio of 0.67:0.33.

[0064] Step 2: Based on the data set constructed in step 1, capture the RGB image and calculate the optical flow image corresponding to the RGB image;

[0065] The specific method of calculating the optical flow image corresponding to the RGB image in step 2 is:

[0066] For each behavior video in the CSIRO V2 dataset constructed in step 1, capture an RGB image every 10 frames, then use the TVL1 optical flow algorithm implemented in opencv to calculate the captured spatial flow RGB image sequence, initialize the DualTVL1OpticalFlow object, and then load the spatial flow image sequence. For the adjacent image frame pairs frame-t and frame-t+1 in the sequence, call the calc method to estimate the horizontal (flow-x) and vertical (flow-y) optical flow components;

[0067] The calculation of optical flow is based on the equation:

[0068] and are the gradients of the horizontal and vertical optical flow components respectively, and each optical flow image sample E cattle is an optical flow vector, which is used to measure the overall error of optical flow estimation. Each optical flow vector consists of a horizontal optical flow component u and a vertical optical flow component v, which represent the pixel displacement in the horizontal and vertical directions, respectively. cattle (x, y, t) represents the brightness of the pixel at time t and position (x, y), I cattle (x+u,y+v,t+1) represents the brightness of the pixel position (x+u,y+v) after the estimated displacement (u,v) in the next time frame t+1. λ is a weight parameter that balances the data term and the smoothness term, and adjusts the accuracy and smoothness of the optical flow estimation.

[0069] In this embodiment, in the created data set CSIRO V2, the RGB image is used as the input of the spatial convolutional network to extract the appearance information of the target object, and the optical flow image corresponding to the RGB image is calculated and used as the input of the temporal convolutional network to extract motion information.

[0070] like Figure 1 The figure below shows the visualization of the optical flow components, which shows the sample image frames and their corresponding optical flow maps of the three behavior modes of cattle in the CSIRO V2 dataset. In the generated image, the u component (flow-x) reflects the horizontal movement, which is manifested as the displacement of the pixel in the horizontal direction, and the v component (flow-y) represents the displacement in the vertical direction. The optical flow map reveals the movement characteristics and intensity in different directions. The sample image frames of the dataset show behavioral changes, scale changes, lighting changes, and diverse backgrounds. For example, the vertical downward movement of the cow's head in the drinking behavior is shown as a clear bright area in the v component map, while in the grazing behavior, the slight movement of the cow's head on the grass causes both the horizontal and vertical optical flow components to show a smaller range of changes.

[0071] Step 3: Build the Efficient Multi-Scale Attention Module (EMA);

[0072] The structure of the EMA module constructed in step 3 is:

[0073] The EMA module design adopts feature grouping and parallel processing strategies. The EMA module accepts an input feature map with C channels and H×W×T spatial dimensions, and divides the input feature map into g groups along the channel dimension, each group contains C / / g channels;

[0074] The EMA module consists of three parallel processing branches, which are used for global information encoding and local information enhancement to extract the attention weight descriptor of the grouped feature map:

[0075] Among them, two parallel branches are 1x1x1 branches, and the third branch is a 3x3x3 branch;

[0076] The 1x1x1 branch encodes the channels along two spatial directions through 3D global average pooling, then connects the encoded features in the image height direction and shares the same 1x1x1 convolution. After decomposing the output of the 1x1x1 convolution into two vectors, two nonlinear Sigmoid functions are used to fit the 3D bilinear distribution on the linear convolution.

[0077] The 3x3x3 branch directly acts on the original feature map through a 3x3x3 kernel, captures local cross-channel interactions to expand the feature space, and outputs it through a Sigmoid activation function to provide attention weights in the spatial dimension.

[0078] Finally, the output feature maps of the 1x1x1 convolution branch and the 3x3x3 convolution branch are directly added and fused in the corresponding dimensions, and the fused feature maps are input into another Sigmoid function for the final attention modulation.

[0079] In this embodiment, in order to deal with the similarity between some behaviors of cattle caused by illumination changes, occlusion, multi-scale changes and intra-class and inter-class differences brought by the complex environment of the pasture during cattle behavior recognition, based on the grouping structure, the sequential processing method of CA is modified to build a novel Efficient Multi-Scale Attention Module (EMA), which does not require dimensionality reduction, retains the information of each channel and reduces the computational overhead, reshapes some channels into batch dimensions, and groups the channel dimensions into multiple sub-features so that the spatial semantic features are well distributed within each feature group. In addition to encoding global information to recalibrate the channel weights in each parallel branch, the output features of the parallel branches are further aggregated through cross-dimensional interactions to capture pixel-level pairwise relationships.

[0080] The constructed EMA module fuses the learned attention maps of parallel sub-networks through a cross-space learning method, using matrix dot product operations, aiming to capture pixel-level pairwise relationships and highlight the global context of all pixels, showing better performance improvement. The overall structure of the EMA module is as follows: Figure 2As shown. The EMA module design adopts a feature grouping and parallel processing strategy. The EMA module accepts an input feature map with C channels and H×W×T spatial dimensions, and divides the input feature map into g groups along the channel dimension, each group containing C / / g channels. The EMA module contains three parallel processing branches, which are used for global information encoding and local information enhancement, respectively, to extract attention weight descriptors of grouped feature maps to deal with the challenges of cattle behavior recognition in pasture environments. Two of the parallel branches are 1x1x1 branches, and the third branch is a 3x3x3 branch. More specifically, the 1x1x1 branch encodes the channels along two spatial directions through 3D global average pooling, and then the encoded features are connected in the image height direction and share the same 1x1x1 convolution. After the output of the 1x1x1 convolution is decomposed into two vectors, two nonlinear Sigmoid functions are used to fit the 3D bilinear distribution on the linear convolution, which enhances the model's ability to recognize intra-class consistency and inter-class differences in cattle behavior. On the other hand, the 3x3x3 branch directly acts on the original feature map through a 3x3x3 kernel, capturing local cross-channel interactions to expand the feature space, and outputs it through a Sigmoid activation function to provide attention weights in the spatial dimension. The 3x3x3 branch directly acts on the original feature map through a single 3x3x3 kernel, and then the output result is used to provide attention weights in the spatial dimension to capture more detailed local multi-scale feature representations to deal with problems such as lighting changes, occlusion and scale changes in dense cattle environments, so that the model can more effectively focus on the key areas of behavioral performance, which is conducive to the model's accurate positioning and identification of the specific behaviors of cattle in different environments. Finally, the output feature maps of the 1x1x1 convolution branch and the 3x3x3 convolution branch are directly added and fused in the corresponding dimensions, and the fused feature map is input into another Sigmoid function for the final attention modulation of the fused feature map.

[0081] Step 4: Embed the Efficient Multi-Scale Attention Module (EMA) module constructed in step 3 into the Inflated 3D ConvNet network (I3D), and integrate the Non-Local operation into the I3D network to construct the EMAInception3D ConvNet network (M3D).

[0082] The specific steps of constructing the EMAInception3D ConvNet network (M3D) in step 4 include:

[0083] (1) First, the front end of M3D is constructed, which consists of multiple 3D convolution and 3D maximum pooling layers. These layers use three-dimensional convolution modules for downsampling and feature extraction, and are responsible for extracting the basic spatial features of video frames; the first convolution layer: uses a 7x7x7 convolution kernel with a step size of 2x2x2. The second maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2. The third convolution layer: uses a 1x1x1 convolution kernel with a step size of 1x1x1. The fourth convolution layer: uses a 3x3x3 convolution kernel with a step size of 1x1x1. The fifth maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2.

[0084] (2) The output of the fifth maximum pooling layer is then input into the EMA module to filter out irrelevant background information of the video. The EMA module divides the input feature map into multiple groups along the channel dimension, and each group contains the same number of channels. It contains three parallel processing branches, two of which use 1x1x1 convolution kernels to perform feature weighting after global average pooling, and the other 3x3x3 branch is used to capture local features. The parallel branches merge their outputs through the Sigmoid activation function to obtain the adjusted feature map. The final output of EMA has the same size as its input;

[0085] (3) The features output by the EMA module are then further refined through a series of Inception modules. Specifically, each Inception module combines convolution kernels of different sizes: 1x1x1 and 3x3x3 to meet the needs of capturing different spatial details. The 1x1x1 convolution kernel focuses on processing and transmitting local fine-grained features without changing the spatial dimension of the feature map, while the 3x3x3 convolution kernel provides a wider receptive field and is suitable for processing medium and large feature maps in the image.

[0086] (4) In the later stage of building the M3D network, the high-level features processed and fused by the Inception module will be input into the Non-Local module. The Non-Local module captures long-range dependency information by calculating the pairwise relationships between all positions in the input feature map, and performs a global analysis of the input feature map through the self-attention mechanism. Its calculation is not limited to neighboring pixels, but involves all positions in the entire feature map.

[0087] (5) After multiple rounds of experimental verification, it was chosen to introduce EMA before the Inc-M-3c module, and the Non-Local module was placed before Inc-M-4f.

[0088] (6) In the final stage of the M3D network, the output of the Non Local module is first processed by the Inc-M-4f module and then passed to a maximum pooling layer with a 2x2x2 convolution kernel and a stride of 2x2x2. The processed features flow into the Inc-M-5b and Inc-M-5c modules, and then enter a global average pooling layer with a 2x7x7 convolution kernel and a stride of 1x1x1. After that, the features are sent to the Logits fully connected layer to map the complex three-dimensional features into a one-dimensional feature vector, which is then input into a Softmax classifier. The Softmax classifier maps each feature vector to a predefined cattle behavior category based on the weights learned by the network, thus successfully completing the construction of the M3D network.

[0089] In this embodiment, in CNN, traditional 2D convolution extracts behavioral features frame by frame and then adds features. However, 3D convolution adds a time dimension based on 2D convolution, thereby effectively processing the association between consecutive frames and directly generating the final features. Due to the ability of 3D convolution to better apply the frame features of behavioral videos, 3D convolution has better recognition performance than 2D convolution. 3D ConvNet can capture the time dimension information in video sequences, which is crucial for understanding continuous actions in videos. Therefore, the feature map generated by 3D convolution can better represent the behavior of cows than 2D convolution. Figure 3 The operations of 2D and 3D convolutions. (a) shows a 3x3 2D convolution kernel on the output feature 5x5 of the i-th layer, with stride=1 and padding=0, performing a convolution operation to generate a 3x3 feature map at the i+1 layer. In (b), a 3x3x3 3D ​​convolution kernel is applied to the output feature 5x5x8 of the i-th layer, also with stride=1 and padding=0, to generate a 3x3x6 feature map at the i+1 layer.

[0090] In order to avoid the complex factors of behavior videos in natural scenes (intra-class and inter-class differences, illumination, occlusion, illumination changes, dynamic background and scale changes, etc.) and increase the recognizability between cow behaviors, the Efficient Multi-Scale Attention Module (EMA) constructed in step 3 is added to the Inflated 3D ConvNet network (I3D), and the Non-Local operation is integrated into the I3D network to construct the EMAInception3DConvNet network (M3D).

[0091] First, the front end of M3D is constructed by multiple 3D convolution and 3D max pooling layers, which use 3D convolution modules for downsampling and feature extraction, responsible for extracting the basic spatial features of the video frame. The first convolution layer: uses a 7x7x7 convolution kernel with a step size of 2x2x2 to capture preliminary spatial and temporal information and perform preliminary downsampling. The second max pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2 to continue extracting spatial features but retain the temporal dimension. The third convolution layer: uses a 1x1x1 convolution kernel with a step size of 1x1x1 to strengthen the extraction of spatial features. The fourth convolution layer: uses a 3x3x3 convolution kernel with a step size of 1x1x1 to further refine the feature extraction. The fifth max pooling layer: uses a 1x3x3 max pooling layer with a step size of 1x2x2 to reduce the feature dimension, reduce the amount of calculation, and enhance the generalization ability of the model.

[0092] The output of the fifth maximum pooling layer is then input into the EMA module to filter out irrelevant background information of the video, retain the information of each channel and reduce the computational overhead, reshape some channels into batch dimensions, and group the channel dimensions into multiple sub-features so that the spatial semantic features are well distributed within each feature group, while encoding global information to recalibrate the channel weights in each parallel branch. The EMA module divides the input feature map into multiple groups along the channel dimension, each containing the same number of channels. It contains three parallel processing branches, two of which use 1x1x1 convolution kernels for feature weighting after global average pooling, and the other 3x3x3 branch is used to capture local features. The parallel branches merge their outputs through the Sigmoid activation function to obtain the adjusted feature map, optimize the channel weights and spatial attention, so as to better identify and distinguish different behavioral patterns of cattle. The final output of EMA is the same size as its input.

[0093] The features output by the EMA module are then further refined through a series of Inception modules. The Inception module is designed to fully capture and process different spatial details through multi-scale convolution kernels to adapt to the diverse visual patterns in complex video data. Specifically, each Inception module combines convolution kernels of different sizes: 1x1x1, and 3x3x3 to meet the needs of capturing different spatial details. The 1x1x1 convolution kernel focuses on processing and transmitting local fine-grained features without changing the spatial dimension of the feature map, while the 3x3x3 convolution kernel provides a wider receptive field and is suitable for processing medium and large feature maps in images.

[0094] In the later stages of building the M3D network, the high-level features processed and fused by the Inception module at different scales are input to the Non-Local module. The Non-Local module captures long-range dependency information by calculating the pairwise relationships between all positions in the input feature map, and performs a global analysis of the input feature map through the self-attention mechanism. Its calculations are not limited to neighboring pixels, but involve all positions in the entire feature map. Unlike traditional convolutional layers that only focus on local neighborhoods, the Non-Local module can integrate information from distant positions, which helps to identify cattle behavior information that spans a long time or a wide space in the video sequence.

[0095] After multiple rounds of experimental verification, we chose to introduce EMA before the Inc-M-3c module. This choice is based on the fact that the network at this stage begins to gradually process abstract feature representations, while the hierarchy after the Inc-M-3c module begins to process more advanced features. Introducing EMA at this stage can effectively enhance the multi-scale integration capabilities of primary features and maximize the effect of the attention mechanism, thereby improving the model's adaptability to targets of different sizes. The Non-Local module is placed before Inc-M-4f, based on the need for global information integration at a deeper level of the network. At this time, the model has extracted rich local features and needs to strengthen the understanding of spatiotemporal relationships, so as to effectively process the behavior recognition of cattle in complex video sequences, especially to capture the long-term dependencies and subtle changes of continuous behaviors.

[0096] In the final stage of the M3D network, the output of the Non Local module is first processed by the Inc-M-4f module, and then passed to a maximum pooling layer with a 2x2x2 convolution kernel and a step size of 2x2x2. The processed features flow into the Inc-M-5b and Inc-M-5c modules, and then into a global average pooling layer with a 2x7x7 convolution kernel and a step size of 1x1x1. Thereafter, the features are sent to the Logits fully connected layer. Through this series of Inception modules and 3D pooling layers, the features of the entire time-space dimension are pooled, and the complex three-dimensional features are mapped into a one-dimensional feature vector. The one-dimensional feature vector is then input into a Softmax classifier. The Softmax classifier maps each feature vector to a predefined cattle behavior category based on the weights obtained by the network through learning, thus successfully completing the construction of the M3D network. The specific network structure of M3D is as follows Figure 4 shown.

[0097] Step 5: Build a two-stream inflated EMAInception3D ConvNet network (two-stream M3D) to obtain the output of the two branches of RGB M3D and Optical Flow M3D.

[0098] The specific structure of the two-stream inflated EMAInception3D ConvNet network (two-stream M3D) in step 5 is:

[0099] Two-stream M3D consists of two parallel branches. The upper branch is the RGB M3D network, which processes the original RGB video frame sequence and extracts spatial features related to visual appearance. The lower branch is the Optical Flow M3D network, which processes the optical flow image frame sequence generated by calculating the difference of superimposed video frames to capture the dynamic movement information of the cow. The features of the RGB image flow and the optical flow image frame sequence are extracted through two independent network branches, thereby obtaining the outputs of the two branches, RGB M3D and Optical Flow M3D.

[0100] In this embodiment, in order to more effectively extract visual features and time series features from the video, a two-stream network architecture is constructed in the present invention, and a video feature extraction method of a two-stream inflated EMA Inception3D ConvNet network (two-stream M3D) is used. The two-stream M3D consists of two parallel branches. The upper branch is an RGBM3D network, which processes the original RGB video frame sequence and extracts spatial features related to the visual appearance. The lower branch is an Optical Flow M3D network, which processes an optical flow image frame sequence generated by calculating the difference between superimposed video frames to capture the dynamic motion information of the cow. The features of the RGB image stream and the optical flow image frame sequence are extracted respectively through two independent network branches, thereby obtaining the outputs of the two branches of RGB M3D and Optical Flow M3D. The two-stream M3D structure is as follows: Figure 5 As shown in the figure, it consists of two parallel branches, which process different types of input data respectively. The first branch is the RGB image stream branch that accepts the original video frame sequence as input, and extracts the spatial features of each frame in the video sample through M3D. Most of the features in the cattle behavior video can be identified through static images; the second branch is the optical flow branch that accepts the optical flow image frame sequence generated by calculating the difference of the superimposed video frames as input. The optical flow image reveals the movement of objects in the scene over time. By learning the optical flow image, M3D can learn the time changes that are not exposed in static images, learn the connection between action changes, and capture the deeper motion feature information of video actions in the time dimension.

[0101] Specifically, in the two-stream inflated EMA Inception3D ConvNet network (two-stream M3D) we constructed, the spatial stream and the temporal stream represent the static features and dynamic changes of the video content respectively. For each input sequence of the spatial stream and the temporal stream, the backbone network GC Inception network is selected to extract the deep features of the spatial stream and the temporal stream. The GC Inception network is based on the Inception-v1 architecture and enhances the stability and efficiency of training through batch normalization. Thus, the two-stream M3D network is successfully constructed.

[0102] Step 6: Spatial flow features F output by the two branches of RGB M3D and OpticalFlowM3D based on the two-streamM3D network space and the time flow feature F time , the output features of the two branches are fused by feature concatenation, the spatial channel dependency is established, and the fused feature F combined After inputting the constructed efficient cattle behavior classifier Classifier, the final cattle behavior classification result is obtained;

[0103] The specific method of step 6 is:

[0104] The outputs of the RGB M3D and OpticalFlowM3D branches of the two-streamM3D network are the spatial flow features F space and the time flow feature F time , where the spatial flow feature F space =

[0105] GCInception s (D space ), used to capture the details and texture information of the static object cow in the video frame, the temporal flow feature F time =GCInception t (D time ), used to capture the fast-changing motion information of cattle in the video frame, D space is the spatial stream image sequence, D time is a temporal stream image sequence.

[0106] The output spatial flow features F of the two branches of RGB M3D and OpticalFlowM3D are space and the time flow feature F timeThe information from the RGB image stream and the optical flow image stream is integrated into a high-level fusion model, which uses a consensus fusion mechanism to fuse the information from the two branches of the RGB image stream and the optical flow image stream to establish spatial channel dependencies. Feature fusion equation: F combined =σ(W f2 ·σ(W f1 Concat(F space ,F time )+b f1 )+b f2 ), Concat(·) is feature concatenation, W f1 , W f2 , b f1 , b f2 are the weight and bias parameters of the fusion network, and σ is the nonlinear activation function.

[0107] Then, the fused feature F combined Input the constructed efficient classifier of cattle behavior. The classification process uses the Softmax function to output the predicted probability of each behavior state. The final classifier Classifier obtains the classification result of the final cattle behavior state based on the output predicted probability: drinking water, eating grass, and other behaviors, providing accurate behavior recognition results for cattle behavior analysis.

[0108] Step 7. We cloud-based the computing architecture of the cattle behavior recognition and monitoring system, and provide an end-to-end solution by deploying it on a cloud server. The system includes a feature-rich Python graphical user interface (GUI), which provides users with a variety of data input options, including obtaining data directly from multiple input sources such as pictures, video files, real-time cameras, and network cameras. The system loads the best network model that has been trained and verified. Users can adjust model parameters such as confidence and threshold size through the GUI interface. These adjustments can optimize the model's recognition accuracy of cattle behavior in real time. The system design takes into account the continuity of the monitoring and analysis process and the needs of post-analysis. Therefore, during or after the monitoring and analysis process, the system supports saving results during or after the monitoring and analysis process, allowing users to conduct subsequent inspections and analyses of the recognition results. In general, the system covers all stages of cattle behavior monitoring, from data collection, transmission, processing, analysis, and full-process automated monitoring to ensure that cattle behavior can be monitored in real time and health warnings can be issued immediately when initial signs appear. The main interface of the cattle behavior detection system is as follows: Figure 6 shown.

[0109] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A cattle behavior recognition method based on a multi-scale attention dual-stream 3D convolutional network, characterized by: The following steps are involved: Step 1: Construct a data set and preprocess the collected videos of different daily behaviors of cattle; Step 2: Based on the data set constructed in step 1, capture the RGB image and calculate the optical flow image corresponding to the RGB image; Step 3: Construct EMA module; Step 4: Embed the EMA module constructed in step 3 into InflatedI3D, and integrate the Non-Local operation into the I3D network to construct M3D; Step 5: Build a two-streamM3D network to obtain the output of the two branches: RGB M3D and OpticalFlowM3D. Step 6: Spatial flow features F output by the two branches of RGB M3D and OpticalFlowM3D based on the two-streamM3D network space and the time flow feature F time , the output features of the two branches are fused by feature concatenation, the spatial channel dependency is established, and the fused feature F combined After inputting the constructed efficient cattle behavior classifier Classifier, the final classification result of the cattle behavior is obtained.

2. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1 is characterized in that: The specific steps of step 1 include: (1) Preliminary preprocessing was performed on the videos of different daily behaviors of cattle collected in the natural environment. The video clips of each behavior of cattle were extracted into 256×256 pixel frames. The peripheral areas of the input images were zero-filled to obtain the Kinectics400 format cattle behavior recognition dataset, which was named CSIROV1. (2) Create a new cattle behavior dataset named CSIROV2. Clip videos longer than 5 seconds in the CSIROV1 dataset into 5-second segments, and filter out clips shorter than 3 seconds. Each clipped video contains an independent behavior. (3) The videos in the CSIRO V2 dataset are classified and uniformly named in the format of "action name + number + splits number", and the training and validation datasets are divided.

3. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1 is characterized in that: The specific method of calculating the optical flow image corresponding to the RGB image in step 2 is: For each behavior video in the CSIRO V2 dataset constructed in step 1, capture an RGB image every 10 frames, then use the TVL1 optical flow algorithm implemented in opencv to calculate the captured spatial flow RGB image sequence, initialize the DualTVL1OpticalFlow object, and then load the spatial flow image sequence. For the adjacent image frame pairs frame-t and frame-t+1 in the sequence, call the calc method to estimate the horizontal flow-x and vertical flow-y optical flow components; Among them, the calculation of optical flow is based on the equation: and are the gradients of the horizontal and vertical optical flow components respectively, and each optical flow image sample E cattle is an optical flow vector, which is used to measure the overall error of optical flow estimation. Each optical flow vector consists of a horizontal optical flow component u and a vertical optical flow component v, which represent the pixel displacement in the horizontal and vertical directions respectively; I cattle (x, y, t) represents the brightness of the pixel at time t and position (x, y), I cattle (x+u,y+v,t+1) represents the brightness of the pixel position (x+u,y+v) after the estimated displacement (u,v) in the next time frame t+1; λ is a weight parameter that balances the data term and the smoothness term, and adjusts the accuracy and smoothness of the optical flow estimation.

4. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1, characterized in that: The structure of the EMA module constructed in step 3 is: The EMA module design adopts feature grouping and parallel processing strategies. The EMA module accepts an input feature map with C channels and H×W×T spatial dimensions, and divides the input feature map into g groups along the channel dimension, each group contains C / / g channels; The EMA module consists of three parallel processing branches, which are used for global information encoding and local information enhancement to extract the attention weight descriptor of the grouped feature map: Among them, two parallel branches are 1x1x1 branches, and the third branch is a 3x3x3 branch; The 1x1x1 branch encodes the channels along two spatial directions through 3D global average pooling, then connects the encoded features in the image height direction and shares the same 1x1x1 convolution. After decomposing the output of the 1x1x1 convolution into two vectors, two nonlinear Sigmoid functions are used to fit the 3D bilinear distribution on the linear convolution. The 3x3x3 branch directly acts on the original feature map through a 3x3x3 kernel, captures local cross-channel interactions to expand the feature space, and outputs it through the Sigmoid activation function to provide attention weights in the spatial dimension; Finally, the output feature maps of the 1x1x1 convolution branch and the 3x3x3 convolution branch are directly added and fused in the corresponding dimensions, and the fused feature maps are input into another Sigmoid function for the final attention modulation.

5. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1, characterized in that: The specific steps of constructing M3D in step 4 include: (1) First, the front end of M3D is constructed, which consists of multiple 3D convolution and 3D maximum pooling layers. These layers use three-dimensional convolution modules for downsampling and feature extraction, and are responsible for extracting the basic spatial features of video frames; the first convolution layer: uses a 7x7x7 convolution kernel with a step size of 2x2x2; the second maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2; the third convolution layer: uses a 1x1x1 convolution kernel with a step size of 1x1x1; the fourth convolution layer: uses a 3x3x3 convolution kernel with a step size of 1x1x1; the fifth maximum pooling layer: uses a 1x3x3 convolution kernel with a step size of 1x2x2; (2) The output of the fifth maximum pooling layer is then input into the EMA module to filter out irrelevant background information of the video. The EMA module divides the input feature map into multiple groups along the channel dimension, each group contains the same number of channels; it contains three parallel processing branches, two of which use 1x1x1 convolution kernels to perform feature weighting after global average pooling, and the other 3x3x3 branch is used to capture local features; the parallel branches merge their outputs through the Sigmoid activation function to obtain the adjusted feature map. The final output of EMA has the same size as its input; (3) The features output by the EMA module are further refined through a series of Inception modules. Each Inception module combines convolution kernels of different sizes: 1x1x1 and 3x3x3 to meet the needs of capturing different spatial details. (4) In the later stage of building the M3D network, the high-level features processed and fused by the Inception module at different scales are input into the Non-Local module. The Non-Local module captures long-distance dependency information by calculating the pairwise relationships between all positions in the input feature map. Through the self-attention mechanism, the input feature map is globally analyzed. Its calculation is not limited to neighboring pixels, but involves all positions in the entire feature map. (5) After multiple rounds of experimental verification, it was chosen to introduce EMA before the Inc-M-3c module, and the Non-Local module was placed before Inc-M-4f; (6) In the final stage of the M3D network, the output of the Non Local module is first processed by the Inc-M-4f module, and then passed to a maximum pooling layer with a 2x2x2 convolution kernel and a stride of 2x2x2; the processed features flow into the Inc-M-5b and Inc-M-5c modules, and then enter a global average pooling layer with a 2x7x7 convolution kernel and a stride of 1x1x1; thereafter, the features are sent to the Logits fully connected layer to map the complex three-dimensional features into a one-dimensional feature vector, which is then input into a Softmax classifier. The Softmax classifier maps each feature vector to a predefined cattle behavior category based on the weights obtained by the network through learning, thus successfully completing the construction of the M3D network.

6. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1, characterized in that: The specific structure of the two-streamM3D in step 5 is: Two-streamM3D consists of two parallel branches. The upper branch is the RGB M3D network, which processes the original RGB video frame sequence and extracts spatial features related to visual appearance. The lower branch is the Optical FlowM3D network, which processes the optical flow image frame sequence generated by calculating the difference of superimposed video frames to capture the dynamic motion information of the cow; two independent network branches are used to extract the features of the RGB image flow and the optical flow image frame sequence respectively, thereby obtaining the outputs of the RGB M3D and OpticalFlowM3D branches.

7. The method for cattle behavior recognition based on a multi-scale attention dual-stream 3D convolutional network according to claim 1, characterized in that: The specific method of step 6 is: The outputs of the RGB M3D and OpticalFlowM3D branches of the two-streamM3D network are the spatial flow features F space and the time flow feature F time , where the spatial flow feature F space =GCInception s (D space ), used to capture the details and texture information of the static object cow in the video frame, the temporal flow feature F time =GCInception t (D time ), used to capture the fast-changing motion information of cattle in the video frame, D space is the spatial stream image sequence, D time is a temporal stream image sequence; The output spatial flow features F of the two branches of RGB M3D and OpticalFlowM3D are space and the time flow feature F time The information from the RGB image stream and the optical flow image stream is integrated into an advanced fusion model, which uses a consensus fusion mechanism to fuse the information from the two branches of the RGB image stream and the optical flow image stream to establish spatial channel dependencies; feature fusion equation: F combined =σ(W f2 ·σ(W f1 Concat(F space ,F time )+b f1 )+b f2 ), Concat(·) is feature concatenation, W f1 , W f2 , b f1 , b f2 are the weight and bias parameters of the fusion network, σ is the nonlinear activation function; Then, the fused feature F combined Input the constructed efficient cattle behavior classifier Classifier; the classification process uses the Softmax function to output the predicted probability of each behavior state, and finally the classifier Classifier obtains the final classification result of the cattle behavior state based on the output predicted probability.

Images