A sow lactation behavior fine-grained identification method based on computer vision

Abstract

The application discloses a kind of sow lactation behavior fine-grained identification method based on computer vision, steps are as follows: collecting sow and piglet's down video during lactation period;Establish sow lactation behavior fine-grained classification data set;Train sow lactation behavior fine-grained classification behavior recognition model;Long section monitoring video is input into behavior recognition model, and the behavior category label sequence corresponding to sow lactation behavior fine-grained classification is generated;Using the prior knowledge of sow lactation behavior, combined with hidden Markov model, the problem of sow lactation behavior fine-grained classification is modeled, and the category label sequence is corrected using Viterbi algorithm, and the classification results of four categories of piglet weaning, lactation, interrupted lactation and non-lactation state are output.The application can avoid the shielding interference of sow delivery bed limiting fence as much as possible, has strong robustness, and consumes less computing resources, runs faster, and is suitable for video monitoring system in actual breeding environment.

Description

Technical Field

[0001] This invention relates to image processing, computer vision, and interactive behavior recognition, and in particular to a fine-grained method for recognizing sow lactation behavior based on computer vision. Background Technology

[0002] Statistics show that under large-scale, enterprise-managed pig farming, the pre-weaning mortality rate for piglets can reach 10%-15%. The average number of healthy piglets produced per sow is a crucial factor affecting a pig farm's profitability. Enterprises strive to intervene artificially in sow selection and lactation to fully exploit their reproductive and lactation potential, aiming for larger litters. Therefore, data collection and analysis of sow lactation behavior are essential for effective intervention. However, manually recording sow lactation status and duration requires significant manpower. Furthermore, manual recording can lead to subjective biases. Therefore, analyzing lactating sow behavior using automatic identification technology has become a critical research topic in pig farming. Current research primarily focuses on single identification techniques, such as whether a sow is lactating, neglecting information about the initiation and termination of lactation. In actual production, the manner and duration of the initiation and termination of lactation largely reflect the sow's lactation habits and level of maternal instinct. Therefore, detailed classification and information collection of the sow lactation process are of great significance for behavioral analysis of different breeds of sows and for the selection and culling of sows in pig farms.

[0003] In the field of animal behavior recognition, scholars have conducted extensive research on behavior classification techniques based on wearable sensors. However, sensors worn on pigs are prone to damage from friction and detachment. With technological advancements, non-contact computer vision technology has begun to be used for the behavior and posture recognition of sows. Typical methods include extracting specific parameters from images and establishing judgment criteria based on these parameters. Zhu Weixing et al. obtained pig contours by using the Otsu thresholding method on pig images and judged pig behavior by calculating contour similarity. Based on this method, they applied for a patent titled "Contour-Based Pig Drinking Behavior Recognition Method" (Publication No. CN107437069A). While such methods can recognize simple behaviors and postures, they do not consider the temporal motion characteristics of pigs during the recognition process, limiting their ability to recognize complex behaviors. Furthermore, in the field of sow behavior recognition, Xue Yueju and Aqing Yang et al. combined deep neural networks and optical flow methods for judging sow lactation behavior. Their findings were published in the international journals *Biosystems Engineering* and *Computers and Electronics in Agriculture*, and they applied for a patent, "A Computer Vision Method for Recognizing Sow Lactation Behavior" (Publication No. CN109492535A), which discloses a method for recognizing lactation behavior using optical flow, convolutional neural networks, and support vector machines. However, this type of sow lactation behavior recognition method only considers the identification of whether lactation behavior has occurred and cannot extract detailed information about the lactation process.

[0004] Therefore, providing a method for achieving detailed classification of sow lactation behavior, i.e., fine-grained classification and identification, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a fine-grained recognition method for sow lactation behavior based on computer vision. This method utilizes a two-stream neural network model to initially detect the fine-grained behavior categories of sow lactation, then uses a Hidden Markov Model to model the fine-grained classification problem of sow lactation behavior, and finally uses the Viterbi algorithm to correct and obtain the final state sequence, thereby achieving fine-grained recognition of lactation behavior.

[0006] Technical solution: The present invention provides a fine-grained method for recognizing sow lactation behavior based on computer vision, comprising the following steps:

[0007] (1) Collect overhead videos of lactating sows and piglets.

[0008] (2) Establish a fine-grained classification dataset of sow lactation behavior.

[0009] (3) Construct and train a behavior recognition model for fine-grained classification of sow lactation behavior in a dual-flow structure.

[0010] (4) Input the monitoring video into the trained model to generate a category label sequence for fine-grained classification of sow lactation behavior.

[0011] (5) Preprocess the behavior category label sequence.

[0012] (6) Set the parameters of the Hidden Markov Model, use the Viterbi algorithm to correct errors, and output the final fine-grained classification results of sow lactation behavior.

[0013] The dataset mentioned in step (2) includes a training dataset for the behavior recognition model and a dataset of sow lactation video clips to verify the feasibility of the method.

[0014] Step (3) specifically involves:

[0015] (3.1) Construct a neural network behavior recognition model with a dual-stream structure to achieve feature extraction of videos with different frame rates.

[0016] (3.2) The extracted features are fused and classified based on the fused feature results.

[0017] (3.3) Input the dataset into the constructed behavior recognition model and train the model.

[0018] Step (4) specifically involves:

[0019] (4.1) The long video segment is divided into short video segments and input into the trained behavior recognition model. The behavior category label sequence of each short video segment is output and saved one by one.

[0020] (4.2) When the entire long video is segmented to the end, obtain the behavior category label sequence of the entire video.

[0021] The preprocessing method described in step (5) includes filtering and binarizing the category label sequence.

[0022] Step (6) specifically involves:

[0023] (6.1) The parameters of the Hidden Markov Model are set according to the logical relationship between the four categories of sow lactation fine-grained classification, and the Hidden Markov Model is used to model the problem of fine-grained classification of sow lactation behavior.

[0024] (6.2) Input the observation sequence composed of the observations into the Viterbi algorithm, use the Viterbi algorithm to correct errors and output the final fine-grained behavior classification label results.

[0025] A computer storage medium storing a computer program that, when executed by a processor, implements the aforementioned fine-grained recognition method for sow lactation behavior based on computer vision.

[0026] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described fine-grained recognition method for sow lactation behavior based on computer vision.

[0027] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0028] 1. A fine-grained classification method for sow lactation behavior based on a behavior recognition technology framework can achieve fine classification of the sow lactation process, thereby extracting richer information;

[0029] 2. The core dual-stream neural network behavior recognition technology and the classification sequence correction technology based on behavior transformation prior knowledge and hidden Markov models have lower sample labeling costs and computational resource requirements than image segmentation and object detection technologies. They are also robust, consume less computational resources, and run faster, making them suitable for application in actual aquaculture environments. Attached Figure Description

[0030] Figure 1 This is a flowchart of the steps of the present invention;

[0031] Figure 2 This is a diagram of the neural network structure of the SlowFast model used in one embodiment of this paper;

[0032] Figure 3 This is a schematic diagram illustrating a method for modifying the label sequence of sow lactation behavior using a hidden Markov model, as described in one embodiment of this paper. Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, a fine-grained method for identifying sow lactation behavior based on spatiotemporal feature fusion includes the following steps:

[0035] S1. Collect overhead video of lactating sows and piglets; install a camera directly above the pigsty at a height of 2.2-2.3m to acquire overhead video of one sow and 8-12 piglets. In this embodiment, the video frame resolution is 2048×1536 pixels, the frame rate is 30 frames / second, the farrowing crate is 2.37m long, 1.75m wide, and 0.5m high, the sow breed is Huang-Huai-Hai Black Pig, and the piglets are 1-28 days old after farrowing.

[0036] S2. Establish a fine-grained classification video dataset of sow lactation behavior.

[0037] S21. Classify sow lactation behaviors into fine-grained categories; based on behavioral characteristics, classify sow lactation-related behaviors into four categories: piglets suckling, currently lactating, interrupted lactation, and non-lactation states; detailed definitions are shown in Table 1.

[0038] Table 1. Definitions of Fine-Grained Behavior Classification in Sows During Lactation

[0039] category Action description Piglets suckling PAM Less than half of the piglets suckle at the sow's udder. Breastfeeding SBF Most piglets suckled continuously in the sow's udder area. Interruption of breastfeeding EBF The sow's large-scale trunk swinging caused the lactation to cease. Non-breastfeeding NBF Behaviors unrelated to breastfeeding

[0040] S22. Establish a fine-grained classification video dataset for sow lactation behavior; edit and organize the original videos, and denote the acquired data as the training set, validation set, and test set. In this embodiment, the image size is uniformly maintained at 2048×1536. The videos used for testing and evaluation include two types: short videos of 30 seconds and long videos of 15-30 minutes, and datasets P and Q are established for each. Dataset P includes: training set, validation set, and test set. Its data composition is shown in Table 2.

[0041] Table 2. Definitions of Fine-Grained Behavioral Classifications for Sow Lactation (Dataset)

[0042]

[0043]

[0044] S3. Construct and train a behavior recognition model for fine-grained classification of sow lactation behavior using a two-stream structure; the neural network model used in this embodiment is the SlowFast model, and its schematic diagram is shown below. Figure 2 As shown.

[0045] S31. Construct the backbone of the two-stream network; the main structure includes two parts: a slow channel and a fast channel. Both have the same structure, arranged in the following order according to the network structure: input layer, convolutional layer, pooling layer, residual block 1, residual block 2, residual block 3, and residual block 4. The number of channels in the convolutional layer and the four residual blocks in the slow channel are 64, 64, 128, 256, and 512 respectively. The number of channels in the corresponding layer of the fast channel is 1 / 8 of the number of channels in the slow channel.

[0046] S32. Implement feature fusion in a two-stream network. The slow and fast channels are fused using lateral connections to achieve different spatiotemporal resolutions. Lateral connections are performed after the pooling layer, residual block 2, residual block 3, and residual block 4. The feature layer output from the spatiotemporal feature extraction network is then subjected to 3D convolution with a kernel size of 5×1×1, and normalized. The ReLU activation function is used in the convolutional layer. The processed fast channel features are then concatenated with the corresponding dimensions of the slow channel features.

[0047] S33. Classification based on fused features: The final output features of the two network channels are converted into vectors of length 512 and 64 respectively using global average pooling. The two vectors are then fed into the classification layer, and classification is achieved using the softmax activation function.

[0048] S34. Train the SlowFast model using dataset P; and use the validation set to validate the weight file generated in each training cycle. In this embodiment, the batch size is set to 8 during training, and 110 training cycles are performed. The weight file with the best performance on the validation set is used for subsequent recognition.

[0049] S4. Input the dataset Q into the trained SlowFast model to identify and generate a preliminary classification sequence of fine-grained classification of sow lactation behavior.

[0050] S41. Use ffmpeg to edit and segment the long video; perform loop segmentation of the long video segments according to time sequence. In this embodiment, the video is segmented into 30-second segments.

[0051] S42. Input the segmented short videos one by one into the SlowFast model to perform video content recognition.

[0052] S43. Record the model's recognition results for the sow lactation video; the model outputs the video classification label and confidence level. In this embodiment, the label results with a confidence level greater than 0.3 are saved and recorded one label per second.

[0053] S44. Save the category label sequence result of the entire video segment; when the entire surveillance video segment is cut to the end, save and output the category label sequence of the entire video segment.

[0054] S5. Preprocess the behavior category label sequence to obtain the preprocessed fine-grained category label sequence of sow lactation behavior.

[0055] S51. Median filtering is applied to the piglets in suckling, nursing and non-lending states. In this embodiment, median filtering with a step size of 60 is used to process these three categories in the obtained sequence, and the results after filtering are recorded.

[0056] S52. Binarize the interrupted lactation behavior in the category sequence; in this embodiment, based on the confidence level of the interrupted lactation behavior, values ​​greater than 0.5 are set to 1, and values ​​less than 0.5 are set to 0. Record the processed results to generate a preprocessed fine-grained classification label sequence for sow lactation behavior.

[0057] S6. A Hidden Markov Model (HMM) is used to model the fine-grained classification problem of sow lactation behavior. The Viterbi algorithm is used to correct errors in the classification label sequences. A schematic diagram of the HMM used for correction is shown below. Figure 3 As shown, the main parameters used in the correction process are denoted as {A, B, o}. t i t Let A and B be the observation probability matrix and the state transition probability matrix, respectively. t and i t Let represent the observed value and the state value at time t, respectively;

[0058] S61. Set the parameters for the Hidden Markov Model. The principle is to set higher observation probabilities for two categories that are prone to misjudgment, and to set extremely low transition probabilities between two behaviors that do not conform to the rules of breastfeeding behavior. In this embodiment, the observation probability matrix is ​​shown in Table 3, and the state transition probability matrix is ​​shown in Table 4.

[0059] Table 3 Observation State Probability Matrix A

[0060]

[0061] Table 4. State transition probability distribution matrix B

[0062]

[0063] S62. Divide the preprocessed category label sequence into observation sequences; in this embodiment, the preprocessed category label sequence of fine-grained classification of sow lactation behavior is used as the observation sequence, and is grouped into sets of {o1, o2, o3, o4...o3} every 30 minutes. t}

[0064] S63. Input the observation sequence into the Viterbi algorithm; read the text content of the 30-minute observation sequence.

[0065] S64. Output and record the state sequence; denote the output state sequence as {i1, i2, i3, i4...i...} t Save this as the revised classification result of sow lactation behavior.

[0066] Implementing all or part of the processes in the above embodiments can be accomplished by computer program instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes as described in the above embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory. In summary, by adopting the above solution, this invention provides a new approach and method for fine-grained behavior recognition of sows nursing in a farrowing house environment. It effectively overcomes the disadvantages of traditional manual monitoring methods, which are time-consuming and labor-intensive. It introduces fine-grained classification of nursing behavior based on traditional sow nursing behavior, allowing for the extraction of more useful information from the sow nursing process.

Claims

1. A fine-grained method for recognizing sow lactation behavior based on computer vision, characterized in that, Includes the following steps: (1) Collect overhead videos of lactating sows and piglets; (2) Classify sow lactation behaviors into fine-grained categories. Based on the behavioral characteristics, sow lactation-related behaviors are classified into four categories: piglets suckling, currently lactating, interrupted lactation, and non-lactation. A fine-grained classification dataset of sow lactation behaviors is established. (3) Construct and train a behavior recognition model for fine-grained classification of sow lactation behavior in a dual-flow structure; (4) Input the video into the trained model to generate a category label sequence for fine-grained classification of sow lactation behavior; (5) Preprocess the behavior category label sequence; (6) Set the parameters of the Hidden Markov Model, take the category label sequence of the preprocessed fine-grained classification of sow lactation behavior as the observation sequence of the Hidden Markov Model, use the Viterbi algorithm to correct errors, and output the final fine-grained classification result of sow lactation behavior.

2. The method for fine-grained recognition of sow lactation behavior based on computer vision according to claim 1, characterized in that, The dataset mentioned in step (2) includes a training dataset for the behavior recognition model and a dataset of sow lactation video clips to verify the feasibility of the method.

3. The method for fine-grained recognition of sow lactation behavior based on computer vision according to claim 1, characterized in that, Step (3) specifically involves: (3.1) Construct a neural network behavior recognition model with a dual-stream structure to achieve feature extraction from two different input frame rate videos; (3.2) The extracted features are fused; (3.3) Input the dataset into the constructed behavior recognition model and train the model.

4. The method for fine-grained recognition of sow lactation behavior based on computer vision according to claim 1, characterized in that, Step (4) specifically involves: (4.1) Using the trained behavior recognition model, the long video segment is divided into short video segments and input into the model, and the behavior category label sequence of each short video segment is saved; (4.2) When the entire long video is segmented to the end, obtain the behavior category label sequence of the entire video.

5. The method for fine-grained recognition of sow lactation behavior based on computer vision according to claim 1, characterized in that, The preprocessing method described in step (5) includes filtering and binarizing the category label sequence.

6. The method for fine-grained recognition of sow lactation behavior based on computer vision according to claim 1, characterized in that, Step (6) specifically involves: (6.1) Set the parameters of the Hidden Markov Model by combining the logical relationship between the four categories of fine-grained classification of sow lactation, and use the Hidden Markov Model to model the fine-grained classification problem of sow lactation behavior. (6.2) Input the observation sequence composed of the observations into the Viterbi algorithm, use the Viterbi algorithm to correct errors and output the final fine-grained behavior classification label results.

7. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements a fine-grained recognition method for sow lactation behavior based on computer vision as described in any one of claims 1-6.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a fine-grained recognition method for sow lactation behavior based on computer vision as described in any one of claims 1-6.

Images