Method for recognizing and judging health condition of dairy cow based on video skeleton

Abstract

The application discloses a method for recognizing and judging the health condition of a dairy cow based on video skeleton, and relates to the technical field of video behavior recognition, and comprises the following steps: step one: a camera collects images and sends the images to a terminal for processing, manual labeling and detection of key points of a dairy cow skeleton; step two: a Bottom-up posture estimation method algorithm is used to obtain the posture of the dairy cow, the Bottom-up posture estimation method comprising feature extraction, landmark positioning, part grouping and tracking; feature extraction: a parallel algorithm of bilinear interpolation is used to process the images, BatchNormalization is used for batch data standardization, self-attention gating is used to highlight the significant features of the region of interest, and the U-net feature extraction network model is improved; the application applies the skeleton calibration technology to the dairy cow to recognize the behavior of the dairy cow, and the recognition accuracy is increased and the innovation is improved; compared with the skeleton data, other modalities will produce more calculation consumption, and the robustness is insufficient when facing complex backgrounds and human scale changes, view angle changes and motion speed changes.

Description

Technical Field

[0001] This invention relates to the field of video behavior recognition technology, and more specifically to a method for judging the health status of dairy cows based on video skeleton behavior recognition. Background Technology

[0002] The modernization level of dairy farming has gradually improved. In 2019, China's milk production reached 32.012 million tons, a year-on-year increase of 4.1%. In 2020, the proportion of large-scale dairy farms (dairy farms with more than 100 cows) in China was nearly 70%, with 6,150,000 dairy cows and milk production of 34.4 million tons, a year-on-year increase of 7.5%. With the improvement of people's living standards, there are higher requirements for the demand and quality of dairy products. Although my country's dairy industry has developed rapidly, many problems still exist. Among them, low levels of informatization, rough management under large-scale farming, and insufficient ability to quickly perceive and control stress behaviors in dairy cows are important reasons hindering the development of dairy farming. Currently, my country's dairy farming is accelerating its shift from extensive, scattered farming to standardized, large-scale farming, paying more attention to refined and scientific management in the farming process. my country's dairy farming industry will inevitably enter a new stage of development.

[0003] Animal behavior is an animal's response to stimuli from its environment or its own body. An animal's internal state, including its physiology, pathology, and nutrition, influences its external behavior. Basic motor behaviors include lying down, standing, walking, drinking, and eating, which indirectly reflect the health status of dairy cows. Dairy cows have their own unique living habits and behavioral characteristics. In dairy farming, reproductive monitoring and the monitoring of cow health are crucial. A correct understanding and mastery of basic motor behaviors in dairy cows, aiming to achieve intelligent perception of their health status during dairy farming, is beneficial for the prevention and treatment of dairy cow diseases, and is of great significance for improving the economic efficiency of farms and the welfare of dairy cows. Therefore, conducting real-time, non-contact, and stress-free perception of basic motor behaviors in dairy cows is of great importance for improving the modernization level of the dairy farming industry.

[0004] Intelligent perception of animal behavior using video analytics has become a research hotspot in precision animal husbandry. To achieve automatic acquisition of dairy cow health status information, contactless intelligent monitoring of dairy cows is an important direction for the intelligent development of information acquisition in the dairy farming industry. Video analytics and image processing technologies have attracted increasing attention from researchers. This study develops an online real-time target dairy cow basic movement behavior recognition algorithm. By analyzing dairy cow monitoring videos in natural environments and using skeletons as key points, it achieves real-time online detection, tracking, and reclining behavior recognition of dairy cows. Therefore, developing deep learning-based dairy cow basic movement behavior recognition is of great significance for large-scale precision dairy farming, intelligent health status perception, and improving dairy cow welfare. With the continuous development of the dairy industry, dairy cow comfort has become a bottleneck affecting dairy cow production performance. More and more dairy farmers have realized that improving comfort can not only improve animal welfare but also significantly improve the results of various funded projects.

[0005] Current common methods for recognizing basic bovine motor behaviors have the following problems: (1) they consume a lot of manpower and resources, resulting in high labor costs, low work efficiency, and high management difficulty; (2) they are severely affected by environmental noise, making it difficult to guarantee monitoring quality; and (3) they are prone to causing stress reactions, leading to misjudgments of behavior. Although contact sensor-based methods have high accuracy, they suffer from high system hardware costs and difficult system maintenance. The information collected is mostly external physical parameters of the bovines, making it difficult to identify advanced bovine behaviors. This study addresses the above problems by sampling intelligent video monitoring, which can perform real-time monitoring with minimal human intervention and has a very high accuracy rate, making it an effective means to improve the management level of dairy farming. Currently, research on animal behavior perception based on video analysis is receiving increasing attention from researchers. Summary of the Invention

[0006] The purpose of this invention is to provide a method for recognizing dairy cow behavior and judging the health status of dairy cows based on video skeletons. This method solves the problems of common dairy cow basic movement behavior recognition methods, which consume a lot of manpower and resources, have high labor costs, low work efficiency, and are difficult to manage; are severely affected by environmental noise, making it difficult to guarantee monitoring quality; and are prone to stress reactions that lead to misjudgments of behavior.

[0007] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0008] Step 1: The camera captures images, which are then sent to the terminal for processing. Key points of the cow's skeleton are manually labeled and detected.

[0009] Step 2: The bottom-up pose estimation method is used to obtain the cow's pose. The bottom-up pose estimation method includes feature extraction, landmark localization, part grouping, and tracking.

[0010] Feature extraction:

[0011] The image is processed using a parallel algorithm of bilinear interpolation, batch data is normalized using BatchNormalization, and self-attention gating is used to highlight the salient features of the region of interest. The U-net feature extraction network model is improved, and the features of CMs and PAF are output.

[0012] Landmark location:

[0013] A confidence map is generated from the two-dimensional skeletal key points of the cow labeled in the video. The confidence maps are then aggregated using operators to generate the true confidence map of the network prediction, thereby locating the skeleton nodes.

[0014] Grouping by body part:

[0015] After detecting numerous skeletal key points, clustering is performed on these key points to connect multiple skeletal key points from different cows, generating multiple individual cows and thus obtaining multiple cow poses.

[0016] track:

[0017] In each frame of the image, the similarity between cow proposals, key points, and adjacent frames is detected, and the entire video is tracked to construct a pose flow using spatiotemporal information.

[0018] Step 3: After obtaining the results using the Bottom-up pose estimation method, the improved U-net network is obtained, and the CNN-GRU network is trained for behavior classification.

[0019] Step 4: Train and test the network performance using the collected cow images in the database to determine the cow's behavior based on the skeletal key points in the video.

[0020] Preferably, the bilinear interpolation algorithm in step two is as follows: Given four function values ​​of a function on a two-dimensional plane, namely Q11 = (x1, y1), Q12 = (x1, y2), Q21 = (x2, y1), and Q22 = (x2, y2), first perform linear interpolation in the y-direction to obtain R1 and R2, and then perform two linear interpolations in the x-direction to obtain point P, thus obtaining the desired two-dimensional coordinates of f(x, y).

[0021] Linear interpolation in the y-direction:

[0022]

[0023] Single linear interpolation in the x-direction:

[0024]

[0025]

[0026] Substituting the result of the single linear interpolation in the x-direction into the linear interpolation in the y-direction, we obtain:

[0027] f(x,y)=f(Q 11 (x²-x)(y²-y)+f(Q) 21 (x-x1)(y2-y)+f(Q) 12 (x²-x)(y-y¹)+f(Q) 22 (x-x1)(y-y1)

[0028] Image information is stored in a two-dimensional array of nW*nH. The image is processed by n processes. One main process is responsible for initialization, sending data before processing and receiving data after processing. The remaining n-1 slave processes are responsible for interpolating the image data and sending the interpolated data to the main process. In other words, multiple processes process the image simultaneously. The parallel algorithm of bilinear interpolation is used to process the image.

[0029] For the boundaries of the image, each process processes an extra row of data at the boundary, so that the data processed by multiple processes can completely cover the entire image.

[0030] Preferably, in step two, BatchNormalization performs standardization on each dimension:

[0031] Input:Values ​​of x over a mini-batch:B={x 1...m};

[0032] Parameters to be learner:γ,β

[0033] Output: {y i =BN γ,β (x i )}

[0034]

[0035]

[0036]

[0037]

[0038] Its operation can be divided into two steps:

[0039] First, standardize the m x values ​​to obtain a distribution with a mean unit variance of zero;

[0040] Then on Perform scaleandshift to scale and shift to a new distribution y with a new mean β and variance γ.

[0041] Preferably, to handle different image sizes and ensure that the feature map after convolution has the same size as the original image, padding is added, i.e., Padding = SAME. The formulas for calculating the length and width of the output feature map are as follows:

[0042]

[0043] Where Wout is the size of the output feature map, Win is the size of the input feature map, F is the kernel size, stride is the convolution stride, and Padding is the number of padding loops for the feature map.

[0044] Preferably, in step two, self-attention gating is used for dense label prediction in a standard CNN-based image analysis model, where the attention coefficient is... Highlight salient image regions and suppress task-irrelevant feature responses. Multiplication with feature maps is element-wise; the formula for additive attention is:

[0045]

[0046]

[0047] in, It's a ReLU function. It is the Sigmoid function, W g W x Both ψ and b are convolution operations. g b ψ These are all bias terms corresponding to the convolution.

[0048] Preferably, the specific steps for landmark localization in step two are as follows: First, generate a separate confidence map for each cow. x j,k It is the ground truth position of body part j of k cows in the image; position The value is defined as:

[0049]

[0050] Among them, σ controls the diffusion of the peak value, and the aggregation of individual confidence graphs by the maximum operator generates the true confidence graph of the network prediction.

[0051]

[0052] The maximum confidence plot is used instead of the average value, so the accuracy of the nearby peaks remains clear.

[0053] Preferably, in step two, the part grouping first predicts the heat map and PAF of the key points in the input image, and then associates them according to the most binary matching of the key points and limbs, and finally obtains the posture of all cows in the image.

[0054] Algorithm flow:

[0055] First, a convolutional network analyzes and processes the image, generating a set of feature maps F, which are then input into the first stage of each branch, where the confidence map S is located. t and vector field L t It is determined by both the input F and the output of the previous stage, that is:

[0056]

[0057]

[0058] Where ρ t and f t It is a CNN used for staged inference;

[0059] To guide the network iteratively in predicting the confidence maps of body parts in the first branch and PAF in the second branch, two loss functions are applied at the end of each stage, with one loss function applied at each branch. L2 loss is used between the estimated predictions and GTmap and PAF, and the loss functions are spatially weighted to address the issue that some datasets cannot completely label all cows. The loss functions at the two branches in stage t are as follows:

[0060]

[0061]

[0062] W represents a binary mask, where W = 0 for non-target locations, and W(p) is zero for unmarked locations p. W(p) with an asterisk represents the ground truth, and W(p) without an asterisk represents the predicted value. L2 loss is used to calculate the loss, and the confidence plot is calculated by finding the maximum value.

[0063]

[0064] f is the overall objective function, where the ground truth of key points and limbs is set using a Gaussian function;

[0065] PAF ground truth design:

[0066] Each PAF is a two-dimensional vector field for each limb. A two-dimensional vector code encodes the direction from one part of the limb to another part. Each type of limb has a corresponding PAF that connects its two associated body parts.

[0067] Here, n represents the keypoint, k represents the k-th key, x represents the position of the keypoint (partaffinityvectorfield), and L... m，k (p) is calculated as follows:

[0068]

[0069] Where P is any point on a limb, for example, if point P is on limb c of the kth cow, then point P is represented by vector V; otherwise, it is 0. That is, the unit vector between two key points;

[0070] Formula for determining whether point P is on limb c:

[0071] 0≤v·(px n1,k )≤l c,k and|v ⊥ ·(px n,k )|≤σ l

[0072] Length between two key points: l c,k =||x n2,k -x n1,k || 2 The width of the limb is σ l ;

[0073] When multiple cows have overlapping limbs, the formula is as follows:

[0074]

[0075] The number of cows overlapping at point P is k, q m (p) is the number of vectors at position P that are not zero;

[0076] The PAF (Personal Affinity Field) is formed by summing and averaging the encoded vectors of all pixels in the entire region. During training, if point P is within the region, the ground truth of point P's vector is the normalized vector v; otherwise, it is a zero vector. Since there are many potential body parts in multi-person images, the final judgment is made by summing and averaging the vector fields corresponding to all body parts. Additionally, it is necessary to determine the "affinity" between keypoints for final combination and representation, which is then used to interpolate the positions of the two body parts.

[0077]

[0078]

[0079] Where n1 and n2 are two candidate positions, and p(u) is the position to be inserted between two cow body parts n1 and n2;

[0080] Predicting multiple dairy cows based on PAFs:

[0081] Let a and b represent the number of key points, using a variable between 0 and 1. To indicate whether two key points are connected, all possible z forms a set;

[0082]

[0083]

[0084]

[0085] Where E is the "relevance confidence" obtained by performing line integration during the test. d For all "relevance confidence" values ​​in the graph for the limbs, E ab Let z be the "relevance confidence" between a and b, z be whether there is an edge between the two key points, a be the a-th point in the n1-th joint set, and b be the b-th point in the n2-th joint set.

[0086] Instead of using the complete graph, we choose the minimum number of edges to obtain the spanning tree skeleton of the cow pose. Secondly, we further decompose the matching problem into a set of binary matching subproblems and independently determine the matches in adjacent tree nodes. Finally, the optimization formula can be summarized as the sum of E on each limb c:

[0087]

[0088] Preferably, the specific tracking steps in step two are as follows: target detection is performed on each frame of the image, and the same targets in each frame of the image are associated to form a PF;

[0089] Use Pi j Let τ(P) represent the i-th pose in the j-th frame, whose candidate matching set is τ(P). i j ),Right now:

[0090] τ(P i j )={P|d c (P,P i j )≤ε},stP∈Ω j+1

[0091] To optimize pose selection, the following objective function is maximized:

[0092]

[0093] stQ0=P i t

[0094] stQ i ∈τ(Q i-1 )

[0095] s(Q i ) = s box (Q i )+mean(s pose (Q i ))+max(s pose (Q i ))

[0096] Then, the optimal PF for frames t to t+T is obtained by solving an optimization problem; the PF is used as the basic unit for NMS processing to determine the distance between two pose flows indicating the same cow.

[0097]

[0098] After calculating the scores between PoseFlows, they are integrated, and d PF When performing NMS using traditional methods, the coordinates and scores of keypoints can be calculated using the following formula:

[0099] and

[0100] Preferably, under the learned model, the category Ck to which the input behavior matrix X belongs can be calculated using the following formula:

[0101]

[0102]

[0103] and

[0104] Where C is the number of categories, T is the length of the test sequence, O is the output vector, and h t The output of the model is the class with the highest probability, and the class with the highest probability is used as the final classification result.

[0105] Preferably, the training process of a convolutional neural network is divided into two stages: Stage 1: the stage where data is propagated from lower levels to higher levels, i.e., the forward propagation stage; Stage 2: the stage where, when the result obtained from forward propagation does not match the expectation, the error is propagated from higher levels to lower levels for training, i.e., the backpropagation stage; the specific training process is as follows:

[0106] S1: The network initializes its weights;

[0107] S2: The input data is propagated forward through the convolutional layer, downsampling layer and fully connected layer to obtain the output value;

[0108] S3: Calculate the error between the network's output value and the target value;

[0109] S4: When the error is greater than the expected value, the error is fed back into the network, and the errors of the fully connected layer, downsampling layer and convolutional layer are calculated in turn; when the error is equal to or less than the expected value, training ends and the weights and thresholds are fixed.

[0110] S5: Update the weights based on the obtained error and proceed to S3.

[0111] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0112] 1. Applying skeletal mapping technology to dairy cows to identify their behavior increases recognition accuracy and enhances innovation. Compared with skeletal data, other modalities generate more computational costs and are less robust to complex backgrounds, changes in human scale, perspective, and movement speed.

[0113] 2. A deep learning network was used to automatically judge the health status of dairy cows based on their lying behavior. This effectively improved the breeding system and increased the accuracy and real-time performance of dairy cow health monitoring compared to manual testing, while greatly reducing manual labor input.

[0114] 3. A database of partial dairy cow video monitoring has been established, which can provide data for various dairy cow studies and increase the scientific rigor of dairy cow research. Attached Figure Description

[0115] Figure 1 This is a schematic diagram of the key points and skeletal structure of the dairy cow in this invention;

[0116] Figure 2 The graph of the coordinates of the unknown function f at point P;

[0117] Figure 3 This is a conceptual framework diagram for parallel algorithms;

[0118] Figure 4 Here is a block diagram of the AttentionGate (AG);

[0119] Figure 5 This is a block diagram of the architecture of a two-branch, multi-level CNN.

[0120] Figure 6 A schematic diagram of PAF ground truth calculation;

[0121] Figure 7 This is a schematic diagram of the computation process of a convolutional neural network.

[0122] Figure 8 This is a flowchart illustrating the training process of a convolutional neural network. Detailed Implementation

[0123] The above-mentioned and other technical features and advantages of the present invention will be described in more detail below with reference to the accompanying drawings.

[0124] Example 1

[0125] This embodiment provides a technical solution: a method for judging the health status of dairy cows based on video skeleton behavior recognition, including the following steps:

[0126] Step 1: The camera captures images, which are then sent to the terminal for processing. Key points of the cow's skeleton are manually labeled and detected.

[0127] The skeletal position of the cow was described by a side-view cow structure model. Four infrared cameras and one fisheye camera were set up to collect cow movement data in different scenarios.

[0128] like Figure 1As shown, a side-view cow structural model is used to describe the location of cow joints (key points). The 16 joints connecting the cow's skeleton, namely the head, neck, shoulder, hip, left forearm (LFL_root), left forearm (LF_knee), left forearm (LF_hoof), right forearm (RFL_root), right forearm (RF_knee), right forearm (RF_hoof), left hind elbow (LBL_root), left hind knee (LB_knee), left hind hoof (LB_hoof), right hind elbow (RBL_root), right hind knee (RB_knee), and right hind hoof (RB_hoof), are set as the key body parts to be detected.

[0129] Cows' lying-down behaviors can be categorized into crawling and lying sideways.

[0130] ①. Crawling: alternating between left and right sides; but the first crawling is mostly on the left side, that is, the body twists to the left and the head bends to the right to return to the middle line. Crawling is often accompanied by rumination.

[0131] ②. Lying down: Like a person lying on their side, the first time lying down after eating usually occurs after several crawling and lying down, and almost always the left side is down, with limbs stretched and relaxed. Sometimes they stand up and move around for a while, and when they lie down again, almost 100% of the time the right side is down. When lying down, they generally do not ruminate, close their eyes and rest. Lying down usually occurs when they are extremely comfortable, quiet, have a certain amount of space, and are not disturbed by other cattle.

[0132] Step 2: The bottom-up pose estimation method is used to obtain the cow's pose. The bottom-up pose estimation method includes feature extraction, landmark localization, part grouping, and tracking.

[0133] Feature extraction:

[0134] Parallel implementation of bilinear interpolation algorithm is used to replace bilinear interpolation algorithm, which improves the speed and efficiency of image processing. Batch Normalization is used to standardize batch data. Self-attention gating is used to highlight the salient features of region of interest (ROI) to enhance the performance of feature extraction network model. The U-net feature extraction network model is improved, and then the features of PAF and CMs are input.

[0135] The parallelism of bilinear interpolation algorithms is as follows: Figure 2 As shown, if we need to find the value of the unknown function f at point P = (x, y), we have four known points: Q11 = (x1, y1), Q12 = (x1, y2), Q21 = (x2, y1), and Q22 = (x2, y2).

[0136] First, perform linear interpolation in the y-direction to obtain R1 and R2. Then, perform two linear interpolations in the x-direction to obtain point P. The desired result f(x,y) can then be obtained (Note: the result of linear interpolation is independent of the interpolation order).

[0137] ①. Linear interpolation in the y-direction:

[0138]

[0139] ②. Single linear interpolation in the x-direction:

[0140]

[0141]

[0142] Substituting the result of the single linear interpolation in the x-direction into the linear interpolation in the y-direction, we obtain:

[0143] f(x,y)=f(Q 11 (x²-x)(y²-y)+f(Q) 21 (x-x1)(y2-y)+f(Q) 12 (x²-x)(y-y¹)+f(Q) 22 (x-x1)(y-y1)

[0144] Parallel algorithms use multiple processes to process images simultaneously, changing the traditional approach of using a single process to process images. They make efficient use of the CPU while trying to balance the load during the operation, effectively improving the efficiency and speed of image processing.

[0145] like Figure 3 As shown, assuming the image information is stored in a two-dimensional array of size nW*nH, and the image is processed by n processes, one master process is responsible for initialization, sending pre-processing data, and receiving post-processing data. The remaining n-1 slave processes are responsible for interpolating the image data and sending the interpolated data back to the master process. Multiple processes can process the image simultaneously, thus improving the image processing speed. For the image boundaries, an overlapping row method is used, meaning each process processes an extra row of data at the boundary, ensuring that the data processed by multiple processes completely covers the entire image. Since the matrix is ​​stored row-wise, distributing the image information row-wise to the n-1 processes is relatively simple, but this may not guarantee load balancing. Considering the situation where the master process is idle while the slave processes are processing data, this algorithm specifies that nH / n rows of data are allocated to each slave process, and nH%n rows of data are allocated to the master process. This maximizes load balancing while also improving processor utilization.

[0146] BatchNormalization: Similar to ordinary data standardization, it is a way to unify scattered data and also a method to optimize neural networks. BatchNormalization does not rely too much on dropout and has a certain regularization effect, which can reduce overfitting.

[0147] "The input X has d dimensions, and each dimension is standardized." Assuming the input x is an RGB three-channel color image, i.e., d=3, where d represents the feature matrix corresponding to the R channel. And so on, the three channels are standardized respectively.

[0148] Input:Values ​​of x over a mini-batch:B={x 1...m};

[0149] Parameters to be learner:γ,β

[0150] Output: {y i =BN γ,β (x i )}

[0151]

[0152]

[0153]

[0154]

[0155] Its operation can be divided into two steps:

[0156] ①.Standardization: First, standardize the m x values ​​to obtain the distribution of zero mean unit variance.

[0157] ②.scaleandshift: Then then... Perform scaleandshift to scale and shift to a new distribution y with a new mean β and variance γ.

[0158] To handle different image sizes and ensure that the feature map after convolution has the same dimensions as the original image, padding is added, i.e., Padding = SAME. The formulas for calculating the length and width of the output feature map are as follows:

[0159]

[0160] Where Wout is the size of the output feature map, Win is the size of the input feature map, F is the kernel size, stride is the convolution stride, and Padding is the number of padding loops for the feature map.

[0161] Introducing AttentionGate: The AttentionGate model automatically learns to distinguish the shape and size of a target. During training, the AttentionGate model learns to suppress irrelevant regions and highlight useful salient features. This application proposes a novel self-attentiongating module, employing a grid-based AG, which makes the attention coefficients more prominent in local region features. This module can be used for dense label prediction in standard CNN-based image analysis models, such as... Figure 4 As shown.

[0162] Attention coefficient This is to highlight significant image regions and suppress task-irrelevant feature responses. Multiplication with the feature map is element-wise (multiplying corresponding elements one by one). The formula for additive attention is:

[0163]

[0164]

[0165] Let's analyze it together with its structure. It's a ReLU function. It is the Sigmoid function, W g W x Both ψ and b are convolution operations. g b ψ These are all bias terms corresponding to the convolution (so we find W) x (No bias), F int Generally compared to F g F l It should be small.

[0166] Landmark location:

[0167] A confidence map is generated from the two-dimensional skeletal key points of the cow labeled in the video. The confidence maps are then aggregated using operators to generate the true confidence map of the network prediction, thereby locating the skeleton nodes.

[0168] This application uses confidence maps (CM) to locate and identify body parts of cows, generating confidence maps S* from labeled two-dimensional key points. Ideally, if a cow appears in the image, there should be a peak in each confidence map if its corresponding part is visible. If multiple cows appear in the image, then each visible part j of each cow k should have a corresponding peak.

[0169] First, generate a separate confidence map for each cow. x j,k It is the ground truth position of body part j of k cows in the image; position The value is defined as:

[0170]

[0171] σ controls the diffusion of the peak value; the aggregation of individual confidence graphs using the maximum operator generates the true confidence graph of the network prediction.

[0172]

[0173] The maximum confidence plot is used instead of the average value, so the accuracy of the nearby peaks remains clear.

[0174] Grouping by body part:

[0175] After detecting numerous skeletal key points, clustering is performed on these key points to connect multiple skeletal key points from different cows, generating multiple individual cows and thus obtaining multiple cow poses.

[0176] track:

[0177] This application employs a top-down multi-cow pose tracking method, detecting cow proposals → keypoints → similarity between adjacent frames in each frame to track the entire video. It utilizes spatiotemporal information to construct a pose flow, reducing uncertainty and significantly improving the tracking performance. First, target detection is performed on each frame, and identical targets in each frame are associated to form a pose flow diagram (PF).

[0178] P i j Let τ(P) represent the i-th pose in the j-th frame, and let τ(P) be the candidate matching set. i j )

[0179] τ(P i j )={P|d c (P,P i j )≤ε},stP ∈ Ω j+1

[0180] To optimize pose selection, the following objective function is maximized:

[0181]

[0182] stQ0=P i t

[0183] stQ i ∈τ(Q i-1 )

[0184] s(Q i ) = s box (Q i )+mean(s pose (Q i ))+max(s pose (Q i ))

[0185] Then, the optimal PF for frames t to t+T is obtained by solving an optimization problem.

[0186] To address redundancy, parametric attitude nonmaximum suppression (NMS) is performed, using the pose parameter (PF) as the basic unit for NMS processing to determine the distance between two pose flows indicating the same cow.

[0187] d PF (y a ,y b ) = median[{d f (P a 1 .P b 1 ),...,d f (P a N .P b N )}]

[0188] After calculating the scores between PoseFlows, integration is needed to optimize and combine overlapping, partially overlapping, and intersecting PoseFlows. PF In traditional NMS methods, the coordinates and scores of keypoints can be calculated using the following formula:

[0189] and

[0190] Step 3: After obtaining the results using the Bottom-up pose estimation method, the improved U-net network is obtained, and the CNN-GRU network is trained for behavior classification.

[0191] Convolutional Neural Networks (CNNs) are among the most popular deep learning tools for machine vision tasks, and have been widely applied to tasks such as object recognition, target detection, and behavior analysis. Currently, commonly used CNNs include one-dimensional convolutional models for speech signal processing, two-dimensional convolutional models for image analysis, and three-dimensional convolutional models for video analysis. Taking a two-dimensional neural network as an example, it uses a set of filters to filter the input data in a sliding window manner. The entire process is similar to performing a convolution operation between the input data and the filter weights, hence the name convolutional neural network. The implementation characteristics of CNNs mean that they can only process data with regular grid structures, which is naturally well-suited to machine vision. Furthermore, each filter uses relatively few parameters; typically, hundreds of filters can be used per layer to filter an image, making it a primary method for spatial feature extraction in machine vision tasks, such as... Figure 7 The diagram illustrates the computational process of a two-dimensional convolutional network.

[0192] Step 4: Train and test the network performance using the collected cow images in the database to determine the cow's behavior based on the skeletal key points in the video;

[0193] like Figure 8 As shown, the training process of a convolutional neural network is divided into two stages: Stage 1: The stage where data propagates from lower levels to higher levels, i.e., the forward propagation stage; Stage 2: The stage where, when the result obtained from forward propagation does not match the expectation, the error is propagated from higher levels to lower levels for training, i.e., the backpropagation stage; The specific training process is as follows:

[0194] S1: The network initializes its weights;

[0195] S2: The input data is propagated forward through the convolutional layer, downsampling layer and fully connected layer to obtain the output value;

[0196] S3: Calculate the error between the network's output value and the target value, i.e., the bias e;

[0197] S4: When the error is greater than the expected value, the error is fed back into the network, and the errors of the fully connected layer, the downsampling layer and the convolutional layer are calculated in turn. The error of each layer can be understood as how much of the total error of the network should be borne by the network. When the error is equal to or less than the expected value, training ends and the weights and thresholds are fixed.

[0198] S5: Update the weights based on the obtained error and proceed to S3;

[0199] After being trained by a convolutional neural network, it can accurately and automatically identify the behavior of dairy cows, and then automatically judge the health status of dairy cows based on their behavior.

[0200] Example 2

[0201] This embodiment is a further optimization based on the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the present invention, the following setup is specifically adopted: In images of many instances, detected landmarks are divided into different sets associated with each individual. For the part grouping problem, it is necessary not only to obtain the surface location but also more information. This application uses the main parts of the bottom-up cow skeleton keypoint detection algorithm, namely keypoint clustering and keypoint detection. The keypoint detection method is similar to that for a single cow, the difference being that keypoint detection here requires detecting all keypoints of all categories in the image, and then clustering these keypoints to connect different keypoints of different cows, thereby generating different individuals. This paper uses PAF (PartAffinityFields) to combine scattered keypoints in multi-person images.

[0202] Algorithm flow:

[0203] First, heatmaps and PAFs of keypoints are predicted for the input image. Then, keypoints and limbs are correlated using the most binary matching method, ultimately yielding all poses of all cows in the pasture shown in the image. Figure 5 As shown.

[0204] The image is first analyzed and processed by a convolutional network (initialized and fine-tuned from the first 10 layers of VGG-19) to generate a set of feature maps F, which are then input into the confidence map S of the current stage in the first stage of each branch. t and vector field L t It is determined by both the input F and the output of the previous stage, that is:

[0205]

[0206]

[0207] Where ρ t and f t It is a CNN used for staged inference;

[0208] To guide the network iteratively in predicting the confidence maps of body parts in the first branch and PAF in the second branch, two loss functions are applied at the end of each stage, with one loss function applied at each branch. L2 loss is used between the estimated predictions and GTmap and PAF, and the loss functions are spatially weighted to address the issue that some datasets cannot completely label all cows. The loss functions at the two branches in stage t are as follows:

[0209]

[0210]

[0211] W represents a binary mask. W = 0 for non-target locations, and W(p) is zero for unmarked locations p. Otherwise, W(p) with an asterisk represents the ground truth, and without an asterisk represents the predicted value. L2 loss is used to calculate the loss. The confidence plot is calculated by taking the maximum value instead of the average value.

[0212]

[0213] f is the overall objective function, where the ground truth of key points and limbs is set using a Gaussian function.

[0214] PAF ground truth design:

[0215] Simultaneously, the position and orientation information of the entire limb support area are preserved. Each PAF is a two-dimensional vector field for each limb; for each pixel belonging to a specific limb region, a two-dimensional vector encodes the direction from one part of the limb to another. Each type of limb has a corresponding PAF connecting its two associated body parts, such as... Figure 6 As shown.

[0216] Where n represents a keypoint, k represents the k-th key, x represents the position of the keypoint (part affinity vector field), and L... m，k (p) is calculated as follows:

[0217]

[0218] The above formula can be understood as follows: For point P, if point P lies on the limb c of the kth cow, then the vector at point P is V; otherwise, it is 0. That is, the unit vector between two key points.

[0219] Formula for determining whether point P is on limb c:

[0220] 0≤v·(px n1,k )≤l c,k and|v ⊥ ·(px n,k )|≤σ l

[0221] Length between two key points: l c,k =||x n2,k -x n1,k || 2 The width of the limb is σ l

[0222] When multiple cows have overlapping limbs, the formula is as follows:

[0223]

[0224] The number of cows overlapping at position P is k, q m (p) is the number of vectors at position P that are not zero.

[0225] During testing, for two candidate locations n1 and n2, the correlation confidence score is calculated as follows. In practice, it is approximated by integrating at uniform sampling intervals. For the two candidate locations n1 and n2, samples are taken from PAFs, L m It represents the confidence level of the relationship between line segments.

[0226] The PAF (Personal Affinity Field) is formed by summing and averaging the encoded vectors of all pixels in the entire region. During training, if point P is within the region, the ground truth of point P's vector is the normalized vector v; otherwise, it is a zero vector. Since there are many potential body parts in multi-person images, the final judgment is made by summing and averaging the vector fields corresponding to all body parts. Additionally, the "affinity" between keypoints needs to be determined for final combination and representation, which is then used to interpolate the positions of the two body parts.

[0227]

[0228]

[0229] p(u) represents the insertion position between two cow body parts n1 and n2:

[0230] Predicting multiple dairy cows based on PAFs:

[0231] Let a and b represent the number of key points, using a variable between 0 and 1. This indicates whether two key points are connected. All possible z values ​​form a set, which is a graph where each edge is used as a weight. The goal is to maximize the sum of all edges using the Hungarian algorithm.

[0232]

[0233]

[0234]

[0235] Where E is the "relevance confidence" obtained by performing line integrals during testing (which can be understood as "whether to connect"). d E refers to all the "relevance confidence scores" in the graph for the limbs. abLet z be the "relevance confidence" between a and b, and z be the number of points between the two key points. Let a represent the a-th point in the n1-th joint set, and b represent the b-th point in the n2-th joint set.

[0236] This paper employs the Hungarian algorithm to obtain the best match, selecting the minimum number of edges to obtain the spanning tree skeleton of the cow pose instead of using the complete graph. Furthermore, the matching problem is further decomposed into a set of binary matching subproblems, and the matches in adjacent tree nodes are determined independently.

[0237] Ultimately, the optimization formula can be summarized as the sum of E for each limb c:

[0238]

[0239] Example 3

[0240] This embodiment illustrates the following through examples: This study uses surveillance videos as raw data, with each video approximately 30 minutes long; the video segments in this embodiment were selected from 300 original videos, excluding those without targets; the videos in five scenarios were manually annotated with 3000 frames each, and divided into training and validation sets according to a 9:1 ratio of video length; to improve the robustness of the pose estimation algorithm, a data augmentation algorithm was used to enrich the training set, with each video frame randomly rotated by an angle ranging from -15° to 15°;

[0241] Based on the occlusion of parts of the cow in different postures, the occluded parts and the visible parts are marked separately, and the key points of the cow are marked manually.

[0242] Images are captured by the camera and sent to the terminal for processing. First, semantic segmentation and key point detection are performed. Then, the data is fed into the Unet network for feature extraction, followed by landmark localization, component grouping and tracking. Finally, the data is fed into TensorFlow for behavior recognition and classification. Finally, the health status of the cows is determined by judging the proportion of time each behavior takes up in a day.

[0243] This application enables dynamic video monitoring of dairy cows. By skeletal mapping of the cows, all photos with skeletal mapping are saved and transmitted back to the system. Convolutional neural networks accurately identify lying-down behavior, and the system filters and sorts these photos in the background to determine the duration of lying down, thereby monitoring the cows' health. Employing skeletal mapping technology, deep learning, and image processing technology, this approach has distinct regional characteristics and broad application prospects, and is of great significance for the automation and intelligentization of large-scale dairy farms.

[0244] The product developed in this application is eventually packaged. Simply place a camera in the dairy farm, connect it to a terminal, start the terminal program, enter the packaging system, turn on the camera, and the terminal can automatically identify the behavior of the dairy cows and judge the duration. If an abnormality is detected, a warning will be issued, thereby achieving low-cost, fully automated healthy dairy farming.

[0245] The above description is merely a preferred embodiment of the present invention and is illustrative rather than restrictive. Those skilled in the art will understand that many changes, modifications, and even equivalents can be made within the spirit and scope defined by the claims of the present invention, all of which will fall within the protection scope of the present invention.

Claims

1. A method for judging the health status of dairy cows based on video skeleton behavior recognition, characterized in that: Includes the following steps: Step 1: The camera captures images, which are then sent to the terminal for processing. Key points of the cow's skeleton are manually labeled and detected. Step 2: The bottom-up pose estimation method is used to obtain the cow's pose. The bottom-up pose estimation method includes feature extraction, landmark localization, part grouping, and tracking. Feature extraction: The image is processed using a parallel algorithm of bilinear interpolation, batch data is normalized using BatchNormalization, and self-attention gating is used to highlight the salient features of the region of interest. The U-net feature extraction network model is improved, and the features of CMs and PAF are output. Landmark location: A confidence map is generated from the two-dimensional skeletal key points of the cow in the video. The confidence maps are then aggregated using operators to generate the true confidence map of the network prediction, thereby locating the skeleton nodes. Grouping by body part: After detecting numerous skeletal key points, clustering is performed on these key points to connect multiple skeletal key points from different cows, generating multiple individual cows and thus obtaining multiple cow poses. track: In each frame of the image, the similarity between cow proposals, key points, and adjacent frames is detected, and the entire video is tracked to construct a pose flow using spatiotemporal information. Step 3: After obtaining the results using the Bottom-up pose estimation method, the improved U-net network is obtained, and the CNN-GRU network is trained for behavior classification. Step 4: Train and test the network performance using the collected cow images in the database to determine the cow's behavior based on the skeletal key points in the video.

2. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, The bilinear interpolation algorithm in step two: Given four function values ​​of a function on a two-dimensional plane: Q11=(x1,y1), Q12=(x1,y2), Q21=(x2,y1), and Q22=(x2,y2); first, perform linear interpolation in the y-direction to obtain R1 and R2, and then perform two linear interpolations in the x-direction to obtain point P, thus obtaining the desired two-dimensional coordinates of f(x,y); Linear interpolation in the y-direction: ； Single linear interpolation in the x-direction: ； ； Substituting the result of the single linear interpolation in the x-direction into the linear interpolation in the y-direction, we obtain: ； Image information is stored in a two-dimensional array of nW*nH. The image is processed by n processes. One main process is responsible for initialization, sending data before processing and receiving data after processing. The remaining n-1 slave processes are responsible for interpolating the image data and sending the interpolated data to the main process. That is, multiple processes process the image simultaneously. The parallel algorithm of bilinear interpolation is used to process the image. For the boundaries of the image, each process processes an extra row of data at the boundary, so that the data processed by multiple processes can completely cover the entire image.

3. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, Step two, BatchNormalization, standardizes each dimension: ； ； ； ； ； Its operation can be divided into two steps: First, standardize the m x values ​​to obtain a distribution with a mean unit variance of zero; Then on Perform scaleandshift to scale and shift to a new distribution y with a new mean β and variance γ.

4. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, To handle different image sizes and ensure that the feature map after convolution has the same size as the original image, padding is added, i.e., Padding = SAME. The formulas for calculating the length and width of the output feature map are as follows: ； Where Wout is the size of the output feature map, Win is the size of the input feature map, F is the kernel size, stride is the convolution stride, and Padding is the number of padding loops for the feature map.

5. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, In step two, self-attention gating is used for dense label prediction in standard CNN-based image analysis models. Attention coefficient ( ∈(0,1]) highlights significant image regions and suppresses task-irrelevant feature responses. Multiplication with feature maps is element-wise; the formula for additive attention is: ； ； in, It's a ReLU function. It is the Sigmoid function. , , Both are convolution operations. , These are all bias terms corresponding to the convolution.

6. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, The specific steps for landmark localization in step two are as follows: First, generate a separate confidence map for each cow. , It is the ground truth position of body part j of k cows in the image; position The value is defined as: ； Among them, σ controls the diffusion of the peak value, and the aggregation of individual confidence graphs by the maximum operator generates the true confidence graph of the network prediction; ； The maximum confidence plot is used instead of the average value, so the accuracy of the nearby peaks remains clear.

7. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, In step two, the part grouping first predicts the heat map and PAF of key points in the input image, and then associates them according to the most binary matching of key points and limbs, finally obtaining the posture of all cows in the image. Algorithm flow: First, a convolutional network analyzes and processes the image, generating a set of feature maps F, which are then input into the first stage of each branch, where the confidence map S is located. t and vector field L t It is determined by both the input F and the output of the previous stage, that is: ； ； in It is a CNN used for staged inference; To guide the network iteratively in predicting the confidence maps of body parts in the first branch and PAF in the second branch, two loss functions are applied at the end of each stage, with one loss function applied at each branch. L2 loss is used between the estimated predictions and the ground truth map and PAF, and the loss functions are spatially weighted to address the issue that some datasets cannot completely label all cows. The loss functions at the two branches in stage t are as follows: ； ； W represents a binary mask; W=0 at non-target locations, and position p is unlabeled. Zero, An asterisk indicates ground truth, while no asterisk indicates predicted value; L2 loss is used to calculate the loss; the confidence plot is calculated by finding the maximum value. ； It is the overall objective function, where the ground truth of key points and limbs is set using a Gaussian function; PAF ground truth design: Each PAF is a two-dimensional vector field for each limb. A two-dimensional vector code encodes the direction from one part of the limb to another part. Each type of limb has a corresponding PAF that connects its two related body parts. Here, n represents a keypoint, k represents the k-th cow, and x represents the position of the keypoint (part affinity vector field). The calculation method is as follows: ； Where P is any point on a limb, for example, if point P is on limb c of the kth cow, then point P is represented by vector V; otherwise, it is 0. , that is, the unit vector between two key points; Formula for determining whether point P is on limb c: ； Length between two key points: The width of the limbs is ; When multiple cows have overlapping limbs, the formula is as follows: ； The number of cows overlapping at point P is k. It is the number of vectors with a non-zero value at position P; The PAF (Place Affinity Field) is formed by summing and averaging the encoded vectors of all pixels in the entire region. During training, if point P is within the region, the ground truth of point P's vector is the normalized vector. If it's not there, it's a zero vector; since there are many potential body parts in a multi-person image, the final judgment is made by summing and averaging the vector fields corresponding to all body parts; additionally, it's necessary to determine the "affinity" between key points for final combination, which is then used to represent the positions of the two parts. ； ； Where n1 and n2 are two candidate positions, and p(u) is the position to be inserted between two cow body parts n1 and n2; Predicting multiple dairy cows based on PAFs: Let a and b represent the number of key points, using a variable between 0 and 1. To indicate whether two key points are connected, all possible z forms a set; ； ； ； Where E is the "relevance confidence" obtained by performing line integral during the test. For the limbs, the graph represents all "relevance confidence scores". Let z be the "relevance confidence" between a and b, z be whether there is an edge between the two key points, a be the a-th point in the n1-th joint set, and b be the b-th point in the n2-th joint set. Instead of using the complete graph, we choose the minimum number of edges to obtain the spanning tree skeleton of the cow pose. Secondly, we further decompose the matching problem into a set of binary matching subproblems and independently determine the matches in adjacent tree nodes. Finally, the optimization formula can be summarized as the sum of E on each limb c: 。 8. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, The specific steps of tracking in step two are as follows: target detection is performed on each frame of the image, and the same targets in each frame of the image are associated to form a PF; use This represents the candidate matching set for the i-th pose in the j-th frame. ,Right now: ； To optimize pose selection, the following objective function is maximized: ； ； ； ； Then, the optimal PF for frames t to t+T is obtained by solving an optimization problem; the PF is used as the basic unit for NMS processing to determine the distance between two pose flows indicating the same cow. ； After calculating the scores between PoseFlows, they are integrated. When performing NMS using traditional methods, the coordinates and scores of keypoints can be calculated using the following formula: 。 9. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, Under the learned model, the category Ck to which the input behavior matrix X belongs can be calculated using the following formula: ； ； ； Where C is the number of categories, T is the length of the test sequence, O is the output vector, and h t The output of the model is the class with the highest probability, and the class with the highest probability is used as the final classification result.

10. The method for recognizing and judging the health status of dairy cows based on video skeletons according to claim 1, characterized in that, The training process of a convolutional neural network is divided into two stages: Stage 1: The stage where data propagates from lower levels to higher levels, i.e., the forward propagation stage; Stage 2: The stage where, when the result obtained from forward propagation does not match the expectation, the error is propagated from higher levels to lower levels for training, i.e., the back propagation stage; the specific training process is as follows: S1: The network initializes its weights; S2: The input data is propagated forward through the convolutional layer, downsampling layer and fully connected layer to obtain the output value; S3: Calculate the error between the network's output value and the target value; S4: When the error is greater than the expected value, the error is fed back into the network, and the errors of the fully connected layer, downsampling layer and convolutional layer are calculated in turn; when the error is equal to or less than the expected value, training ends and the weights and thresholds are fixed. S5: Update the weights based on the obtained error and proceed to S3.

Images