Dairy cow limp detection method based on depth image processing

Abstract

The invention discloses a dairy cow limp detection method based on depth image processing, and relates to the field of intelligent detection of livestock and poultry breeding. The method comprises the following steps: acquiring a depth image in a cow walking process, and processing the depth image to obtain a cow body terrain clearance image with a uniform size; calculating 3D point cloud coordinates corresponding to the cow body terrain clearance image by using the cow body terrain clearance image after pixel completion and combining a built-in parameter matrix of a depth camera; synchronous positioning and tracking of multiple cows in a depth image are realized by combining instance segmentation and a multi-target tracking algorithm, and classification of a cow body terrain clearance image and a cow back geometric feature image is realized by using a deep learning classifier; and realizing limp detection in the walking process of the dairy cow by using score weight fusion and time sequence feature fusion. The method has the advantages of high working efficiency, high detection precision and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection in livestock and poultry farming, and in particular to a method for detecting lameness in dairy cows based on depth image processing. Background Technology

[0002] Lameness in dairy cows is primarily caused by hoof diseases such as interdigital dermatitis, plantar ulcers, and laminitis, leading to limb and hoof pain, reduced weight-bearing capacity, and abnormal gait. Lameness not only affects cow welfare, reduces milk production and reproductive performance, and increases the risk of premature culling, but also causes significant economic losses for dairy farmers. If lameness is not treated promptly, it will gradually develop into chronic lameness. Treatment at this stage not only increases costs but also significantly reduces the cure rate and may still result in premature culling. Therefore, timely and efficient detection of lameness in dairy cows is of significant practical importance for promoting the healthy and sustainable development of the dairy industry.

[0003] Traditional methods for detecting lameness in dairy cows primarily rely on manual observation. This involves assessing the cow's motor abilities and detecting lameness by observing abnormalities in its gait, such as arched back, head nodding, and other unusual movements. However, this method suffers from low efficiency, high cost, and strong subjectivity, making it unsuitable for large-scale dairy farms. Contact sensors (e.g., accelerometers, gyroscopes, and force platforms) measure motion parameters during walking by directly contacting the cow's body, and then use data analysis techniques to infer lameness. However, direct contact with these sensors can easily cause stress in the cow and is susceptible to signal loss due to environmental factors, reducing the accuracy of lameness detection. Computer vision technology, on the other hand, can acquire video data of cow walking without contact or stress, and automatically detect lameness using feature extraction and classification methods. This technology has become an important tool in dairy cow lameness detection research.

[0004] The computer vision-based method for detecting lameness in dairy cows mainly consists of three steps: key body part detection, lameness feature extraction, and lameness classification. Key body part detection effectively removes background interference from the image and is the foundation for accurately extracting lameness features. Key body part detection methods are mainly divided into traditional image processing methods and deep learning methods. Traditional image processing methods mainly include frame differencing and background subtraction; deep learning methods mainly include object detection, image segmentation, and keypoint detection. Accurate lameness feature extraction is crucial for detecting lameness in dairy cows. Lameness feature extraction methods mainly refer to the external behavioral characteristics of dairy cows, such as arched back, head nodding, poor gait following, and symmetry. They utilize manually designed feature extraction techniques or deep learning automatic feature extraction techniques to obtain lameness-related features. Finally, traditional machine learning algorithms or deep learning algorithms are used to classify the lameness features and detect lameness in dairy cows. In summary, the computer vision-based method for detecting lameness in dairy cows mainly relies on the above detection steps to achieve accurate and rapid detection of lameness.

[0005] Some researchers have deployed visible light cameras to the side of cows as they walk, collecting side-view video data and analyzing abnormalities in back arching, head swaying, and gait. However, this side-view visible light camera-based method for detecting lameness in cows can only handle scenarios with a single cow crossing a path. If multiple cows are crossing in parallel or occluding each other, the method suffers from significant blind spots, leading to false positives and false negatives. A top-view depth camera-based method for detecting lameness in cows can simultaneously locate and track multiple cows, improving detection efficiency and broadening its applicability. Currently, further research is needed on depth image processing and back feature extraction for lameness detection to more comprehensively characterize the degree of back arching and improve accuracy. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to provide a high-efficiency and high-accuracy method for detecting lameness in dairy cows based on depth image processing.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for detecting lameness in dairy cows based on depth image processing, comprising the following steps: Depth images of cows walking were collected and processed to obtain uniformly sized images of the cows' body height above the ground. Using the pixel-completed image of the cow's body height above the ground, combined with the built-in parameter matrix of the depth camera, the 3D point cloud coordinates corresponding to the image of the cow's body height above the ground are calculated. This paper combines instance segmentation and multi-object tracking algorithms to achieve synchronous localization and tracking of multiple cows in depth images. For individual cows in depth images, a deep learning classifier is used to classify images of the cow's body height above the ground and images of the geometric features of the cow's back. By using score weight fusion and temporal feature fusion, lameness detection can be achieved during the walking process of dairy cows.

[0008] The beneficial effects of adopting the above technical solution are as follows: The method utilizes depth image processing to deeply mine the features of the cow's back, refine the back structure and contour features related to lameness, improve the accurate description of the surface structure of the cow's back, significantly improve the accuracy of lameness detection, and optimize the lameness detection process: the pixel values ​​in the depth image are converted into the height value of the cow from the ground and the geometric feature value of the cow's back, reducing the influence of random noise on the depth image, improving the fine-grained expression and semantic representation ability of the data, and improving the accuracy and efficiency of lameness detection. Attached Figure Description

[0009] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0010] Figure 1 This is a general flowchart of the method described in the embodiments of the present invention; Figure 2a It is the visible light image in the method described in the embodiments of the present invention; Figure 2b It is the depth image in the method described in the embodiments of the present invention; Figure 2c It is a pseudo-color image in the method described in the embodiments of the present invention; Figure 3 This is a diagram illustrating the cow segmentation and image processing process in the method described in this embodiment of the invention. Figure 4 This is a diagram illustrating the process of dairy cow image processing and lameness classification in the method described in this embodiment of the invention; Figure 5 This is a flowchart of the process for obtaining geometric feature images of the cow's back in the method described in this embodiment of the invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0012] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0013] Example 1 like Figure 1 As shown in the figure, this invention discloses a method for detecting lameness in dairy cows based on depth image processing: Depth images of cows walking were collected and processed to obtain uniformly sized images of the cows' body height above the ground. Using the pixel-completed image of the cow's body height above the ground, combined with the built-in parameter matrix of the depth camera, the 3D point cloud coordinates corresponding to the image of the cow's body height above the ground are calculated. This paper combines instance segmentation and multi-object tracking algorithms to achieve synchronous localization and tracking of multiple cows in depth images. For individual cows in depth images, a deep learning classifier is used to classify images of the cow's body height above the ground and images of the geometric features of the cow's back. By using score weight fusion and temporal feature fusion, lameness detection can be achieved during the walking process of dairy cows.

[0014] In general, this application places a Gemini 2L Orbbec depth camera at the top center of the cow walking path to collect RGB-D depth video data during the cow's walking process. An instance segmentation algorithm is used to segment the cow's body parts and remove background interference from the images. Based on the depth camera's placement height, a cow body height threshold is set to remove abnormal data values ​​from the depth image. A method for converting cow depth image data is studied to convert the distance between the cow's body and the depth camera into the cow's body height above the ground. Pixel value padding is used to obtain a uniformly sized image showing the cow's body height above the ground.

[0015] By combining images of the cow's body height above the ground with the built-in parameter matrix of the depth camera, 3D point cloud data of the cow's body is obtained through numerical calculation. The least squares method is used to obtain the local fitting plane of each point cloud block on the cow's body, and the surface normal vector of each point on the cow's body is obtained by taking the vertical direction of the plane. By combining the surface normal vectors of each point on the cow's body, the direction of gravity field of the entire scene is determined by statistical analysis techniques. The angle between the surface normal vector of each point on the cow's body and the direction of gravity is calculated to obtain the geometric feature image of the cow's back.

[0016] Two deep learning models are constructed to extract cow back feature vectors from images of cow body height above the ground and images of cow back geometric features, respectively. By using a weighted score fusion method, fine-grained structural and semantic features of the cow back are fully obtained, improving the accuracy of cow lameness detection. By combining instance segmentation and multi-object tracking methods, the localization, tracking, and lameness detection of multiple cows are realized, improving the efficiency and accuracy of cow lameness detection.

[0017] The "cow's body height above ground image" and "cow's back geometric feature image" proposed in this invention constitute two complementary and synergistic dimensions for characterizing the lameness state of cows. In the cow's body height above ground image, by converting depth values ​​to height above ground, the semantic information of the features is enhanced, allowing the absolute height difference between the cow's "high point on the back" and "low point on the waist" to be directly presented, reducing data interference caused by camera distance in the original depth image. In the cow's back geometric feature image, by calculating the angle between the surface normal vector and the direction of gravity, the fine-grained geometric structure of the back is captured, such as local curvature changes and asymmetries. These are microscopic manifestations of arched back behavior, which may be smoothed out in a simple height above ground image. The features of these two dimensions describe the same physiological phenomenon (arched back) from the perspectives of macroscopic contour and microscopic structure, respectively. Their fusion effectively forms a more comprehensive and robust feature representation, which cannot be achieved by single-dimensional image processing.

[0018] The above steps will be explained in detail below with specific methods: Image of the height of a cow's body off the ground: First, the instance segmentation algorithm is used to segment the body parts of the cow and remove background interference from the image. The specific steps are as follows: (1) Convert the depth image of the cow walking obtained by the depth camera into a pseudo-color image, wherein the visible light image, depth image and pseudo-color image of the cow walking process are as follows: Figures 2a-2c As shown.

[0019] (2) Randomly select 1000 pseudo-color images containing cows and use labelme data annotation software to annotate the body outline of the cows in the images.

[0020] (3) Randomly select 100 pseudo-color images without cows as background datasets. No annotation is required. These images are directly added to the labeled pseudo-color images of cows to improve the instance segmentation model’s ability to resist noise in depth images.

[0021] (4) The obtained pseudo-color images and labels are randomly divided into a training set and a test set for cow instance segmentation in an 8:2 ratio.

[0022] (5) Construct the YOLOv8n-Seg instance segmentation model, train and test it on the cow instance segmentation dataset, and obtain the optimal model weight file.

[0023] Secondly, based on the instance segmentation results of the cows, the target detection bounding boxes for the cows are obtained. The depth data within these boxes is then optimized by removing irrelevant background data and outlier values. The pixel values ​​in the depth image represent the distance between the depth camera and the target object, in millimeters. Assuming the depth camera is placed at a height... h The height of the cow's body is x min ~ x max Next, using the distance relationship between the ground, the cows, and the depth camera as a reference, irrelevant and outlier data values ​​are removed from the depth image. The specific steps are as follows: (1) Based on the original image size and the size of the cow segmentation mask image output by the instance segmentation model, restore the size of the cow segmentation mask image to the original image size.

[0024] (2) Multiply the processed cow segmentation mask image with the original depth image by corresponding pixels to obtain the segmented cow depth image. At this time, the pixel values ​​of the cow target remain the original depth values, and the pixel values ​​of the background environment are set to 0.

[0025] (3) Set the threshold to ( h - x min The pixel values ​​within the detection box that are greater than the threshold are set to 0, which represents the background environment, while the values ​​of other pixels remain unchanged.

[0026] (4) Based on the output of the instance segmentation model, the depth image is cropped according to the location of the target detection box belonging to the cow in the depth image to obtain the depth image within the detection box.

[0027] The operation process is as follows: Figure 3 As shown, for ease of representation, the depth image is subsequently converted into a pseudo-color image. Figure a represents the RGB image captured by the depth camera; Figure b represents the depth image captured by the depth camera; Figure c represents the cow segmentation mask image output by the instance segmentation model; Figure d represents the cow segmentation depth image obtained by multiplying the segmentation mask image with the original depth image; Figure e represents the cow depth image after thresholding to remove noise; and Figure f represents the cow depth image cropped from the reference object detection box.

[0028] Finally, an image showing the cow's height above the ground was obtained. The cow's depth image data was converted into a value representing the cow's height above the ground. Then, the cropped depth image was augmented using zero-padding to obtain the final image.m × m The depth map, in which m This indicates the length of the depth image after cropping. The specific steps are as follows: (1) Further optimize the pixel values ​​in the depth image, filter all pixel values ​​in the depth image to ensure that pixel values ​​within the range of the cow's body height are retained, and set the minimum and maximum limits of pixel values ​​outside the range of the cow's body height according to the numerical distribution to facilitate the difference conversion of pixel values. Specifically, all pixel values ​​greater than ( h - x min The pixel values ​​of ) are all set to ( h - x min ); all smaller than ( h - x max The pixel values ​​of ) are all set to ( h - x max ).

[0029] (2) Reference depth camera placement height h ,Will h By subtracting all pixel values ​​from the depth image, the distance between the depth camera and the cow's body can be converted into the cow's height above the ground.

[0030] (3) Because the height of a dairy cow will not exceed x max The previous operation will take you above the ground for more than x max All pixel values ​​are set to x max Therefore, the pixel value here is equal to x max All pixel values ​​are set to 0, representing background pixels.

[0031] (4) By padding with pixel values, the cropped cow depth image is converted into a depth image with the same aspect ratio and a size of [missing value]. m × m ,in m This represents the long side of the cropped depth image of the cow.

[0032] (5) Collect images of the height of the cow's body off the ground, create category labels for the images of the height of the cow's body off the ground, including normal cows and lame cows, and construct the first lame cow classification dataset.

[0033] (6) Input the image of the cow's body height off the ground into the MobileNetv2 classification model for model training to obtain the cow lameness classification weight file.

[0034] The operation process is as follows: Figure 4 As shown, Figure a represents the cropped depth image; Figure b represents the cow depth image with minimum and maximum limits set according to the cow's body height range; Figure c represents the cow depth image obtained by subtracting the pixel values ​​in the depth image from the camera height; Figure d represents the image of the cow's body height above the ground; and Figure e represents the image with dimensions of [size missing] after pixel padding. m × m Images showing the height of a cow's body off the ground. From Figure 4 As can be seen, with the removal of noise and the conversion of the depth image into an image showing the height of the cow's body off the ground, the cow's back contour information becomes more obvious, and the cow's arched back feature is easier to extract.

[0035] Image of the geometric features of a cow's back: First, using the pixel-completed image of the cow's body height above the ground, combined with the built-in parameter matrix of the depth camera, the 3D point cloud coordinates corresponding to the image of the cow's body height above the ground are calculated.

[0036] The image of a cow's body height off the ground is a two-dimensional matrix, where each pixel... High ( u , v This represents the height of the cow's body above the ground. The depth camera's intrinsic parameters include... f _ x , f _ y , c _ x and c _ y ,in f _ x and f _ y This indicates the focal length of the depth camera. c _ x and c _ y The principal point is represented by the following formula, which is used to calculate the height of the cow off the ground for each pixel in the image. u , v ) corresponding 3D points ( X , Y , Z The coordinates of the camera in the camera coordinate system.

[0037] (1) (2) (3) Secondly, a point cloud is formed by combining the 3D coordinates of the cow's body with several surrounding pixels. The best-fit plane corresponding to the point cloud is obtained using the least squares method, leading to the unit normal vector corresponding to the 3D coordinates, which is then normalized. By statistically analyzing the unit normal vector corresponding to each 3D coordinate point of the cow's body, the normal vector field corresponding to the cow's body can be obtained.

[0038] The plane equation fitted to the point cloud block is shown in Equation (4): (4) Where (a, b, c) are the parameters of the fitted plane.

[0039] The least squares optimization objective is shown in formula (5): (5) in,( x i , y i , z i Given a set of 3D point cloud coordinates, i This indicates the index of the 3D coordinate point in the point cloud block. i The range of values ​​for is 1≤ i ≤ n , n This indicates the number of 3D coordinate points in the point cloud block. M This represents the sum of distances between each 3D coordinate point in the point cloud block and the fitted plane. The goal is to find the sum of distances when... M When the value is at its minimum, the parameters (a, b, c) of the fitted plane are obtained, and then the surface normal vector corresponding to the fitted plane is obtained.

[0040] Then, the dominant direction is statistically analyzed from the normal vectors of all pixels to determine the gravity direction. Specifically, this involves finding a direction that is approximately perpendicular to a large number of surface normal vectors; this direction typically corresponds to the ground normal direction in the scene, i.e., the gravity direction. Through iterative optimization algorithms, the system can robustly estimate the gravity field direction of the entire scene.

[0041] Finally, for each pixel on the cow's body, the angle between its surface normal vector and the estimated direction of gravity is calculated. This angle can be calculated using the vector dot product formula, where the dot product of two unit vectors equals the cosine of the angle between them. The calculated angle ranges from 0° to 180°. To encode this into an image (pixel value range of 0-255), the angle values ​​undergo a linear transformation and offset to fit the numerical range of an 8-bit image. This image represents the geometric features of the cow's back, as shown below. Figure 5As shown, part a represents the image of the cow's body height above the ground; part b represents the 3D point cloud image; and part c represents the geometric feature image of the cow's back.

[0042] Geometric feature images of cow backs utilize local angular information of the cow's body to describe back geometric features highly relevant to lameness, such as back curvature angle, lumbar dimple depth, degree of spinal protrusion, and back asymmetry. By collecting these back geometric feature images, category labels were created for each cow, including normal cows and lame cows, constructing a second lameness classification dataset. The back geometric feature images were then input into a MobileNetv2 classification model for training, resulting in a lameness classification weight file.

[0043] Lameness detection in dairy cows: First, a simultaneous localization and tracking system for multiple cows in depth images is achieved by combining instance segmentation and multi-object tracking algorithms. Second, for individual cows in the depth images, a MobileNetV2 deep learning classifier is constructed to classify images of the cow's body height above the ground; simultaneously, a MobileNetV2 deep learning classifier is also constructed to classify images of the geometric features of the cow's back. Then, using score-weighted fusion and temporal feature fusion methods, lameness detection during cow walking is achieved. Finally, a cow lameness detection system is constructed using data storage, data processing, and result synchronization technologies to achieve simultaneous localization, tracking, and lameness detection for multiple cows in a pasture environment. The lameness detection results are uploaded to a cloud server, improving the accuracy and efficiency of cow lameness detection and enhancing the level of intelligent pasture management.

[0044] This invention acquires depth images of cows walking, segments the cow's body parts using an instance segmentation algorithm, and removes background interference from the image. Based on the output detection boxes of the instance segmentation model, the cow's depth image is cropped. A height threshold is set according to the depth camera's placement height to remove outlier values. A depth image value range is defined based on the cow's body height, mapping values ​​outside the range to within it. Combining the camera placement height and the cow's body height, a value conversion method is used to convert the depth image values ​​corresponding to the cow's body to its ground clearance value. Finally, pixel value completion technology is used to obtain a uniformly sized image showing the cow's ground clearance.

[0045] This invention combines images of the cow's body height above the ground with the built-in parameter matrix of a depth camera to obtain 3D point cloud data of the cow's body through numerical calculation. Using the 3D coordinates of the cow's body and other 3D coordinates in the surrounding area, a local plane is fitted using the least squares method, and the surface normal vectors of each point on the cow's body are obtained by taking the perpendicular direction of the plane. By combining the surface normal vectors of each point on the cow's body and using statistical analysis techniques, the direction of the gravity field of the entire scene is determined. The angle between the surface normal vectors of each point on the cow's body and the direction of gravity is calculated to obtain the geometric feature image of the cow's back, which represents the fine-grained structure of the cow's back. Finally, a deep learning model is constructed to extract the complete feature vector of the cow's back from the images of its body height above the ground and the geometric feature image of the cow's back, improving the accuracy of cow lameness detection.

[0046] This invention combines instance segmentation and multi-target tracking algorithms to achieve synchronous localization and tracking of multiple dairy cows in depth images; based on images of the height of the cow's body off the ground and geometric feature images of the cow's back, a lameness detection model is established to achieve continuous localization, tracking, and lameness detection of dairy cows in the pasture; utilizing data storage, data processing, and processing result synchronization technologies, a lameness detection system for dairy cows is constructed to improve the accuracy and efficiency of lameness detection and enhance the level of intelligent management in the pasture.

[0047] Example 2 A depth camera was placed at the top center of the cow walking path to collect depth video data of the cows walking naturally. A YOLOv8n-Seg instance segmentation model was constructed to segment the cow body parts and remove background interference. A cow body height threshold was set based on the depth camera placement height to remove abnormal data values ​​from the depth image. A depth image numerical conversion method was established to convert the depth image corresponding to the cow body into a cow body height image. Combining the cow body height image with the depth camera's built-in parameter matrix, 3D point cloud data of the cow body was obtained. The surface normal vectors of each element point on the cow body were obtained using the least squares method, and combined with the scene's gravity field direction, the geometric feature image of the cow's back was obtained. A deep learning model was constructed to classify the cow body height image and the cow back geometric feature image. Through score weight fusion and temporal feature fusion, the cow lameness detection results were obtained. The cow lameness detection results were saved to a local database and uploaded to a cloud server at midnight for farm staff to promptly understand the lameness situation of the cows.

[0048] The method described in this embodiment of the invention specifically includes the following steps: (1) Use Orbbec Gemini 2L depth camera to collect depth images of cows walking and convert the depth images into pseudo-color images.

[0049] (2) Randomly select 1000 pseudo-color images containing cows and use labelme data annotation software to annotate the body outline of the cows in the images.

[0050] (3) Randomly select 100 pseudo-color images without cows as background datasets. No annotation is required. These images are directly added to the labeled pseudo-color images of cows to improve the instance segmentation model’s ability to resist noise in depth images.

[0051] (4) The obtained pseudo-color images and labels are randomly divided into a training set and a test set for cow instance segmentation in an 8:2 ratio.

[0052] (5) Construct a YOLOv8n-Seg instance segmentation model, use the model weights pre-trained on the ImageNet dataset as the initial training weights of the model, train and test on the constructed cow instance segmentation dataset, and obtain the optimal model weight file.

[0053] (6) The ByteTrack multi-object tracking algorithm is used to associate high-scoring and low-scoring boxes in the detection box, reducing missed detections and improving tracking stability. In the process of using the ByteTrack multi-tracking algorithm, no additional training is required. The output results of the YOLOv8n-Seg instance segmentation model can be directly reused to achieve the tracking and localization of cows between different video frames.

[0054] (7) Based on the output of the instance segmentation model, the original cow segmentation mask image is obtained.

[0055] (8) Based on the original image size and the size of the cow segmentation mask image, the size of the cow segmentation mask image is transformed to the size of the original image size through size transformation.

[0056] (9) Multiply the processed cow segmentation mask image with the original depth image by corresponding pixels to obtain the segmented cow depth image. At this time, the pixel values ​​of the cow target remain the original depth values, and the pixel values ​​of the background environment are set to 0.

[0057] (10) Assume that the depth camera is placed at a height of 3400mm above the ground, and the height of the cows ranges from 600mm to 1800mm. Set the height threshold to (3400-600=2800)mm, set the pixel values ​​in the detection box that are greater than the threshold to 0, which is the background environment, and keep the other pixel values ​​unchanged.

[0058] (11) Based on the output of the instance segmentation model, the depth image is cropped according to the location of the target detection box belonging to the cow in the depth image to obtain the depth image within the detection box.

[0059] (12) Further optimize the pixel values ​​in the depth image. Based on the height range of the cow, ensure that pixel values ​​within the height range are retained. Set the minimum and maximum limits of pixel values ​​outside the height range according to their numerical distribution to facilitate the difference conversion of pixel values. Specifically, set all pixel values ​​greater than (3400-600=2800) mm to 2800 mm; set all pixel values ​​less than (3400-1800=1600) mm to 1600 mm.

[0060] (13) With reference to the depth camera placement height of 3400mm, subtract all pixel values ​​in the depth image from 3400mm to convert the distance value between the depth camera and the cow's body into the height value of the cow's body from the ground.

[0061] (14) Since the height of a cow will not exceed 1800mm, the previous operation set all pixel values ​​that are more than 1800mm above the ground to 1800mm. Therefore, here we set all pixel values ​​that are equal to 1800mm to 0, representing background pixels.

[0062] (15) By zero-padding the pixel values, the cropped image of the cow's body height above the ground is transformed into an image of the cow's body height above the ground with the same aspect ratio and a size of m×m, where m is the long side of the cropped cow depth image.

[0063] (16) Collect images of the height of the cow's body off the ground, create classification labels for the images of the height of the cow's body off the ground, and construct a classification dataset for cow lameness.

[0064] (17) Construct a MobileNetv2 cow lameness classification model. Input the image dataset of cow body height off the ground into the lameness classification model for model training to obtain the first cow lameness classification weight file.

[0065] (18) Combine the image of the cow's body height above the ground with the intrinsic parameter matrix of the depth camera to obtain the 3D coordinate points of the cow's body point cloud.

[0066] (19) Take the 3D coordinate points of the cow's body and the 7×7 3D coordinate points around it to form a local point cloud block. Use the least squares method to fit the local plane and obtain the surface normal vector of the fitted plane.

[0067] (20) Take the 3D coordinates of the cow's body and 21×21 3D coordinates around it to form a local large point cloud. Use the least squares method to fit the local plane and obtain the surface normal vector of the fitted plane. From the normal vectors of all pixels, find the direction that is approximately perpendicular to a large number of surface normal vectors and take it as the gravity direction of the scene.

[0068] (21) By calculating the angle between the surface normal vector and the direction of gravity, and mapping it to a range of 0-255, the geometric feature image of the cow's back is obtained.

[0069] (22) Construct a MobileNetv2 cow lameness classification model. Input the dataset of cow back geometric feature images into the lameness classification model for model training to obtain a second cow lameness classification weight file.

[0070] (23) The output results of the two models are fused by score weight and all video frames during the cow walking process are combined to obtain the final cow lameness detection result.

[0071] (24) At 11:30 p.m., save the results of the day’s lameness test of dairy cows to the local database.

[0072] (25) At 12:00 a.m., the lameness detection results of dairy cows saved in the local database are uploaded to the cloud server to ensure that the lameness detection results of dairy cows can be obtained both locally and in the cloud, so that farm managers can promptly detect and treat lameness in dairy cows and improve the health management level of dairy cow limb and hoof diseases.

[0073] This invention provides a method and system for detecting lameness in dairy cows based on depth image processing. It employs an Orbbec Gemini 2L depth camera to acquire depth images of dairy cows during their walking process. The YOLOv8n-Seg deep learning model is used for depth image-based cow segmentation, and the ByteTrack multi-object tracking algorithm ensures continuous localization and tracking of the cows in the video. Based on pixel value conversion and depth image processing methods, images of the cow's body height above the ground and geometric features of its back are obtained. The MobileNetv2 deep learning model is used to extract fine-grained structural and semantic features of the cow's back, comprehensively classifying the cow's walking state. Finally, the lameness detection results are automatically saved and uploaded to a cloud server. This method can monitor the walking process of dairy cows in large batches in real time and provide real-time walking status. By utilizing depth image processing methods, this method deeply mines the features of the cow's back, refines the back structure and contour features related to lameness, improves the accuracy of describing the surface structure of the cow's back, significantly improves the accuracy of lameness detection, and optimizes the lameness detection process.

[0074] This method and system for detecting lameness in dairy cows, based on depth image processing, ensures that the system's operation does not affect the working environment of the pasture. Furthermore, it improves work efficiency by addressing the common problem of cows walking in wide aisles on most pastures. In terms of data acquisition, the top-view depth camera ensures that all cows passing through the aisle are captured completely, avoiding the occlusion issues that can occur with side-view cameras when multiple cows are walking side-by-side. Combining deep learning and depth image processing algorithms for cow segmentation, target tracking, and lameness detection enhances the system's resilience to complex pasture environments, improving the accuracy of lameness detection and the system's operational efficiency. By converting pixel values ​​in the depth image into the cow's height above the ground and geometric features of the cow's back, the influence of random noise on the depth image is reduced, improving the fine-grained representation and semantic representation capabilities of the data, thus enhancing the accuracy and efficiency of lameness detection.

Claims

1. A method for detecting lameness in dairy cows based on depth image processing, characterized in that... Includes the following steps: Depth images of cows walking were collected and processed to obtain uniformly sized images of the cows' body height above the ground. Using the pixel-completed image of the cow's body height above the ground, combined with the built-in parameter matrix of the depth camera, the 3D point cloud coordinates corresponding to the image of the cow's body height above the ground are calculated. This paper combines instance segmentation and multi-object tracking algorithms to achieve synchronous localization and tracking of multiple cows in depth images. For individual cows in depth images, a deep learning classifier is used to classify images of the cow's body height above the ground and images of the geometric features of the cow's back. By using score weight fusion and temporal feature fusion, lameness detection can be achieved during the walking process of dairy cows.

2. The method for detecting lameness in dairy cows based on depth image processing as described in claim 1, characterized in that, The method for obtaining images of the height of a cow's body off the ground includes the following steps: Depth images of cows walking are collected, and instance segmentation algorithms are used to segment the body parts of the cows to remove background interference from the images. Based on the output detection box of the instance segmentation model, the depth image of the cow is cropped, and an abnormal data value is removed by setting a height threshold according to the placement height of the depth camera. Based on the range of cow body height, a depth image value range is set, and values ​​outside the range are mapped to values ​​within the range. Combining the camera placement height and the cow body height, a value conversion method is used to convert the depth image values ​​corresponding to the cow body into the cow body height above the ground values. Finally, using pixel value completion technology, images of the cow's body height above the ground of uniform size were obtained.

3. The method for detecting lameness in dairy cows based on depth image processing as described in claim 2, characterized in that, The method for segmenting cow body parts using an instance segmentation algorithm and removing background interference from the image includes the following steps: Convert depth images of cows walking captured by a depth camera into pseudo-color images; Randomly select a set of pseudo-color images containing cows, and use data annotation software to annotate the body outlines of the cows in the images; Randomly select a set of pseudo-color images without cows as the background dataset, and add them directly to the labeled pseudo-color images of cows without labeling. The obtained pseudo-color images and labels are randomly divided into a training set and a test set for cow instance segmentation according to a set ratio; A YOLOv8n-Seg instance segmentation model was constructed, trained and tested on a cow instance segmentation dataset, and the optimal model weight file was obtained to optimize the instance segmentation model. The image is segmented using an optimized instance segmentation model.

4. The method for detecting lameness in dairy cows based on depth image processing as described in claim 3, characterized in that: 1000 pseudo-color images containing cows were randomly selected, and the body outlines of the cows in the images were labeled using the labelme data annotation software. 100 pseudo-color images without cows were randomly selected as the background dataset and directly added to the labeled pseudo-color images of cows without any annotation. The obtained pseudo-color images and labels are randomly divided into a training set and a test set for cow instance segmentation in an 8:2 ratio.

5. The method for detecting lameness in dairy cows based on depth image processing as described in claim 2, characterized in that, The method for cropping a cow depth image based on the output detection bounding box of the instance segmentation model includes the following steps: Based on the original image size and the size of the cow segmentation mask image output by the instance segmentation model, restore the size of the cow segmentation mask image to the original image size; The processed cow segmentation mask image is multiplied with the original depth image by corresponding pixels to obtain the segmented cow depth image. At this time, the pixel values ​​belonging to the cow target retain the original depth values, while the pixel values ​​belonging to the background environment are set to 0. Set the threshold to h - x min Pixel values ​​within the detection box that exceed the threshold are set to 0, which represents the background environment, while other pixel values ​​remain unchanged. Based on the output of the instance segmentation model, the depth image is cropped according to the location of the target detection box belonging to the cow in the depth image to obtain the depth image within the detection box.

6. The method for detecting lameness in dairy cows based on depth image processing as described in claim 2, characterized in that, The method for obtaining a uniformly sized image of the cow's body height above the ground includes the following steps: All greater than h - x min Set all pixel values ​​to h - x min ; All smaller than h - x max Set all pixel values ​​to h - x max ; Reference depth camera placement height h ,Will h Subtract all pixel values ​​from the depth image to convert the distance between the depth camera and the cow's body into the cow's height above the ground. Because the height of a dairy cow will not exceed x max The previous operation will take you to a height of more than [a certain distance] from the ground. x max All pixel values ​​are set to x max Set pixel value equal to x max All pixel values ​​are set to 0, representing background pixels. By padding with pixel values, the cropped cow depth image is transformed into a depth image with the same aspect ratio and a size of [size missing]. m × m ,in m The long side of the cropped depth image of the cow; Collect images of cows' body height off the ground, create category labels for these images, including normal cows and lame cows, and construct the first lame cow classification dataset. The image of the cow's body height off the ground is input into the MobileNetv2 classification model for model training, and the lameness classification weight file of the cow is obtained. The trained classification model was used to process the images to obtain images of cows' body height above the ground of uniform size.

7. The method for detecting lameness in dairy cows based on depth image processing as described in claim 2, characterized in that, The calculation method for the geometric feature image of a cow's back includes the following steps: The image of a cow's body height off the ground is a two-dimensional matrix, where each pixel... High ( u , v The depth camera's internal parameters include: (1) representing the height of the cow's body above the ground; (2) representing the height of the cow's body above the ground. f _ x , f _ y , c _ x and c _ y ,in f _ x and f _ y This indicates the focal length of the depth camera. c _ x and c _ y The principal point is represented by the following formula, which is used to calculate the height of the cow off the ground for each pixel in the image. u , v ) corresponding 3D points ( X , Y , Z Coordinates in the camera coordinate system: （1） （2） （3） By combining the 3D coordinates of the cow's body with several surrounding pixels to form a point cloud, the best-fit plane corresponding to the point cloud is obtained by calculating using the least squares method, and then the unit normal vector corresponding to the 3D coordinates is obtained and normalized. By statistically analyzing each unit normal vector corresponding to the 3D coordinates of the cow's body, the normal vector field corresponding to the cow's body can be obtained. The plane equation fitted to the point cloud block is shown in formula (4): （4） Where (a, b, c) are the parameters of the fitted plane; The least squares optimization objective is shown in formula (5): （5） in,( x i , y i , z i Given a set of 3D point cloud coordinates, i This indicates the index of the 3D coordinate point in the point cloud block. i The range of values ​​for is 1≤ i ≤ n , n Indicates the number of 3D coordinate points in a point cloud block; M This represents the sum of distances between each 3D coordinate point in the point cloud block and the fitted plane. The goal is to find the sum of distances when... M When the value is at its minimum, the parameters (a, b, c) of the fitted plane are obtained, and then the surface normal vector corresponding to the fitted plane is obtained. Then, the dominant direction is statistically analyzed from the normal vectors of all pixels and taken as the direction of gravity. Finally, for each pixel of the cow's body, the angle between its surface normal vector and the estimated direction of gravity is calculated. This angle is calculated using the vector dot product formula, which states that the dot product of two unit vectors is equal to the cosine of the angle between them. The angle value is then linearly transformed and shifted to fit the numerical range of an 8-bit image. This image is the geometric feature image of the cow's back.

8. The method for detecting lameness in dairy cows based on depth image processing as described in claim 1, characterized in that, The method for detecting lameness in dairy cows during walking includes the following steps: A method combining instance segmentation and multi-object tracking algorithms was used to achieve simultaneous localization and tracking of multiple cows in depth images; For individual cows in depth images, a MobileNetV2 deep learning classifier is constructed to classify images of the height of the cow's body off the ground; at the same time, a MobileNetV2 deep learning classifier is constructed to classify images of the geometric features of the cow's back. The method of score weight fusion and temporal feature fusion is used to realize lameness detection in dairy cows during walking. By utilizing data storage, data processing, and processing result synchronization technologies, a dairy cow lameness detection system is constructed to achieve synchronous positioning, tracking, and lameness detection of multiple dairy cows in a pasture environment, and upload the lameness detection results to a cloud server.

9. The method for detecting lameness in dairy cows based on depth image processing as described in claim 1, characterized in that, An Orbbec Gemini 2L binocular structured light 3D camera was used as the depth camera for data acquisition and was directly connected to an industrial computer via a USB interface.

10. The method for detecting lameness in dairy cows based on depth image processing as described in claim 1, characterized in that: The depth camera is fixed at the top center of the cow walking path using a top-down installation method, at a height of 3-5 meters above the ground, so that the field of view of the depth camera can cover the entire width of the path.

Images