A method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds.

Abstract

This invention provides a method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds. It involves collecting 3D point clouds of cattle to construct an initial point cloud dataset, using a spatial point colorization model to obtain a three-view image set, and dividing it into training, validation, and test sets. A keypoint recognition model is constructed, trained using the training set, and validated using the validation set. After obtaining the optimal keypoint recognition model, keypoint recognition is performed on the test set to obtain the two-dimensional planar coordinates of several key points. A heterogeneous keypoint coordinate transformation algorithm is used to convert the two-dimensional planar coordinates into three-dimensional spatial coordinates. The body size parameters of the cattle are calculated using the three-dimensional spatial coordinates of all key points. This invention achieves precise localization of key points in the 3D point cloud body size measurement of cattle, improving measurement accuracy. Furthermore, based on the precisely located multiple key points, various cattle body size parameters can be accurately calculated, achieving complete parameter calculation.

Description

Technical Field

[0001] This invention relates to the fields of 3D point cloud processing and deep learning technology, and more specifically, to a method for measuring the body size of cattle based on multi-view key point localization of colorized point clouds. Background Technology

[0002] Body measurement is a precise method for directly assessing the health status of cattle in modern animal husbandry. Measuring various body measurement indicators helps farmers assess the growth and development, physical quality, and production performance of cattle. Based on this data, farmers can select suitable breeding stock for selective breeding and reproduction of superior cattle, improving genetic quality and increasing production performance and economic benefits.

[0003] However, traditional manual measurement methods can no longer meet modern needs and are prone to causing stress in livestock, affecting the accuracy of results. Furthermore, long-standing problems in animal husbandry, such as insufficient technological innovation, low resource utilization efficiency, unstable product quality, low value conversion rate, and difficulty in achieving environmental benefits, have hindered the progress of agricultural modernization. Adopting non-contact technologies such as automated measurement can not only improve measurement efficiency and reduce the consumption of human resources, but also reduce the probability of stress reactions in cattle, thus enabling a more accurate assessment of health status. Therefore, automated morphological measurement technology is indispensable in improving cattle breeding techniques and selecting superior breeds to increase yield.

[0004] In recent years, non-contact automated body measurement has developed rapidly. For example, existing patent documents disclose a method for measuring the body size of cattle based on complementary information from dual cameras. This method uses two cameras, one directly above and one to the side, mounted in a passageway, to capture images of the same cow from different angles. Even when the cow's side view image is incomplete, the method first extracts measurement points from the cow's back image using a scapula and rump measurement point extraction algorithm based on YOLOv5s and envelope corner point detection. Then, it uses an auxiliary scale in the side camera image for target detection to obtain the parameters needed for height calculation. Finally, the proposed visual triangulation height measurement algorithm based on scapula measurement point extraction is used to calculate the body size. The height of the cow's scapula is measured from the ground, which is its height. Then, the difference between the cow's height and the height of the camera directly above is used to obtain the depth of field of the cow in the camera's view. Finally, the proposed similar triangle straight length measurement algorithm based on the scapula and tailbone measurement points is used to obtain the cow's body length data. However, most non-contact automatic body size measurement methods for cattle, including this one, face challenges such as: first, the external morphological features of the cow after 3D reconstruction are not obvious, making it difficult to accurately locate the key points of the cow's body size, thus affecting the accuracy of the measurement results; second, the large size of the cow makes it impossible to calculate all body size parameters from a single or non-full coverage viewpoint, resulting in incomplete measurement parameters. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies in accurately locating key points for body size measurement in 3D point clouds of cattle and inability to calculate all body size parameters, this invention provides a method for measuring cattle body size based on multi-view key point localization using colorized point clouds. Starting from the data annotation stage of deep learning, it constructs a spatial point colorization model based on global spatial features and local image features, and a heterogeneous key point coordinate transformation algorithm based on depthless camera intrinsic parameters. This achieves precise localization of key points for measuring the body size of cattle in 3D point clouds, thereby improving the accuracy of body size measurement. Simultaneously, based on the precise localization of multiple key points, various cattle body size parameters can be accurately calculated, achieving complete parameter calculation.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] A method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds includes the following steps:

[0008] S1: Collect 3D point clouds of cattle and construct an initial point cloud dataset;

[0009] S2: Input the initial point cloud dataset into the pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and annotate the key points of the training set and the validation set;

[0010] The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view.

[0011] S3: Construct a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance.

[0012] S4: Input the test set into the key point recognition model with the best performance and perform key point recognition to obtain the two-dimensional plane coordinates of several key points;

[0013] S5: Use the preset heterogeneous key point coordinate transformation algorithm to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; use the three-dimensional spatial coordinates of all key points to calculate the body size parameters of the cattle and complete the body size measurement of the cattle.

[0014] Preferably, in step S2, inputting the initial point cloud dataset into a pre-constructed spatial point colorization model for processing to obtain a three-view image set includes:

[0015] The pre-built spatial point colorization model performs the following operations on each of the 3D point clouds of cattle in the initial point cloud dataset:

[0016] Let the 3D point cloud of the cattle be denoted as point set P. The 3D spatial coordinate set of point set P is represented as: p = {(x1,y1,z1),(x2,y2,z2),..,(x n ,y n ,z n )}, where n is the number of points in the 3D point cloud of the cow, (x i ,y i ,z i () represents the 3D spatial coordinates of the i-th point in the 3D point cloud;

[0017] Adjust the orientation of the 3D point cloud of the cattle so that the head of the cattle is facing the same direction as the X-axis, Y-axis and Z-axis respectively, and the longitudinal and transverse directions of the cattle are parallel to the X-axis, Y-axis and Z-axis respectively.

[0018] The normal vectors of the three view planes of the side view, top view and rear view of the 3D point cloud of cattle are estimated based on the local surface fitting method.

[0019] Color mapping of 3D point clouds of cattle: For the side view, top view, and rear view of the cattle, extract the Z-axis coordinate set Z = {z1, z2, z3, ..., z} of the point set P. n The set of Y-axis coordinates is Y = {y1, y2, y3, ..., y...} n} and the set of X-axis coordinates X = {x1, x2, x3, ..., x n Normalize the three coordinate sets respectively, mapping them to the interval [0,1], and obtain the three normalized coordinate sets;

[0020] Based on the three normalized coordinate sets, the terrain color band in the Matplotlib data visualization library is used to perform three-dimensional color mapping on the 3D point cloud of the cow, generating 3D colored point clouds corresponding to the side view, top view and rear view of the cow respectively.

[0021] The three sets of 3D color point clouds were visualized using Open3D's physical lighting engine, and the initial 2D images of the cow from the side, top, and rear views were extracted. By calculating the normal vectors of the three views, the rendering effect can be improved.

[0022] The initial 2D images of the cow in three views are cropped and sharpened to obtain the three-view images corresponding to the 3D point cloud of the cow.

[0023] Save all the three-view images corresponding to the 3D point clouds of all cattle as a three-view image set.

[0024] Preferably, the estimation of the normal vector of the 3D point cloud of cattle based on the local surface fitting method includes:

[0025] Select a point in the 3D point cloud of cattle, and use a hybrid search strategy to search for several points within a spherical range of a preset radius around the selected point, construct a local neighborhood of the selected point, and fit a plane using the local neighborhood.

[0026] Construct the following problem and solve it:

[0027]

[0028] stA 2 +B 2 +C 2 =1

[0029] Where A, B, C, and D are the first, second, third, and fourth parameters of the fitted plane, respectively;

[0030] Save the solutions for A, B, and C as the normal vectors of the 3D point cloud of the cow. Specifically:

[0031]

[0032] Where, n x n y and n z Normal vectors The three-dimensional coordinates of A, B, and C are: ±(A0, B0, and C0) are the positive and negative terms of A, B, and C, respectively; T represents the transpose of the matrix.

[0033] Preferably, for the three coordinate sets, the following normalization processing is performed respectively:

[0034]

[0035] Where, x i,n0rm y i,norm and z i,norm These are the normalized X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; x max y max and z max x represents the maximum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; min y min and z min These are the minimum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively.

[0036] Preferably, in step S2, the Labelme data annotation tool is used to annotate key points on the training set and validation set.

[0037] Preferably, in step S3, the key point recognition model constructed based on the deep learning algorithm is specifically a YOLOv8-pose neural network model.

[0038] Preferably, in step S5, converting the two-dimensional planar coordinates of all key points into three-dimensional spatial coordinates using a preset heterogeneous key point coordinate transformation algorithm specifically involves:

[0039] Obtain the size of the 2D image used for key point recognition in the test set; construct the AABB bounding box of the 3D point cloud of the cow corresponding to the 2D image, and obtain the coordinates of the eight vertices of the AABB bounding box;

[0040] If the pixel color of each key point identified in step S3 is white, then the key point is located outside the cow's body. The two-dimensional plane coordinates of the key point are moved to the nearest color pixel for position correction. Otherwise, the key point is located inside the cow's body. Continue to detect the next key point until all key points have been detected.

[0041] Based on the size of the two-dimensional image, the vertex coordinates of the AABB bounding box, and the two-dimensional planar coordinates of all the key points that have passed the detection, the approximate planar coordinates of each key point in the two coordinate axes in the three-dimensional space are calculated, and the local region of interest of each key point in the 3D point cloud of the cattle is obtained based on the approximate planar coordinates.

[0042] The local region of interest contains several point cloud slices distributed along the third coordinate axis. The center point of the point cloud slice closest to the AABB bounding box interface is taken as the mapped key point, and the three-dimensional spatial coordinates of the center point are obtained and used as the three-dimensional spatial coordinates of the mapped key point.

[0043] Preferably, the size of the two-dimensional image is denoted as a×b, where a and b are the length and width of the two-dimensional image, respectively; the two-dimensional plane coordinates of the key points after detection are denoted as (p,q);

[0044] Starting from the top right corner of the upper surface of the AABB bounding box, the coordinates of the four vertices on the upper surface of the AABB bounding box are denoted as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4) in a clockwise direction. The coordinates of the corresponding vertices on the lower surface of the AABB bounding box are denoted as (x5, y5, z5), (x6, y6, z6), (x7, y7, z7), and (x8, y8, z8). The dimensions of the AABB bounding box are l × w × h, where l, w, and h are the length, width, and height of the AABB bounding box, respectively.

[0045] If the 3D coordinates of the mapped keypoints are denoted as (x0, y0, z0), then:

[0046] If the 2D image is a side view, calculate the approximate coordinates of the keypoints (p, q) on the X and Y axes using the following formula:

[0047]

[0048] At this point, the acquired point cloud slices are distributed along the Z-axis, and the value of z0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0049] If the 2D image is a top view, calculate the approximate coordinates of the key points (p, q) on the X and Z axes using the following formula;

[0050]

[0051] At this point, the acquired point cloud slices are distributed along the Y-axis, and the value of y0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0052] If the 2D image is a rear view, calculate the approximate coordinates of the key points (p, q) in the plane on the Y and Z axes according to the following formula;

[0053]

[0054]

[0055] At this point, the acquired point cloud slices are distributed along the X-axis, and the value of x0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0056] Preferably, in step S5, the body size parameters of the cattle include: body length, chest depth, rump height, croup length, body height, body length, abdominal circumference, cross-section height, chest circumference, chest width, waist angle width, hip width, and ischial end width.

[0057] This invention also provides a cattle body size measurement system based on multi-view keypoint localization using colorized point clouds. The cattle body size measurement method based on multi-view keypoint localization using colorized point clouds described above includes:

[0058] Point cloud acquisition unit: used to acquire 3D point clouds of cattle and build an initial point cloud dataset;

[0059] Dataset construction unit: used to input the initial point cloud dataset into a pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and perform key point annotation on the training set and the validation set;

[0060] The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view.

[0061] Model training unit: used to build a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance.

[0062] Key point recognition unit: used to perform key point recognition on the key point recognition model with the best performance of the test set, and obtain the two-dimensional plane coordinates of several key points;

[0063] Body size measurement unit: Used to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates using a preset heterogeneous key point coordinate transformation algorithm; calculate the body size parameters of cattle using the three-dimensional spatial coordinates of all key points, and complete the body size measurement of cattle.

[0064] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0065] This invention provides a method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds. First, 3D point clouds of cattle are collected to construct an initial point cloud dataset. This initial point cloud dataset is then input into a pre-constructed spatial point colorization model for processing, resulting in a three-view image set divided into training, validation, and test sets. Keypoints are labeled on both the training and validation sets. Next, a keypoint recognition model is constructed based on a deep learning algorithm. This model is trained using the labeled training set and validated using the labeled validation set to obtain the optimal keypoint recognition model. The test set is then input into the optimal keypoint recognition model for keypoint recognition, obtaining the two-dimensional planar coordinates of several keypoints. Finally, a pre-defined heterogeneous keypoint coordinate transformation algorithm is used to convert the two-dimensional planar coordinates of all keypoints into three-dimensional spatial coordinates. The body size parameters of the cattle are calculated using the three-dimensional spatial coordinates of all keypoints, completing the body size measurement.

[0066] In existing technologies, improving deep learning networks is commonly used to enhance the accuracy of keypoint localization. However, this often overlooks the accuracy of data annotation in deep learning. Therefore, this invention addresses the data annotation stage of deep learning by constructing a spatial point colorization model based on global spatial features and local image features. It employs a strategy combining geographic terrain color bands with cattle body point clouds to transform the color of the cattle phenotypic point cloud into terrain-like colors from different orthogonal perspectives. This, combined with physical rendering of the point cloud, highlights surface texture effects and generates 2D images from different perspectives, transforming the 3D point cloud data annotation problem into a 2D image data annotation problem. This reduces the difficulty of data annotation and improves the accuracy of keypoint localization. Secondly, this invention, along with a heterogeneous keypoint coordinate transformation algorithm based on depthless camera intrinsic parameters, enables the mutual conversion between 3D point cloud coordinates and 2D image coordinates, achieving precise localization of key points for cattle 3D point cloud body size measurement, thereby improving the accuracy of body size measurement. Furthermore, based on multiple precisely located key points, this invention can accurately calculate various cattle body size parameters, achieving complete parameter calculation. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method for measuring the body size of cattle based on multi-view key point localization using colorized point clouds, as provided in Example 1.

[0068] Figure 2 This is an architecture diagram of a method for measuring the body size of cattle based on multi-view key point localization using colorized point clouds, as provided in Example 2.

[0069] Figure 3 This is an example image of a 3D point cloud of a cow provided in Example 2.

[0070] Figure 4 This is a flowchart of the spatial point colorization model data processing provided in Example 2.

[0071] Figure 5 This is a schematic diagram of the possible locations of key points estimated using statistical methods, as provided in Example 2.

[0072] Figure 6 This is a schematic diagram of the data annotation heatmap auxiliary mechanism provided in Example 2.

[0073] Figure 7 This is a schematic diagram of key point annotations provided in Example 2.

[0074] Figure 8 This is a schematic diagram of the AABB bounding box provided in Example 2.

[0075] Figure 9 This is a schematic diagram of the bovine body key point mapping calculation provided in Example 2.

[0076] Figure 10 This is a schematic diagram of the three-dimensional spatial coordinates of the key points after mapping, as provided in Example 2.

[0077] Figure 11 This is a schematic diagram of the final mapping effect of the key points in the top view provided in Example 2.

[0078] Figure 12 This is a schematic diagram of the measurement parameters of various body dimensions of cattle provided in Example 2.

[0079] Figure 13 This is a comparison image of the original color map and the colorized map of the cattle point cloud provided in Example 2.

[0080] Figure 14 This is a comparison of the training results of the original color map and the colorized map of the cattle point cloud provided in Example 2.

[0081] Figure 15 This is a box-type diagram for measuring various body size indicators of cattle provided in Example 2.

[0082] Figure 16 This is a structural diagram of a cattle body size measurement system based on multi-view key point localization using colorized point clouds, as provided in Example 3. Detailed Implementation

[0083] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0084] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0085] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0086] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0087] Example 1

[0088] like Figure 1 As shown, this embodiment provides a method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds, including the following steps:

[0089] S1: Collect 3D point clouds of cattle and construct an initial point cloud dataset;

[0090] S2: Input the initial point cloud dataset into the pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and annotate the key points of the training set and the validation set;

[0091] The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view.

[0092] S3: Construct a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance.

[0093] S4: Input the test set into the key point recognition model with the best performance and perform key point recognition to obtain the two-dimensional plane coordinates of several key points;

[0094] S5: Use the preset heterogeneous key point coordinate transformation algorithm to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; use the three-dimensional spatial coordinates of all key points to calculate the body size parameters of the cattle and complete the body size measurement of the cattle.

[0095] In the specific implementation process, the first step is to collect 3D point clouds of cattle and construct an initial point cloud dataset;

[0096] The initial point cloud dataset is input into a pre-built spatial point colorization model for processing to obtain a three-view image set and divide it into a training set, a validation set and a test set. Key point annotation is performed on the training set and the validation set.

[0097] Subsequently, a key point recognition model was constructed based on deep learning algorithms. The key point recognition model was trained using the training set after key point annotation, and the trained key point recognition model was validated using the validation set after key point annotation, so as to obtain the key point recognition model with the best performance.

[0098] Then, the best-performing key point recognition model is input into the test set to perform key point recognition and obtain the two-dimensional plane coordinates of several key points;

[0099] Finally, the preset heterogeneous key point coordinate transformation algorithm is used to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; the body size parameters of the cattle are calculated using the three-dimensional spatial coordinates of all key points, and the body size measurement of the cattle is completed.

[0100] This method starts with the data annotation stage of deep learning. By constructing a spatial point colorization model based on global spatial features and local image features, and a heterogeneous key point coordinate transformation algorithm based on depthless camera intrinsic parameters, it achieves accurate positioning of key points for measuring the body size of cattle in 3D point cloud, thereby improving the accuracy of body size measurement. At the same time, based on multiple accurately positioned key points, it can accurately calculate various body size parameters of cattle, realizing complete parameter calculation.

[0101] Example 2

[0102] This embodiment provides a method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds, including the following steps:

[0103] S1: Collect 3D point clouds of cattle and construct an initial point cloud dataset;

[0104] S2: Input the initial point cloud dataset into the pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and annotate the key points of the training set and the validation set;

[0105] The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view.

[0106] S3: Construct a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance.

[0107] S4: Input the test set into the key point recognition model with the best performance and perform key point recognition to obtain the two-dimensional plane coordinates of several key points;

[0108] S5: Use the preset heterogeneous key point coordinate transformation algorithm to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; use the three-dimensional spatial coordinates of all key points to calculate the body size parameters of the cattle and complete the body size measurement of the cattle.

[0109] In step S2, the initial point cloud dataset is input into a pre-constructed spatial point colorization model for processing to obtain a three-view image set, including:

[0110] The pre-built spatial point colorization model performs the following operations on each of the 3D point clouds of cattle in the initial point cloud dataset:

[0111] Let the 3D point cloud of the cattle be denoted as point set P. The 3D spatial coordinate set of point set P is represented as: P = {(x1,y1,z1),(x2,y2,z2),..,(x n ,y n ,z n )}, where n is the number of points in the 3D point cloud of the cow, (x i ,y i ,z i () represents the 3D spatial coordinates of the i-th point in the 3D point cloud;

[0112] Adjust the orientation of the 3D point cloud of the cattle so that the head of the cattle is facing the same direction as the X-axis, Y-axis and Z-axis respectively, and the longitudinal and transverse directions of the cattle are parallel to the X-axis, Y-axis and Z-axis respectively.

[0113] The normal vectors of the three view planes of the side view, top view and rear view of the 3D point cloud of cattle are estimated based on the local surface fitting method.

[0114] Color mapping of 3D point clouds of cattle: For the side view, top view, and rear view of the cattle, extract the Z-axis coordinate set Z = {z1, z2, z3, ..., z} of the point set P. n The set of Y-axis coordinates is Y = {y1, y2, y3, ..., y...} n} and the set of X-axis coordinates X = {x1, x2, x3, ..., x n Normalize the three coordinate sets respectively, mapping them to the interval [0,1], and obtain the three normalized coordinate sets;

[0115] Based on the three normalized coordinate sets, the terrain color band in the Matplotlib data visualization library is used to perform three-dimensional color mapping on the 3D point cloud of the cow, generating 3D colored point clouds corresponding to the side view, top view and rear view of the cow respectively.

[0116] The three sets of 3D color point clouds were visualized using Open3D's physical lighting engine, and the initial two-dimensional images of the cow's side view, top view, and rear view were extracted.

[0117] The initial 2D images of the cow in three views are cropped and sharpened to obtain the three-view images corresponding to the 3D point cloud of the cow.

[0118] Save the three-view images corresponding to the 3D point clouds of all cattle into a three-view image set;

[0119] The method for estimating the normal vector of the 3D point cloud of cattle based on local surface fitting includes:

[0120] Select a point in the 3D point cloud of cattle, and use a hybrid search strategy to search for several points within a spherical range of a preset radius around the selected point, construct a local neighborhood of the selected point, and fit a plane using the local neighborhood.

[0121] Construct the following problem and solve it:

[0122]

[0123] stA 2 +B 2 +C 2 =1

[0124] Where A, B, C, and D are the first, second, third, and fourth parameters of the fitted plane, respectively;

[0125] Save the solutions for A, B, and C as the normal vectors of the 3D point cloud of the cow. Specifically:

[0126]

[0127] Where, n x n y and n z Normal vectors The three-dimensional coordinates of A, B, and C are given; ±(A0, B0, and C0) represent the positive and negative terms of A, B, and C, respectively; T denotes the transpose of the matrix.

[0128] For the three coordinate sets, the following normalization process is performed respectively:

[0129]

[0130] Where, x i,norm y i,norm and z i,norm These are the normalized X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; x max y max and z max x represents the maximum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; min y min and z min These are the minimum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively.

[0131] In step S2, the Labelme data annotation tool is used to annotate key points on the training set and validation set.

[0132] In step S3, the key point recognition model constructed based on the deep learning algorithm is specifically the YOLOv8-pose neural network model.

[0133] In step S5, the preset heterogeneous key point coordinate transformation algorithm is used to convert the two-dimensional planar coordinates of all key points into three-dimensional spatial coordinates. Specifically, this is as follows:

[0134] Obtain the size of the 2D image used for key point recognition in the test set; construct the AABB bounding box of the 3D point cloud of the cow corresponding to the 2D image, and obtain the coordinates of the eight vertices of the AABB bounding box;

[0135] If the pixel color of each key point identified in step S3 is white, then the key point is located outside the cow's body. The two-dimensional plane coordinates of the key point are moved to the nearest color pixel for position correction. Otherwise, the key point is located inside the cow's body. Continue to detect the next key point until all key points have been detected.

[0136] Based on the size of the 2D image, the vertex coordinates of the AABB bounding box, and the 2D planar coordinates of all the key points that have passed the detection, the approximate planar coordinates of each key point in the 3D space corresponding to the two coordinate axes are calculated, and the local region of interest of each key point in the 3D point cloud of the cow is obtained based on the approximate planar coordinates; the local region of interest contains several point cloud slices distributed along the third coordinate axis.

[0137] The size of the two-dimensional image is denoted as a×b, where a and b are the length and width of the two-dimensional image, respectively; the two-dimensional plane coordinates of the key points after detection are denoted as (p,q);

[0138] Starting from the top right corner of the upper surface of the AABB bounding box, the coordinates of the four vertices on the upper surface of the AABB bounding box are denoted as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4) in a clockwise direction. The coordinates of the corresponding vertices on the lower surface of the AABB bounding box are denoted as (x5, y5, z5), (x6, y6, z6), (x7, y7, z7), and (x8, y8, z8). The dimensions of the AABB bounding box are l × w × h, where l, w, and h are the length, width, and height of the AABB bounding box, respectively.

[0139] If the 3D coordinates of the mapped keypoints are denoted as (x0, y0, z0), then:

[0140] If the 2D image is a side view, calculate the approximate coordinates of the keypoints (p, q) on the X and Y axes using the following formula:

[0141]

[0142] At this point, the acquired point cloud slices are distributed along the Z-axis, and the value of z0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0143] If the 2D image is a top view, calculate the approximate coordinates of the key points (p, q) on the X and Z axes using the following formula;

[0144]

[0145] At this point, the acquired point cloud slices are distributed along the Y-axis, and the value of y0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0146] If the 2D image is a rear view, calculate the approximate coordinates of the key points (p, q) in the plane on the Y and Z axes according to the following formula;

[0147]

[0148] At this point, the acquired point cloud slices are distributed along the X-axis, and the value of x0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0149] The center point of the point cloud slice that is closest to the AABB bounding box interface is used as the key point after mapping. The three-dimensional spatial coordinates of the center point are obtained and used as the three-dimensional spatial coordinates of the key point after mapping.

[0150] In step S5, the body size parameters of the cattle include: body length, chest depth, rump height, croup length, body height, body length, abdominal circumference, cross-section height, chest circumference, chest width, waist angle width, hip width, and ischial end width.

[0151] In the specific implementation process, such as Figure 2 The architecture diagram shown first involves collecting 3D point clouds of cattle and constructing an initial point cloud dataset, such as... Figure 3 As shown, Figure 3 (a) and (b) are 3D point cloud examples of dairy buffalo and dairy cow, respectively;

[0152] After acquiring the point cloud dataset, a spatial point colorization model is constructed. Spatial color mapping utilizes the spatial distribution characteristics of the reconstructed 3D point cloud to map specific colors onto the 3D point cloud, overwriting the original body colors. Based on this, Open3D lighting and physical model rendering is performed. After determining the overall color characteristics of the point cloud, a two-dimensional view of the point cloud is obtained, and the two-dimensional view is sharpened to highlight local image details. In this embodiment, the model is used to transform the initial point cloud dataset into a training dataset for a neural network model. In this embodiment, the initial point cloud dataset is input into the pre-constructed spatial point colorization model for processing, obtaining a three-view image set and dividing it into a training set, a validation set, and a test set in a 7:2:1 ratio.

[0153] Specifically, in this embodiment, such as Figure 4 The diagram shows the data processing flow of the spatial point colorization model. The following operations are performed on each cow's 3D point cloud:

[0154] Let the 3D point cloud of the cattle be denoted as point set P. The 3D spatial coordinate set of point set P is represented as: P = {(x1,y1,z1),(x2,y2,z2),..,(x n ,y n ,z n )}, where n is the number of points in the 3D point cloud of the cow, (x i ,y i ,z i () represents the 3D spatial coordinates of the i-th point in the 3D point cloud;

[0155] Adjust the orientation of the 3D point cloud of the cattle so that the head of the cattle is facing the same direction as the X-axis, Y-axis and Z-axis respectively, and the longitudinal and transverse directions of the cattle are parallel to the X-axis, Y-axis and Z-axis respectively.

[0156] The method of local surface fitting is used to estimate the normal vectors of the three view planes of the side view, top view and rear view of the 3D point cloud of cattle. The method fits a plane based on the points in the neighborhood of a point, and the normal direction of the plane is the normal vector of the point.

[0157] The steps for constructing the normal vector in this embodiment are as follows:

[0158] Select a point in the 3D point cloud of cattle, and use a hybrid search strategy to search for up to 30 points within a spherical range with a radius of 0.1 around the selected point. Construct a local neighborhood of the selected point and fit a plane using the local neighborhood.

[0159] Construct the following problem and solve it:

[0160]

[0161] stA 2 +B 2 +C 2 =1

[0162] Where A, B, C, and D are the first, second, third, and fourth parameters of the fitted plane, respectively;

[0163] Save the solutions for A, B, and C as the normal vectors of the 3D point cloud of the cow. Specifically:

[0164]

[0165] Where, n x n y and n z Normal vectors The three-dimensional coordinates of A, B, and C are given; ±(A0, B0, and C0) represent the positive and negative terms of A, B, and C, respectively; T denotes the transpose of the matrix.

[0166] Color mapping of 3D point clouds of cattle: For the side view, top view, and rear view of the cattle, extract the Z-axis coordinate set Z = {z1, z2, z3, ..., z} of the point set P. n The set of Y-axis coordinates is Y = {y1, y2, y3, ..., y...} n} and the set of X-axis coordinates X = {x1, x2, x3, ..., x n Normalize the three coordinate sets respectively, mapping them to the interval [0,1], and obtain the three normalized coordinate sets;

[0167] For the three coordinate sets, the following normalization process is performed respectively:

[0168]

[0169] Where, x i,norm y i,norm and z i,norm These are the normalized X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; x max y max and z max x represents the maximum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively; min y min and z min These are the minimum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively.

[0170] Based on the three normalized coordinate sets, the terrain color band in the Matplotlib data visualization library is used to perform three-dimensional color mapping on the 3D point cloud of the cow, generating 3D colored point clouds corresponding to the side view, top view and rear view of the cow respectively.

[0171] Matplotlib is a widely used data visualization library in Python. It provides a rich set of colormaps for generating high-quality charts. Colormaps are arrays representing the mapping between data values ​​and colors, including five main categories suitable for different numerical scenarios: Sequential colormaps are suitable for scenarios where data values ​​are ordered and continuously change from low to high or high to low; Discrete colormaps are suitable for scenarios where data is not ordered and only represents different categories or discrete values; Divergent colormaps are suitable for scenarios where data has a central point and colors radiate outwards from the center; Cyclic colormaps are suitable for scenarios where data changes periodically; and Miscellaneous colormaps are suitable for user-defined scenarios. Therefore, considering the distribution characteristics of cow body point clouds, this method uses the "terrain_r" colormap. "Rain" is a special color band typically used to represent terrain and elevation data. This color band simulates the color variations of natural terrain, with blue and green representing low-altitude areas and brown and white representing high-altitude areas, similar to the color allocation scheme of topographic maps. This color band, combined with the point cloud showing the unevenness of the cattle's body surface, ensures consistency between the color distribution and the unevenness of the point cloud, enhancing the visualization of the data and facilitating researchers to quickly identify and analyze the surface features of the cattle based on color differences. In this method, to achieve the best visual effect, the "terrain" color band is inverted, resulting in a bluer color for higher point cloud elevations and a whiter color for lower elevations. This processing creates color differences between the uneven parts of the cattle's point cloud, helping annotators identify the locations of joints and key points on the cattle's body.

[0172] The three sets of 3D color point clouds are visualized using Open3D's physical lighting engine to make the cow point cloud closer to its physical representation. Initial two-dimensional images of the cow's side view, top view, and rear view are extracted from three orthogonal perspectives. In this embodiment, the orthogonal side view, top view, and rear view of the cow point cloud are captured by setting the "camera orientation" of the physical lighting engine.

[0173] The initial 2D images of the cow in three views are cropped and sharpened to obtain the three-view images corresponding to the 3D point cloud of the cow.

[0174] The purpose of cropping is to remove excess background pixels so that the edges of the cow's body are close to the edges of the image, achieving the same bounding effect as the AABB bounding box of the point cloud, thus laying the groundwork for the transformation of point coordinates in different dimensions. The main process is to convert the image to grayscale, search for non-white pixels in the image, crop the smallest region of the cow's color image based on the boundary coordinates of the non-white pixels, and save the cropped image.

[0175] The purpose of image sharpening is to enhance its contrast. Let the filter matrix K be:

[0176]

[0177] This filter is an edge detection filter that can detect edges along the diagonal direction. It reacts to changes along a specific diagonal direction. Applying this filter in image processing can detect and extract edge features in cattle images, making the contours of different parts of the cattle clearer. Assuming the image matrix I is (i,j), representing the pixel value at position (i,j), and (m,n) is the coordinate convolution result of the output matrix O, matrix O can be represented as:

[0178]

[0179] Then, the three-view images corresponding to the 3D point clouds of all cattle are saved together as a three-view image set and further divided into datasets; after the division is completed, they need to be labeled before they can be used for neural network model training.

[0180] This embodiment completes data annotation with the aid of a keypoint heatmap based on statistical patterns. It uses the statistical patterns of the 3D point cloud of the bovine body to predict key point regions of the bovine body size, and combines a heterogeneous keypoint coordinate transformation method to map these 3D keypoint regions onto a 2D image. The mapped 2D coordinate information generates a "keypoint region heatmap," providing a reference for data annotation. Specifically, the bovine body point cloud is horizontally sliced ​​along the point cloud space, with a slice distance of 1cm. The number of point clouds in each slice is counted, such as... Figure 5As shown, the possible areas for measuring key points such as body height, body length, chest circumference, and abdominal circumference of cattle are located based on the difference in the number of point clouds between the previous and subsequent slices. Furthermore, the possible areas for the key points in the 3D point cloud are statistically analyzed (for...). Figure 5 The three vertical lines in the lower right image are used to transform the heterogeneous keypoint coordinates into two-dimensional image keypoint coordinates using the proposed heterogeneous keypoint coordinate transformation algorithm. Finally, the keypoint coordinates of the two-dimensional image are used to form a thermal-assisted data annotation map, as shown below. Figure 6 As shown, in Figure 6 In the right figure, the three vertical lines indicate the possible locations of key measurement points;

[0181] While the generated heatmap can provide information about the labeled areas, it does not represent the actual manual annotation; it only provides a reference for the location of the data annotations. This is because the keypoint localization algorithm based on statistical point clouds itself has insufficient robustness, which can cause the keypoint area to deviate. In this case, manual correction of the annotation position is required during the data annotation stage. In this embodiment, with the assistance of the heatmap, the Labelme data annotation tool is finally used to... Figure 7 Keypoints are labeled on the training and validation sets; English letters and numbers are used to represent the keypoint codes.

[0182] Subsequently, a key point recognition model is constructed based on deep learning algorithms. In this embodiment, the key point recognition model is specifically a YOLOv8-pose neural network model.

[0183] The YOLOv8-pose neural network model was trained using the training set with keypoint annotations, and the trained YOLOv8-pose neural network model was validated using the validation set with keypoint annotations to obtain the YOLOv8-pose neural network model with the best performance.

[0184] Then, the YOLOv8-pose neural network model with the best performance was input into the test set for key point recognition, and the two-dimensional plane coordinates of several key points were obtained.

[0185] The pre-defined heterogeneous key point coordinate transformation algorithm is used to convert the two-dimensional planar coordinates of all key points into three-dimensional spatial coordinates.

[0186] Specifically, in this embodiment, the size of the 2D image used for key point recognition in the test set is obtained; the AABB bounding box of the 3D point cloud of the cow corresponding to the 2D image is constructed, and the coordinates of the eight vertices of the AABB bounding box are obtained; the AABB bounding box is a commonly used geometric representation method used to enclose and approximate the boundary of point cloud data. Its characteristic is that it is a cube or cuboid, and all six faces are parallel to the coordinate axes of the point cloud, so it is called an axis-aligned bounding box, and its boundary is aligned with the minimum and maximum coordinate values ​​of the point cloud data;

[0187] The size of the two-dimensional image is denoted as a×b, where a and b are the length and width of the two-dimensional image, respectively; the two-dimensional plane coordinates of the key points after detection are denoted as (p,q);

[0188] like Figure 8 As shown, starting from the top right corner of the top surface of the AABB bounding box, the coordinates of the four vertices on the top surface of the AABB bounding box are denoted as (x1, y1, z1), (x2, y2, z2), (x3, y3, z3), and (x4, y4, z4) in a clockwise direction. The coordinates of the corresponding vertices on the bottom surface of the AABB bounding box are denoted as (x5, y5, z5), (x6, y6, z6), (x7, y7, z7), and (x8, y8, z8). The dimensions of the AABB bounding box are l × w × h, where l, w, and h are the length, width, and height of the AABB bounding box, respectively.

[0189] Keypoint localization in 2D images is problematic because deep learning directly participates in inference, and many keypoints are correctly located at the edges of the cow's body. This can lead to keypoints being incorrectly located outside the cow's body. However, colorization techniques give each pixel of the cow's body a different color, which, when contrasted with a pure white background, can help detect whether a keypoint is inside the cow's body. This method detects whether the pixel color of each keypoint identified by the best-performing YOLOv8-pose neural network model is white. If it is, the keypoint is outside the cow's body, and its 2D coordinates are migrated to the nearest colored pixel for position correction. Otherwise, the keypoint is inside the cow's body, and the process continues to detect the next keypoint until all keypoints have been detected.

[0190] Based on the size of the 2D image, the vertex coordinates of the AABB bounding box, and the 2D planar coordinates of all the key points that have passed the detection, the approximate planar coordinates of each key point in the 3D space corresponding to the two coordinate axes are calculated, and the local region of interest of each key point in the 3D point cloud of the cow is obtained based on the approximate planar coordinates; the local region of interest contains several point cloud slices distributed along the third coordinate axis.

[0191] If the 3D coordinates of the mapped keypoints are denoted as (x0, y0, z0), then:

[0192] If the 2D image is a side view, calculate the approximate coordinates of the keypoints (p, q) on the X and Y axes using the following formula:

[0193]

[0194] At this point, the acquired point cloud slices are distributed along the Z-axis, and the value of z0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0195] If the 2D image is a top view, calculate the approximate coordinates of the key points (p, q) on the X and Z axes using the following formula;

[0196]

[0197] At this point, the acquired point cloud slices are distributed along the Y-axis, and the value of y0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0198] If the 2D image is a rear view, calculate the approximate coordinates of the key points (p, q) in the plane on the Y and Z axes according to the following formula;

[0199]

[0200] At this point, the acquired point cloud slices are distributed along the X-axis, and the value of x0 is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

[0201] like Figure 9 The diagram shows a key point mapping diagram in the top view. The left image is a 2D image after key point detection, and the right image is a top view of the 3D point cloud. The outer solid line represents the AABB bounding box of the point cloud. The mapping process for the side view and the rear view is the same as that for the top view.

[0202] Taking the top-down view as an example, the coordinates of keypoints on the X and Z axes in the point cloud space are determined through two-dimensional coordinate mapping. These coordinates further define the region of interest (ROI) of the keypoint in the 3D point cloud. Thanks to the high-resolution sampling capability of the depth camera, the single pixel extracted from the two-dimensional view corresponds to a small but complex ROI in the 3D point cloud; for example... Figure 10 As shown, the region of interest consists of two or more point cloud blocks arranged at certain intervals in 3D space. After determining the 3D key points, this method starts from the top view and adopts a top-down search strategy. Within the local region of interest, it searches along the Y-axis starting from the maximum Y value (i.e., the top of the point cloud bounding box). During this process, it retrieves the regional discrete point cloud group closest to the mapping line's direction of travel and selects the center point of the point cloud in that region as the actual coordinate of the key point. The final mapping effect is shown in the figure. Figure 11 As shown;

[0203] Finally, the body size parameters of the cattle were calculated using the three-dimensional spatial coordinates of all key points, thus completing the body size measurement of the cattle; for example... Figure 12The measurement diagram shown categorizes cattle body measurements into three types: point-to-plane distance, point-to-point distance, and point-to-circular distance. Specifically, these include: body length, chest depth, rump height, croup length, body height, body length, abdominal circumference, cross-shaped height, chest circumference, chest width, loin angle width, hip width, and ischial end width. The detailed definitions of each body measurement parameter are shown in Table 1.

[0204] Table 1 Definitions of Cattle Body Size Measurement Indicators

[0205] Body size name code name explain body height WH Vertical distance from the highest point of the withers to the ground Deep chest CD The vertical distance along the posterior border of the scapula from the withers to the sternum. chest circumference CC The length around the chest at the back of the scapula. Waist circumference AC Length around the largest part of the abdomen Cross-shaped high HH The vertical height from the center of the two waist angles (the cross) to the ground Body oblique length BDL The straight-line distance from the anterior end of the scapula to the posterior end of the ipsilateral ischial tuberosity High hip HEH Vertical distance from the ischial tuberosity to the ground Body straight length DL Horizontal distance from the shoulder to the posterior edge of the ischial tuberosity chest width CB The width of the posterior border of the scapula at the widest point of the chest Buttock length CL The straight-line distance from the front edge of the waist to the back edge of the buttocks. Waist Angle Width WW Maximum width of the left and right lumbar angles (hip tuberosities) Hip width HW Maximum width of the left and right hips (hip joints) Ischial end width IW The width between the outermost protuberances of the left and right ischial tuberosities

[0206] The height of the cow's body, the height of its rump, and the height of its cross can be converted into the calculation of the distance from a point to a plane. For this type of body size parameter, first capture the ground point cloud plane, and then calculate the distance from the three-dimensional spatial coordinates of the key point to the ground point cloud plane.

[0207] The calculation of body dimensions of cattle, such as body length, chest depth, rump length, body length, chest width, loin angle width, hip width, and ischial end width, can be converted into the calculation of point-to-point distance, which can be directly used by the Euclidean distance between the three-dimensional spatial coordinates of two key points.

[0208] The abdominal and chest circumferences of cattle can be converted into point-based or arc-based distance calculations. For this type of body size parameter, the polar coordinate point method is used for calculation. This method takes into account that a circular curve in a rectangular coordinate system is a straight line perpendicular to the polar coordinate system. Calculating the abdominal circumference in the polar coordinate system is simpler than calculating the body size in the rectangular coordinate system. Furthermore, the method of converting the point set of the perimeter shape into a one-dimensional straight line is more conducive to filling in the data loss caused by gaps, so as to achieve a result close to the actual body size measurement of cattle. Since both are calculations of arc lengths, this repair method is applicable to the above two body size calculations.

[0209] To verify the practicality and effectiveness of spatial color mapping, this method also designed a key point recognition comparison experiment between the original point cloud color image and the color-mapped point cloud image. Both were trained under the same key point detection network in YOLOv8-pose.

[0210] Visual annotation comparison experiment analysis: In order to scientifically and accurately determine the location of key points on the cow's body during data annotation, detailed contour information of the cow's surface is required; however, there are differences between the point cloud images of the cow's body without color mapping and those with color mapping in the legs and waist areas; for example... Figure 13 As shown, the color-mapped cow body exhibits richer texture details. For example, when identifying key points near the cow's legs, the color map can accurately mark the key point locations, avoiding marking the points on the cow's legs. In contrast, the original image makes it difficult to see the boundary between the cow's legs and body, and the outline of the waist angle can be identified in the cow's waist angle area. Therefore, the color mapping scheme provides convenience in the data annotation stage.

[0211] A comparative experiment was conducted to label keypoints: Images without a color mapping model were labeled with the same number of keypoints as those used in this method. After labeling, the data was input into the YOLOv8-pose keypoint detection network for training and prediction. The training process for both models is as follows: Figure 14 As shown; by Figure 14 As can be seen, in this method, the mAP value is relatively high from the beginning, basically maintained above 0.8, showing high accuracy from the start. Although there are a few slight fluctuations, overall, the mAP value remains around 0.95, showing extremely high stability and superiority. Combined with the analysis, the results of this method are significantly better than the original color results, maintaining a higher mAP value almost throughout the entire training process. In terms of training stability, the method in this paper also performs better, with smaller fluctuations in its mAP value, which is closer to the ideal performance.

[0212] Therefore, the model using color mapping in the key point annotation stage facilitates the work of annotators, making the data annotation process more efficient and accurate; secondly, from the perspective of deep learning training, by combining spatial point cloud color technology, the model shows advantages in the recognition of key points on the cow's body.

[0213] Secondly, this embodiment also evaluates the performance of the YOLOv8-pose network in the keypoint detection task; the final results show that the average accuracy (mAP50-95) for target detection is 99.5%, and the average accuracy for keypoints is 96.2%.

[0214] In addition, to statistically analyze the error of the bovine body size index and verify the repeatability and reliability of this method, this embodiment randomly selected four cows for multiple repeatability error experiments. When each cow passed through the acquisition channel, a front railing was used to control the cow within the acquisition channel, and acquisition was performed every 10 seconds. Ultimately, point cloud data was captured 10 times for each of the four cows, totaling 40 point cloud data points. A dataset of 120 keypoint detection images was generated using the color mapping model presented in this paper. Under the guidance of professional surveyors at the farm, multiple repeated measurement experiments were conducted on these four cows using a soft measuring tape, determining 13 body size lengths for each cow. The average of each body size length was taken as the true body size value for that cow. The keypoints of the two-dimensional images of the cows after keypoint detection were converted into 3D actual spatial coordinates, and the body size was calculated using the actual 3D coordinates of the cows. Finally, error analysis was performed between the measurement results and the actual values, which effectively evaluated the accuracy of this method. An example of the average error of the final bovine body size data is shown below. Figure 15 The relative error of the overall body size data is shown in Table 2. It can be seen from the table that the overall average error of the body size measurement is 3.61%, and the measurement is relatively accurate.

[0215] Table 2 MAE and MAPE for body size measurements

[0216]

[0217]

[0218] This method starts with the data annotation stage of deep learning. By constructing a spatial point colorization model based on global spatial features and local image features, and a heterogeneous key point coordinate transformation algorithm based on depthless camera intrinsic parameters, it achieves accurate positioning of key points for measuring the body size of cattle in 3D point cloud, thereby improving the accuracy of body size measurement. At the same time, based on multiple accurately positioned key points, it can accurately calculate various body size parameters of cattle, realizing complete parameter calculation.

[0219] Example 3

[0220] like Figure 16 As shown, this embodiment provides a cattle body size measurement system based on multi-view keypoint localization using colorized point clouds. It applies the cattle body size measurement method based on multi-view keypoint localization using colorized point clouds described in Embodiment 1 or 2, including:

[0221] Point cloud acquisition unit 301: used to acquire 3D point clouds of cattle and build an initial point cloud dataset;

[0222] Dataset construction unit 302: is used to input the initial point cloud dataset into a pre-constructed spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and annotate the training set and the validation set with key points;

[0223] The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view.

[0224] Model training unit 303: is used to build a key point recognition model based on deep learning algorithm, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation, so as to obtain the key point recognition model with the best performance.

[0225] Key point recognition unit 304: used to perform key point recognition on the key point recognition model with the best performance of the test set input, and obtain the two-dimensional plane coordinates of several key points;

[0226] Body size measurement unit 305: Used to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates using a preset heterogeneous key point coordinate transformation algorithm; calculate the body size parameters of the cattle using the three-dimensional spatial coordinates of all key points, and complete the body size measurement of the cattle.

[0227] In the specific implementation process, the point cloud acquisition unit 301 first acquires the 3D point cloud of the cattle and constructs the initial point cloud dataset;

[0228] The dataset construction unit 302 inputs the initial point cloud dataset into the pre-built spatial point colorization model for processing, obtains a three-view image set and divides it into a training set, a validation set and a test set, and performs key point annotation on the training set and the validation set;

[0229] Subsequently, the model training unit 303 constructs a key point recognition model based on a deep learning algorithm, trains the key point recognition model using the training set after key point annotation, and validates the trained key point recognition model using the validation set after key point annotation, thereby obtaining the key point recognition model with the best performance.

[0230] Then, the key point recognition unit 304 will input the test set and the key point recognition model with the best performance to perform key point recognition and obtain the two-dimensional plane coordinates of several key points.

[0231] Finally, the body size measurement unit 305 uses a preset heterogeneous key point coordinate transformation algorithm to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; it uses the three-dimensional spatial coordinates of all key points to calculate the body size parameters of the cattle and completes the body size measurement of the cattle.

[0232] This system starts with the data annotation stage of deep learning. By constructing a spatial point colorization model based on global spatial features and local image features, and a heterogeneous key point coordinate transformation algorithm based on depthless camera intrinsic parameters, it achieves precise positioning of key points for measuring the body size of cattle in 3D point cloud, thereby improving the accuracy of body size measurement. At the same time, based on multiple precisely positioned key points, it can accurately calculate various body size parameters of cattle, realizing complete parameter calculation.

[0233] The same or similar labels correspond to the same or similar parts;

[0234] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0235] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds, characterized in that, Includes the following steps: S1: Collect 3D point clouds of cattle and construct an initial point cloud dataset; S2: Input the initial point cloud dataset into the pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and annotate the key points of the training set and the validation set; The initial point cloud dataset is input into a pre-built spatial point colorization model for processing to obtain a three-view image set, including: The pre-built spatial point colorization model performs the following operations on each of the 3D point clouds of cattle in the initial point cloud dataset: Let the 3D point cloud of the cattle be denoted as point set P. The 3D spatial coordinate set of point set P is represented as: Where n is the number of points in the 3D point cloud of the cow. Represents the 3D spatial coordinates of the i-th point in a 3D point cloud; Adjust the orientation of the 3D point cloud of the cattle so that the head of the cattle is facing the same direction as the X-axis, Y-axis and Z-axis respectively, and the longitudinal and transverse directions of the cattle are parallel to the X-axis, Y-axis and Z-axis respectively. The normal vectors of the three view planes of the side view, top view and rear view of the 3D point cloud of cattle are estimated based on the local surface fitting method. Color mapping of 3D point clouds of cattle: For the side view, top view and rear view of the cattle, extract the Z-axis coordinate set of point set P respectively. Y-axis coordinate set and X-axis coordinate set Normalize the three coordinate sets respectively and map them to the interval [0,1] to obtain the three normalized coordinate sets; Based on the three normalized coordinate sets, the terrain color band in the Matplotlib data visualization library is used to perform three-dimensional color mapping on the 3D point cloud of the cow, generating 3D colored point clouds corresponding to the side view, top view and rear view of the cow respectively. The three sets of 3D color point clouds were visualized using Open3D's physical lighting engine, and the initial two-dimensional images of the cow's side view, top view, and rear view were extracted. The initial 2D images of the cow in three views are cropped and sharpened to obtain the three-view images corresponding to the 3D point cloud of the cow. Save the three-view images corresponding to the 3D point clouds of all cattle into a three-view image set; The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view. S3: Construct a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance. S4: Input the test set into the key point recognition model with the best performance and perform key point recognition to obtain the two-dimensional plane coordinates of several key points; S5: Use the preset heterogeneous key point coordinate transformation algorithm to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates; use the three-dimensional spatial coordinates of all key points to calculate the body size parameters of the cattle and complete the body size measurement of the cattle.

2. The method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds according to claim 1, characterized in that, The method for estimating the normal vector of the 3D point cloud of cattle based on local surface fitting includes: Select a point in the 3D point cloud of cattle, and use a hybrid search strategy to search for several points within a spherical range of a preset radius around the selected point, construct a local neighborhood of the selected point, and fit a plane using the local neighborhood. Construct the following problem and solve it: in, These are the first, second, third, and fourth parameters of the fitted plane, respectively. Will The solution results are saved as the normal vectors of the 3D point cloud of the cow. Specifically: in, Normal vectors 3D coordinates; They are respectively The results of solving for the positive and negative terms; This represents the transpose of a matrix.

3. The method for measuring the body size of cattle based on multi-view key point localization using colorized point clouds according to claim 1, characterized in that, For the three coordinate sets, the following normalization process is performed respectively: in, , and These are the normalized X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively. , and These are the maximum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively. , and These are the minimum values ​​of the X-axis, Y-axis, and Z-axis coordinates of the i-th point, respectively.

4. The method for measuring the body size of cattle based on multi-view key point localization using colorized point clouds according to claim 1, characterized in that, In step S2, the Labelme data annotation tool is used to annotate key points on the training set and validation set.

5. The method for measuring the body size of cattle based on multi-view key point localization using colorized point clouds according to claim 1, characterized in that, In step S3, the key point recognition model constructed based on the deep learning algorithm is specifically the YOLOv8-pose neural network model.

6. The method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds according to claim 1, characterized in that, In step S5, the preset heterogeneous key point coordinate transformation algorithm is used to convert the two-dimensional planar coordinates of all key points into three-dimensional spatial coordinates. Specifically, this is as follows: Obtain the size of the 2D image used for key point recognition in the test set; construct the AABB bounding box of the 3D point cloud of the cow corresponding to the 2D image, and obtain the coordinates of the eight vertices of the AABB bounding box; If the pixel color of each key point identified in step S3 is white, then the key point is located outside the cow's body. The two-dimensional plane coordinates of the key point are moved to the nearest color pixel for position correction. Otherwise, the key point is located inside the cow's body. Continue to detect the next key point until all key points have been detected. Based on the size of the two-dimensional image, the vertex coordinates of the AABB bounding box, and the two-dimensional planar coordinates of all the key points that have passed the detection, the approximate planar coordinates of each key point in the two coordinate axes in the three-dimensional space are calculated, and the local region of interest of each key point in the 3D point cloud of the cattle is obtained based on the approximate planar coordinates. The local region of interest contains several point cloud slices distributed along the third coordinate axis. The center point of the point cloud slice closest to the AABB bounding box interface is taken as the mapped key point, and the three-dimensional spatial coordinates of the center point are obtained and used as the three-dimensional spatial coordinates of the mapped key point.

7. The method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds according to claim 6, characterized in that, The size of the two-dimensional image is denoted as . × ,in and Let p be the length and width of the 2D image, respectively; and let (p, q) be the 2D plane coordinates of the key points after detection. Starting from the top right corner of the upper surface of the AABB bounding box, the coordinates of the four vertices of the upper surface of the AABB bounding box are denoted as follows, clockwise: , , and The vertex coordinates corresponding to the lower surface of the AABB bounding box are denoted as follows: , , and The dimensions of the AABB enclosure × × ,in, , and These are the length, width, and height of the AABB bezel box; The 3D spatial coordinates of the mapped key points are denoted as follows: ,but: If the 2D image is a side view, calculate the approximate coordinates of the keypoints (p, q) on the X and Y axes using the following formula: At this point, the acquired point cloud slices are distributed along the Z-axis. The value is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface; If the 2D image is a top view, calculate the approximate coordinates of the key points (p, q) on the X and Z axes using the following formula; At this point, the acquired point cloud slices are distributed along the Y-axis. The value is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface; If the 2D image is a rear view, calculate the approximate coordinates of the key points (p, q) in the plane on the Y and Z axes according to the following formula; At this point, the acquired point cloud slices are distributed along the X-axis. The value is determined by the coordinates of the center point of the point cloud slice closest to the AABB bounding box interface.

8. A method for measuring the body size of cattle based on multi-view keypoint localization using colorized point clouds, as described in any one of claims 1 to 7, characterized in that... In step S5, the body size parameters of the cattle include: body length, chest depth, rump height, croup length, body height, body length, abdominal circumference, cross-section height, chest circumference, chest width, waist angle width, hip width, and ischial end width.

9. A cattle body size measurement system based on multi-view keypoint localization using colorized point clouds, employing the cattle body size measurement method based on multi-view keypoint localization using colorized point clouds as described in any one of claims 1 to 8, characterized in that... include: Point cloud acquisition unit: used to acquire 3D point clouds of cattle and build an initial point cloud dataset; Dataset construction unit: used to input the initial point cloud dataset into a pre-built spatial point colorization model for processing, obtain a three-view image set and divide it into a training set, a validation set and a test set, and perform key point annotation on the training set and the validation set; The three-view image set includes several two-dimensional images of cattle in three views: side view, top view, and rear view. Model training unit: used to build a key point recognition model based on deep learning algorithms, train the key point recognition model using the training set after key point annotation, and verify the trained key point recognition model using the validation set after key point annotation to obtain the key point recognition model with the best performance. Key point recognition unit: used to perform key point recognition on the key point recognition model with the best performance of the test set, and obtain the two-dimensional plane coordinates of several key points; Body size measurement unit: Used to convert the two-dimensional plane coordinates of all key points into three-dimensional spatial coordinates using a preset heterogeneous key point coordinate transformation algorithm; calculate the body size parameters of cattle using the three-dimensional spatial coordinates of all key points, and complete the body size measurement of cattle.

Images