Livestock segmentation method based on multi-view point cloud reconstruction and segmentation network model improvement

Abstract

The invention discloses a beef cattle segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model, and the method comprises the steps: obtaining to-be-segmented livestock video data through a data collection platform, obtaining an initial RGB image and a depth image through frame segmentation, and generating a multi-view livestock point cloud image through camera parameters; performing multi-view point cloud registration on the multi-view livestock point cloud atlas by using a method of combining manual feature point labeling and a PnP-RANSAC method to obtain spliced complete point cloud data; performing denoising processing on the registered complete point cloud data, labeling a denoised point cloud image, inputting the denoised point cloud data into a segmentation network model, and outputting segmented livestock point cloud; the segmentation network model is based on a Point Net + + model, and an SGP + + position sensing module is introduced by using stratified sampling and a local feature aggregation mechanism of the Point Net + + model; and finally, body size measurement is carried out. According to the method, the space grouping enhancement module SGP and the position sensing pooling technology are introduced, so that the beef cattle point cloud segmentation precision and the body size measurement accuracy are remarkably improved.

Description

Technical Field

[0001] This invention relates to point cloud segmentation and livestock detection technology, specifically to a livestock segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model. Background Technology

[0002] In recent years, point cloud segmentation methods based on deep learning (such as PointNet, PointNet++, DGCNN, etc.) have been applied to livestock morphology analysis.

[0003] Early work relied heavily on two-dimensional images. For example, Yue Mengmeng et al. published a method for estimating the body size and weight of beef cattle based on Image 2 and ridge regression models. This method only uses two-dimensional images for prediction, so it has high accuracy in predicting two-dimensional length traits such as body height, body length, and cross height, but low accuracy in predicting three-dimensional body size traits such as chest circumference (i.e., three-dimensional spatial lengths that cannot be clearly identified in two-dimensional images).

[0004] Pezzuolo et al. used a structured light camera to collect multi-view 3D data of cattle. When measuring body size, they used the camera's shooting slope to determine the body size measurement point. However, this method is easily affected by interference from various parts of the animal's body and is also affected by the animal's posture during the acquisition, which leads to deviations in the positioning of the measurement point and affects the body size measurement results.

[0005] Yin Ling et al. proposed an automatic pig body size measurement technique based on 3D point cloud segmentation. Using PointNet++, they extracted and encoded local feature points through farthest-point sampling to segment the pig body, and then measured its body size based on the 3D point cloud segmentation. Compared to overall point cloud measurement methods, this technique effectively reduces the interference of posture on key body size positioning points, more accurately locates and measures key points within local areas, and improves the accuracy of body size calculation. However, the accuracy can be further improved by directly using PointNet++ segmentation. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a livestock segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model.

[0007] Technical solution: The present invention provides a livestock segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model, comprising the following steps:

[0008] Step 1: Build a data acquisition platform to obtain multi-view MKV format video data of livestock to be segmented; and segment the obtained video data frame by frame to obtain the original RGB image and depth map of the corresponding livestock to be segmented. Then, use camera parameters to generate multi-view livestock point cloud map.

[0009] Step 2: Use a combination of manually labeled feature points and PnP-RANSAC method to perform multi-view point cloud registration on the multi-view livestock point cloud map in Step 1 to obtain the stitched complete point cloud data.

[0010] The obtained registered complete point cloud data is denoised to filter out background and noise (e.g., using radius-based outlier removal (ROR) algorithm, RANSAC, DBSCAN and outlier removal methods to remove ground effects); here, the complete view of livestock point cloud (including noisy points) is obtained through point cloud registration, and the clean livestock point cloud is obtained by denoising.

[0011] Step 3: Label the point cloud map obtained after denoising in Step 2. Use CloudCompare software to label the eight main parts of the livestock: head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb and right hindlimb, and classify them into labels from 0 to 7.

[0012] Step 4: Construct a segmentation network model. Input the denoised point cloud data into the segmentation network model and output the segmented livestock point cloud. The segmentation network model is based on the PointNet++ model. It utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model and introduces the SGP++ position-aware module to improve the segmentation accuracy. The SGP++ position-aware module uses a lightweight attention network to dynamically learn the importance of each point in the neighborhood based on its own position and feature information, and performs weighted aggregation based on this.

[0013] Step 5: Based on the segmented livestock point cloud obtained in Step 4, measure the body size.

[0014] Furthermore, the data acquisition platform in step 1 includes a gantry, an aisle scale located below the gantry, and multiple depth cameras installed on the gantry; when images need to be acquired, the livestock to be divided are first driven to the aisle scale, and each depth camera is activated.

[0015] Each time data was collected, a 6-12 second video of a domestic animal was recorded, including its top-down view and color and depth stream videos from both sides.

[0016] Next, Open3D was used to segment the image frame by frame to obtain the RGB image and depth map of the livestock, and then combined with the camera intrinsics to obtain the multi-view point cloud map of the livestock.

[0017] During data collection, at least one depth camera is installed on the top, left, and right sides of the livestock, and the depth cameras on the left and right sides are adjustable to accommodate livestock of different sizes.

[0018] Furthermore, the specific details of step 2 in processing the multi-view livestock point cloud image are as follows:

[0019] Step 2.1: Point cloud registration. Using a checkerboard as a reference object, a registration method combining manually marked feature points and PnP-RANSAC is adopted. The rotation and translation matrices between different viewpoints are calculated to bring the multi-view point clouds from different coordinate systems to the same coordinate system, thus completing the spatial fusion of the multi-view point clouds.

[0020] Step 2.2: Point cloud denoising, i.e., using radius-based outlier removal (ROR); ground removal is achieved using RANSAC, DBSCAN, and outlier removal operations.

[0021] Further, in step 3, CloudCompare software is used to manually segment and label the parts of the livestock. During the manual labeling process, the head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb and right hindlimb of the individual livestock are selected and classified into labels number 0-7 in sequence.

[0022] Then, all the labeled image data are stored in text form according to the ShapeNet Part dataset, where each line represents the coordinates (x, y, z) and label of a single point in the point cloud; here, manual annotation ensures that each point corresponds to a label, and each line represents the coordinates and corresponding label of one point, with the number of lines equal to the number of points in the point cloud.

[0023] Furthermore, the input to the segmentation network model is the denoised livestock point cloud, and the output is the livestock point cloud with segmentation labels; the segmentation network model is based on the PointNet++ model, utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model, and introduces the SGP++ position-aware module.

[0024] The workflow of the SGP++ location awareness module is as follows:

[0025] First, let the input features be X∈R B×C×N The channel dimension is divided into G groups, and the feature X of each group is obtained. (g) :

[0026]

[0027] In the above formula, each Let X represent the feature response of the i-th point in the g-th group, where g takes values ​​from 1 to G; X∈R B×C×N Let X represent a three-dimensional tensor whose elements belong to the set of real numbers R. The three dimensions of the tensor are B, C, and N, respectively. C usually represents the number of channels. For example, in image data, RGB images have 3 channels, and grayscale images have 1 channel. N represents point cloud data.

[0028] Ideally, if the g-th group learns the features of a certain semantic part (such as the limbs of livestock), its activation should form a strong response in that region, which is manifested as: large vector magnitude and consistent direction between adjacent points (high cosine similarity). However, in actual training, due to the uneven density of point clouds, background interference, etc., the response may be unfocused and the amplitude may fluctuate greatly.

[0029] To suppress this noise, a global statistical vector is introduced (PointNet++ generates global vectors of different scales through multi-level "downsampling-grouping-feature extraction," and finally fuses them to form the global statistical vector of the overall point cloud). The formula for the global statistical vector is:

[0030] Then, normalization and learnable affine transformations are used to enhance its nonlinear representation capability:

[0031]

[0032] In the above formula, The spatial response weights are adjusted to the [0,1] interval using the Sigmoid function to control the activation intensity of each point within the group;

[0033] Secondly, the output for each point is:

[0034] Next, the overall enhancement feature X is calculated using the following formula:

[0035]

[0036] Furthermore, in the local feature aggregation stage of the point cloud, the enhanced feature response X′ is obtained through element-level weighting operations. The formula for calculating X′ is: X′=X⊙w; w is the weight.

[0037] Finally, the enhanced feature responses X′ of each channel are spatially weighted and converged to obtain the global response features. This weight can be viewed as the salience of the response of each channel at each location; The calculation formula is:

[0038] i is a control point.

[0039] In other words, the weights w obtained earlier are applied to the original features and weighted aggregation is performed to obtain... Where X' (weighted feature matrix) represents the feature matrix after weight adjustment; (Weighted aggregation result) represents the summation result of the weighted features along the last dimension (i = 1 to N); This refers to the overall enhancement feature.

[0040] Let the input point cloud features be tensors where X∈R B×N×C First, through a one-dimensional convolutional mapping with shared parameters... The significance weight of each point is learned, and the calculation formula is as follows:

[0041] Here, represents the pointwise Sigmoid activation function, used to compress the weights to the [0,1] interval. This weight matrix can be understood as the confidence (or significance score) of each channel's response at each point, reflecting its contribution to the overall semantic expression.

[0042] Subsequently, enhanced feature responses are obtained through element-level weighting operations, and global response features are obtained by spatial weighted aggregation across all channels. This weight can be viewed as the saliency of each channel's response at each location. The weighted aggregation strategy here applies the weights to the original features and performs weighted aggregation, which can extract location-sensitive semantic centers from the responses of multiple points and avoid diluting key features by low-activation regions.

[0043] Furthermore, the SGP++ position-aware module of the segmentation network model utilizes the spatial relationships of features within a group to weight and enhance key regions, weakening or applying attention suppression to features in interfering regions, and placing greater emphasis on the feature distribution of spatial location. The PointNet++ model introduces a hierarchical local feature extraction mechanism, using strategies such as Farthest PointSampling and spherical neighborhood search to extract local geometric features at different scales layer by layer, significantly enhancing the model's representation ability on unstructured point cloud data. It is particularly suitable for capturing multi-scale spatial structures, alleviating the problem of insufficient local relationship modeling ability of the original PointNet.

[0044] The body measurements include chest circumference, abdominal circumference, height, and cross-shaped height. Chest circumference, height, and cross-shaped height are all located on the chest and need to be measured in the segmented chest area, while abdominal circumference needs to be measured in the abdomen area. The detailed method is as follows: the chest point cloud is segmented in the segmentation network model, and slices are made with the maximum x value as the end point to obtain the chest circumference slices. The perimeter is calculated using a closed curve fitting method with control point constraints, which is the chest circumference.

[0045] In the segmentation network model, the abdominal point cloud is segmented and projected onto the xz plane to obtain the top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain the abdominal cross-section point cloud set. The perimeter and abdominal circumference are calculated using the closed curve fitting method with control point constraints.

[0046] Calculate the maximum y value in the chest point cloud, and calculate the distance between y and the ground y value, which is the body height;

[0047] The hip point cloud is segmented in the segmentation network model. The top view of the hip point cloud is obtained by projecting it onto the xz plane. The protruding parts on the left and right sides, i.e. the y-points with the largest absolute value of z, are the outer edges of the two waist angles. The cross-shaped part is the center point of the two waist angles. The distance from the y-value of the cross-shaped part to the ground y-value is the height of the cross-shaped part.

[0048] For the measurement process of each part of the body mentioned above, let the coordinates of the corresponding measuring points A(x1,y1,z1) and B(x2,y2,z2) be set. The length L of the straight-line body scale between the two points is calculated using the Euclidean distance formula, as follows:

[0049]

[0050] To achieve accurate measurement of body dimensions such as chest circumference and abdominal circumference in livestock, a geometric modeling method using point cloud slicing, closed curve fitting, and B-spline interpolation is employed. This method aims to address the shortcomings of traditional measurement methods in terms of robustness and accuracy under complex postures and local occlusion conditions (taking abdominal circumference as an example).

[0051] First, the abdominal point cloud is segmented in the segmentation network model and projected onto the yz plane to obtain a top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain a cross-sectional point cloud set.

[0052] Then, a closed curve fitting method with control point constraints is used. After the initial fitting, the B-spline interpolation algorithm is introduced to optimize the contour. The perimeter of the fitted curve is the livestock dimension body size.

[0053] A B-spline is a smooth curve defined over multiple control points, and its expression is as follows:

[0054]

[0055] c i N is the control point; i,k (u) is the B-spline basis function of the i-th control point of order k, and C(u) represents the coordinates of the interpolation curve under u, where u is an index, such as time or progress, used to traverse the curve.

[0056] Beneficial Effects: This invention significantly improves the segmentation accuracy of cattle point clouds and the measurement accuracy of body size by introducing the Spatial Grouping Enhancement Module (SGP) and position-aware pooling technology. The improved SGPAPNet++ model used in this invention's segmentation network model performs excellently in livestock point cloud segmentation tasks, showing significant improvements in key indicators such as overall accuracy and average intersection-over-union ratio (AUC) compared to the original PointNet++ model. The introduction of the Spatial Grouping Enhancement Module effectively enhances the model's ability to perceive local geometric features, while the position-aware pooling technology further optimizes the feature extraction process, enabling the model to more accurately identify different parts of the livestock's body.

[0057] The local feature extraction mechanism of this invention overcomes the deficiency of the original PointNet in capturing local geometric structures. It simulates the hierarchical feature extraction method of CNNs, gradually constructing hierarchical features from local to global through multi-scale local region segmentation and feature learning. The hierarchical sampling and local feature aggregation mechanism effectively enhances the modeling ability of point cloud local geometric structures. However, its neighborhood feature processing is insensitive to different spatial locations and is easily affected by redundant or noisy points, limiting the representation effect on key regions and further improving the model's spatial perception capability. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the segmentation network model structure of the present invention;

[0059] Figure 2 This is a schematic diagram of the overall segmentation process of the present invention;

[0060] Figure 3 This is a schematic diagram of the data acquisition platform structure in the embodiment;

[0061] Figure 4 This is a schematic diagram of multi-view point cloud processing of individual beef cattle in the embodiment;

[0062] Figure 5 This is a schematic diagram illustrating the standard for measuring the body size of beef cattle in the embodiment.

[0063] Figure 6 The image shows the outline of the back of a beef cattle obtained in the example.

[0064] Figure 7 This is a schematic diagram illustrating the measurement of abdominal circumference slices and ellipse fitting in an example.

[0065] Figure 8 This is a schematic diagram of the average intersection-union ratio of the model validation set in the embodiment;

[0066] Figure 9 Here is a box plot showing the relative error of the automatic measurement values ​​of the scale in the example.

[0067] Figure 10This is a diagram showing the final beef cattle cutting result obtained in the example. Detailed Implementation

[0068] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0069] This invention significantly improves the accuracy of livestock point cloud segmentation and body size measurement by introducing the Spatial Grouping Enhancement Module (SGP) and position-aware pooling technology. The improved SGPAPNet++ model in this invention performs excellently in livestock point cloud segmentation tasks, showing significant improvements in key indicators such as overall accuracy and average intersection-over-union ratio compared to the original PointNet++ model. The introduction of the Spatial Grouping Enhancement Module effectively enhances the model's ability to perceive local geometric features, while the position-aware pooling technology further optimizes the feature extraction process, enabling the model to more accurately identify different parts of the livestock body.

[0070] like Figure 1 and Figure 2 As shown, the livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model in this embodiment includes the following steps:

[0071] Step 1: Set up a data acquisition platform (e.g., Figure 3 As shown, multi-view MKV format video data of livestock to be segmented is obtained through a data acquisition platform; the obtained video data is segmented by frame to obtain the original RGB image and depth map of the corresponding livestock to be segmented, and then multi-view livestock point cloud map is generated using camera parameters.

[0072] Step 2: Use a combination of manually labeled feature points and PnP-RANSAC method to perform multi-view point cloud registration on the multi-view livestock point cloud map in Step 1 to obtain the stitched complete point cloud data.

[0073] The obtained registered point cloud data is then denoised to remove background and noise.

[0074] Step 3: Label the point cloud map obtained after denoising in Step 2. Use CloudCompare software to label eight parts of the livestock: head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb, and right hindlimb, and classify them into labels from 0 to 7.

[0075] Step 4: Construct a segmentation network model. Input the denoised point cloud data into the segmentation network model and output the segmented livestock point cloud. The segmentation network model is based on the PointNet++ model. It utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model and introduces the SGP++ position-aware module.

[0076] Step 5: Based on the segmented livestock point cloud obtained in Step 4, measure the body size.

[0077] The point cloud segmentation results of the method described in this invention successfully enabled the automatic measurement of four key body size parameters in beef cattle: cross-section height, body height, chest circumference, and abdominal circumference. The measurement accuracy was high with minimal error, validating the effectiveness of the improved model in practical applications. This method overcomes the drawbacks of traditional manual measurement, such as being time-consuming, labor-intensive, and highly subjective, providing a reliable technical means for modern ranch management. It not only advances the application of deep learning in animal husbandry at the technical level but also provides crucial data support for livestock health monitoring, growth assessment, and precision feeding management, possessing significant theoretical and practical value. Future applications can be further expanded to measure more body size parameters and other livestock species, promoting the development of smart animal husbandry.

[0078] The data acquisition platform in step 1 of this embodiment includes a gantry, an aisle scale located below the gantry, and multiple depth cameras installed on the gantry. When it is necessary to acquire images, the livestock to be divided are first driven to the aisle scale, and each depth camera is activated.

[0079] Each time data was collected, a 6-12 second video of a domestic animal was recorded, including its top-down view and color and depth stream videos from both sides.

[0080] Next, Open3D was used to segment the image frame by frame to obtain the RGB image and depth map of the livestock, and then combined with the camera intrinsics to obtain the multi-view point cloud map of the livestock.

[0081] During data collection, at least one depth camera is installed directly above, to the left and to the right of the livestock, and the depth cameras on the left and right sides are adjustable in height.

[0082] The specific details of step 2 in this embodiment regarding the processing of multi-view livestock point cloud images are as follows:

[0083] Step 2.1: Point cloud registration. Using a checkerboard as a reference object, a registration method combining manually marked feature points and PnP-RANSAC is adopted. The rotation and translation matrices between different viewpoints are calculated to bring the multi-view point clouds from different coordinate systems to the same coordinate system, thus completing the spatial fusion of the multi-view point clouds.

[0084] Step 2.2, point cloud denoising, which uses radius-based outlier removal (ROR) and employs RANSAC, DBSCAN, and outlier removal operations to remove ground effects.

[0085] In step 3 of this embodiment, CloudCompare software is used to manually segment and label the parts of the livestock. During the manual labeling process, the head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb, and right hindlimb of the individual livestock are selected and classified into 0-7 label numbers in sequence.

[0086] Then, all the labeled image data are stored in text form according to the ShapeNet Part dataset, where each line represents the (x,y,z) coordinates and label of a single point in the point cloud; here, manual annotation is used to ensure that each point corresponds to a label.

[0087] The input of the segmentation network model in this embodiment is the denoised livestock point cloud, and the output is the livestock point cloud with segmentation labels. The segmentation network model is based on the PointNet++ model, utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model, and introduces the SGP++ position-aware module.

[0088] The workflow of the SGP++ location awareness module is as follows:

[0089] First, let the input features be X∈R B×C×N The channel dimension is divided into G groups, and the feature X of each group is obtained. (g) :

[0090]

[0091] In the above formula, each Let X represent the feature response of the i-th point in the g-th group, where g takes values ​​from 1 to G; X∈R B×C×N Let X be a three-dimensional tensor whose elements belong to the set of real numbers R. The three dimensions of the tensor are B, C, and N, respectively, where C represents the number of channels, N represents the number of point clouds, and B represents the batch size.

[0092] To suppress noise in point cloud data, a global statistical vector is introduced, with the following formula:

[0093] Then, normalization and learnable affine transformations are used to enhance its nonlinear representation capability:

[0094]

[0095] In the above formula, The spatial response weights are adjusted to the [0,1] interval using the Sigmoid function to control the activation intensity of each point within the group;

[0096] Secondly, the output for each point is:

[0097] Next, the overall enhancement features are calculated. The formula is:

[0098]

[0099] Furthermore, in the local feature aggregation stage of the point cloud, the enhanced feature response X′ is obtained through element-level weighting operations. The formula for calculating X′ is: X′=X⊙w; w is the weight.

[0100] Finally, the enhanced feature responses X′ of each channel are spatially weighted and converged to obtain the global response features. This weight can be viewed as the salience of the response of each channel at each location; The calculation formula is:

[0101] i is a control point.

[0102] This embodiment measures the body size of livestock based on the segmented point cloud. The detailed method is as follows: Assume that the head of the livestock point cloud faces the positive x-axis, the back faces the positive y-axis, and the left side of the body faces the positive z-axis, that is, the head faces the right side when standing. The chest point cloud is segmented in the segmentation network model, and slices are made with the maximum x value as the end point to obtain the chest circumference slice. The perimeter is calculated using the closed curve fitting method with control point constraints, which is the chest circumference.

[0103] In the segmentation network model, the abdominal point cloud is segmented and projected onto the xz plane to obtain the top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain the abdominal cross-section point cloud set. The perimeter and abdominal circumference are calculated using the closed curve fitting method with control point constraints.

[0104] Calculate the maximum y value in the chest point cloud, and calculate the distance between y and the ground y value, which is the body height;

[0105] The hip point cloud is segmented in the segmentation network model. The top view of the hip point cloud is obtained by projecting it onto the xz plane. The protruding parts on the left and right sides, i.e. the y-points with the largest absolute value of z, are the outer edges of the two waist angles. The cross-shaped part is the center point of the two waist angles. The distance from the y-value of the cross-shaped part to the ground y-value is the height of the cross-shaped part.

[0106] For the body size measurement process of each part mentioned above, let the coordinates of the corresponding body size measurement points A(x1,y1,z1) and B(x2,y2,z2) be set, and the length of the straight body size between the two points is calculated by the Euclidean distance formula; in order to achieve accurate measurement of the body size of livestock such as chest circumference and abdominal circumference, a geometric modeling method of point cloud slicing, closed curve fitting and B spline interpolation is used.

[0107] First, the abdominal point cloud is segmented in the segmentation network model and projected onto the yz plane to obtain a top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain a cross-sectional point cloud set.

[0108] Then, a closed curve fitting method with control point constraints is used to optimize the contour after initial fitting, followed by B-spline interpolation. The perimeter of the fitted curve is the livestock's body size. A B-spline is a smooth curve defined on multiple control points, and its expression is as follows:

[0109]

[0110] c i N is the control point; i,k (u) is the B-spline basis function of the i-th control point with order k, and C(u) represents the coordinates of the interpolation curve under index u.

[0111] This embodiment presents a livestock segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model. The method is characterized by the following: Step 1's data acquisition platform includes a gantry 8, a walkway scale 6 located below the gantry 8, and multiple depth cameras mounted on the gantry 8. When image acquisition is required, the livestock to be segmented is first driven to the walkway scale 6, and each depth camera is activated. To prevent the livestock from escaping, restraint fences 7 are installed around the walkway scale 6, and the livestock's identity information is read using ear tag readers 1. During each acquisition, a 6-12 second video is captured of one livestock, recording its top-down view and color and depth stream videos from both sides. Then, Open3D is used to segment the images frame by frame to obtain the livestock's RGB image and depth map. Finally, the multi-view livestock point cloud map is obtained by combining camera intrinsic parameters. During acquisition, at least one depth camera is installed directly above, to the left, and to the right of the livestock, and the depth cameras on the left and right sides are height-adjustable.

[0112] like Figure 3 As shown, the data acquisition platform in this embodiment includes three depth cameras (Azure Kinect DK) to acquire 3D structural information of livestock from different angles: left-side, right-side, and top-down views. The cameras are fixedly mounted on the gantry 8. To accommodate livestock of different sizes, the right-side camera 2 and left-side camera 3 are height-adjustable (controlled by the industrial computer 5), facing the middle of the livestock's body. The top-down camera is mounted in the middle of the crossbeam, vertically pointing downwards at the back of the livestock. The relative positions of each camera are pre-calibrated to ensure good overlap and spatial consistency of the multi-view point clouds during subsequent registration and reconstruction. During each acquisition, a 12-second video is taken of one livestock, recording color and depth streams from both top-down and side views. Then, Open3D is used to segment the images frame by frame to obtain the livestock's color and depth maps. Figures 4 to 6 As shown, the specific details of step 2 in this embodiment regarding the processing of multi-view livestock point cloud images are as follows:

[0113] Step 2.1: Point cloud registration. Using a checkerboard as a reference object, a registration method combining manually marked feature points and PnP-RANSAC is adopted. The rotation and translation matrices between different viewpoints are calculated to bring the multi-view point clouds from different coordinate systems to the same coordinate system, thus completing the spatial fusion of the multi-view point clouds.

[0114] Step 2.2, point cloud denoising, which uses radius-based outlier removal (ROR) and employs RANSAC, DBSCAN, and outlier removal operations to remove ground effects.

[0115] In step 3 of this embodiment, CloudCompare software is used to manually segment and label the parts of the livestock. During the manual labeling process, the head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb, and right hindlimb of the individual livestock are selected and classified into labels numbered 0-7. Then, all the labeled image data are stored in text form according to the ShapeNet Part dataset, where each line represents the coordinates (x, y, z) of a single point in the point cloud and its label. Here, manual labeling ensures that each point corresponds to a label.

[0116] The input of the segmentation network model in this embodiment is the denoised livestock point cloud, and the output is the livestock point cloud with segmentation labels. The segmentation network model is based on the PointNet++ model, utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model, and introduces the SGP++ position-aware module.

[0117] The workflow of the SGP++ location awareness module is as follows:

[0118] First, let the input features be X∈R B×C×N The channel dimension is divided into G groups, and the feature X of each group is obtained. (g) :

[0119]

[0120] In the above formula, each Let X represent the feature response of the i-th point in the g-th group, where g takes values ​​from 1 to G; X∈R B×C×N Let X be a three-dimensional tensor whose elements belong to the set of real numbers R. The three dimensions of the tensor are B, C, and N, respectively, where C represents the number of channels, N represents the number of point clouds, and B represents the batch size.

[0121] To suppress noise in point cloud data, a global statistical vector is introduced, with the following formula:

[0122] Then, normalization and learnable affine transformations are used to enhance its nonlinear representation capability:

[0123]

[0124] In the above formula, The spatial response weights are adjusted to the [0,1] interval using the Sigmoid function to control the activation intensity of each point within the group;

[0125] Secondly, the output for each point is:

[0126] Next, the overall enhancement features are calculated. The formula is:

[0127] Then, in the local feature aggregation stage of the point cloud, the enhanced feature response X′ is obtained through element-level weighting operations. The formula for calculating X′ is: X′=X⊙w;

[0128] Finally, the enhanced feature responses X′ of each channel are spatially weighted and converged to obtain the global response features. This weight can be viewed as the salience of the response of each channel at each location; The calculation formula is:

[0129]

[0130] like Figure 6 and Figure 7 As shown, this embodiment measures the body size of eight parts sequentially based on the segmented livestock point cloud. The detailed method is as follows: the chest point cloud is segmented in the segmentation network model, and slices are made with the maximum x value as the end point to obtain the chest circumference slices. The perimeter is calculated using the closed curve fitting method with control point constraints, which is the chest circumference.

[0131] In the segmentation network model, the abdominal point cloud is segmented and projected onto the xy plane to obtain the top view of the abdomen. The x value corresponding to the largest absolute value of y on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain the abdominal cross-section point cloud set. The perimeter and abdominal circumference are calculated using the closed curve fitting method with control point constraints.

[0132] Calculate the maximum y value in the chest point cloud, and calculate the distance between y and the ground y value, which is the body height;

[0133] The buttock point cloud is segmented in the segmentation network model. The top view of the buttock point cloud is obtained by projecting it onto the xy plane. The y-value with the largest absolute value on the left and right sides is the outer edge of the two waist angles. The cross is the center point of the two waist angles. The distance from the y-value of the cross point to the ground y is the height of the cross.

[0134] For the measurement process of each part of the body mentioned above, let the coordinates of the corresponding measuring points A(x1,y1,z1) and B(x2,y2,z2) be set. The length L of the straight-line body scale between the two points is calculated using the Euclidean distance formula, as follows:

[0135]

[0136] To achieve accurate measurement of body dimensions such as chest circumference and abdominal circumference in livestock, a geometric modeling method using point cloud slicing, closed curve fitting, and B-spline interpolation was employed.

[0137] First, the abdominal point cloud is segmented in the segmentation network model and projected onto the xy plane to obtain a top view of the abdomen. The x value corresponding to the largest absolute value of y on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain a cross-sectional point cloud set.

[0138] Then, a closed curve fitting method with control point constraints is used to optimize the contour after initial fitting, followed by B-spline interpolation. The perimeter of the fitted curve is the livestock's body size. A B-spline is a smooth curve defined on multiple control points, and its expression is as follows:

[0139]

[0140] c i N is the control point; i,k (u) represents the B-spline basis function of the i-th control point with order k, and C(u) represents the coordinates of the interpolation curve under parameter u.

[0141] To verify the technical effect and feasibility of the present invention, point cloud data of beef cattle was first obtained. The relevant data came from a beef cattle farm and was collected in August and November 2024. A total of 82 three-dimensional point cloud data of 72 beef cattle were collected. The dataset was divided into training set, test set and validation set in a ratio of 7:2:1. 62 data were used as training set, 11 as validation set and 9 as test set.

[0142] This embodiment further employs the technical solution of the present invention to conduct a complete training, testing, and verification experiment on the proposed model using the point cloud data of beef cattle obtained in the aforementioned data acquisition stage. The segmentation performance is quantitatively evaluated and statistically analyzed using two indicators: overall accuracy (OA) and mean intersection over union (MIoU). Figure 8 As shown in Tables 1 and 2.

[0143] Table 1 shows the overall performance of beef cattle segmentation, and Table 2 presents the statistical results of the average crossover ratio of segmentation of each body part.

[0144] The final body size measurement result in this embodiment is as follows: Figure 10 As shown, this study demonstrates the measurement accuracy of key body size parameters of cattle. Mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and standard deviation (STD) were selected as evaluation indicators. Error analysis was performed on four key indicators: chest circumference, abdominal circumference, body height, and cross-shaped height, and box plots were generated. Figure 9 As shown. Among them, Figure 10 Figures (a) to (d) are schematic diagrams showing the carcass results of four beef cattle with different identification marks and body sizes.

[0145] Table 1. Segmentation structure of different algorithms on the beef cattle dataset.

[0146]

[0147] Table 2. Results of Errors in Body Size Parameters

[0148] Evaluation indicators MAE(cm) RMSE (cm) MAPE (%) STD(cm) chest circumference 1.78 2.19 1.58％ 2.2 Waist circumference 2.63 3.34 2.35％ 3.37 body height 2.89 3.42 1.54％ 3.13 Cross-shaped high 2.45 3.1 1.42％ 2.95

[0149] During the training of the proposed model for 200 epochs, this embodiment recorded the trends of the model's accuracy, mean intersection-over-union ratio (mIoU), and loss value as a function of training epochs.

[0150] Experimental results show that in the early stages of training, the existing model exhibits a rapid increase in both accuracy and mean intersection-over-union (MIU), while the loss value decreases rapidly. Furthermore, the rate of accuracy improvement is more significant than that of the MIU. Around the 50th epoch, the model's accuracy exceeds 90%, and the loss value drops below 0.5; by the 100th epoch, the MIU also surpasses 90%. As training continues, the model gradually converges between the 50th and 200th epochs, with the accuracy eventually stabilizing at 93%, the MIU reaching 86.7%, and the loss value continuously decreasing and approaching 0.

[0151] The segmentation network model of this invention converges faster on the same dataset, with smaller fluctuations in various indicators during training. It can accurately identify target feature parts in the training and validation sets, demonstrating good generalization ability and robustness.

[0152] The analysis revealed differences in measurement accuracy across the four items. Chest circumference showed the most concentrated relative error distribution, with most data remaining within +2%, demonstrating the best consistency. Body height also exhibited relatively stable relative errors, mostly concentrated between -2% and +4%. In contrast, cross-section height showed the greatest fluctuation in relative error, ranging from -3.5% to +5.5%, making it the least stable measurement among the four. The median error for waist circumference was approximately +1.5%, slightly higher than the other items.

[0153] In body size measurement applications, the effectiveness of this embodiment was verified through comparative analysis. Related studies have shown that traditional holistic measurement strategies based on complete livestock point clouds have significant limitations, and their measurement accuracy is often constrained by the complexity of animal body size. When existing holistic measurement algorithms are applied to the 3D point cloud data of cattle collected in this study, they demonstrate a significant improvement in accuracy.

[0154] Experimental results show that the average intersection-union ratio (IUU) of this embodiment reaches 87.39% on the beef cattle dataset, which is 2.6% higher than the average of existing methods. The SGPAPNet++ network outperforms some mainstream segmentation networks (such as PointNet, PointNet++, and PointMLP). This invention can accurately segment different parts of beef cattle and improve the accuracy of non-contact body size measurement.

Claims

1. A livestock segmentation method based on multi-view point cloud reconstruction and an improved segmentation network model, characterized in that, Includes the following steps: Step 1: Build a data acquisition platform to obtain multi-view MKV format video data of livestock to be segmented; and segment the obtained video data frame by frame to obtain the original RGB image and depth map of the corresponding livestock to be segmented. Then, use camera parameters to generate multi-view livestock point cloud map. Step 2: Use a combination of manually labeled feature points and PnP-RANSAC method to perform multi-view point cloud registration on the multi-view livestock point cloud map in Step 1 to obtain the stitched complete point cloud data. The obtained registered point cloud data is then denoised to remove background and noise. Step 3: Label the point cloud map obtained after denoising in Step 2. Use CloudCompare software to label eight parts of the livestock: head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb, and right hindlimb, and classify them into labels from 0 to 7. Step 4: Construct a segmentation network model. Input the denoised point cloud data into the segmentation network model and output the segmented livestock point cloud. The segmentation network model is based on the PointNet++ model. It utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model and introduces the SGP++ position-aware module. Step 5: Based on the segmented livestock point cloud obtained in Step 4, measure the body size.

2. The livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model according to claim 1, characterized in that, The data acquisition platform in step 1 includes a gantry, an aisle scale located below the gantry, and multiple depth cameras mounted on the gantry. When images need to be acquired, the livestock to be divided are first driven to the aisle scale, and each depth camera is activated. Each time data was collected, a 6-12 second video of a domestic animal was recorded, including its top-down view and color and depth stream videos from both sides. Next, Open3D was used to segment the image frame by frame to obtain the RGB image and depth map of the livestock, and then combined with the camera intrinsics to obtain the multi-view point cloud map of the livestock. During data collection, at least one depth camera is installed directly above, to the left and to the right of the livestock, and the depth cameras on the left and right sides are adjustable in height.

3. The livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model according to claim 1, characterized in that, Step 2 involves the following steps for processing the multi-view livestock point cloud images: Step 2.1: Point cloud registration. Using a checkerboard as a reference object, a registration method combining manually marked feature points and PnP-RANSAC is adopted. The rotation and translation matrices between different viewpoints are calculated to bring the multi-view point clouds from different coordinate systems to the same coordinate system, thus completing the spatial fusion of the multi-view point clouds. Step 2.2, point cloud denoising, which uses radius-based outlier removal (ROR) and employs RANSAC, DBSCAN, and outlier removal operations to remove ground effects.

4. The livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model according to claim 1, characterized in that, Step 3: Use CloudCompare software to manually segment and label the parts of the livestock. During the manual labeling process, select the head, chest, abdomen, rump, left forelimb, right forelimb, left hindlimb, and right hindlimb of the individual livestock and classify them into labels number 0-7. Then, all the labeled image data are stored in text form according to the ShapeNet Part dataset, where each line represents the (x,y,z) coordinates and label of a single point in the point cloud; here, manual annotation is used to ensure that each point corresponds to a label.

5. The livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model according to claim 1, characterized in that, The input to the segmentation network model is the denoised livestock point cloud, and the output is the livestock point cloud with segmentation labels. The segmentation network model is based on the PointNet++ model, utilizes the hierarchical sampling and local feature aggregation mechanism of the PointNet++ model, and introduces the SGP++ position-aware module. The workflow of the SGP++ location awareness module is as follows: First, let the input features be X∈R B×C×N The channel dimension is divided into G groups, and the feature X of each group is obtained. (g) : In the above formula, each Let X represent the feature response of the i-th point in the g-th group, where g takes values ​​from 1 to G; X∈R B×C×N Let X be a three-dimensional tensor whose elements belong to the set of real numbers R. The three dimensions of the tensor are B, C, and N, respectively, where C represents the number of channels, N represents the number of point clouds, and B represents the batch size. To suppress noise in point cloud data, a global statistical vector is introduced, with the following formula: Then, normalization and learnable affine transformations are used to enhance its nonlinear representation capability: In the above formula, The spatial response weights are adjusted to the [0,1] interval using the Sigmoid function to control the activation intensity of each point within the group; Secondly, the output for each point is: Next, the overall enhancement features are calculated. The formula is: Furthermore, in the local feature aggregation stage of the point cloud, the enhanced feature response X is obtained through element-level weighting operations. ′ X ′ The calculation formula is: X ′ =X⊙w; w is the weight; Finally, the enhanced characteristic response X of each channel is obtained. ′ Spatial weighted convergence is performed to obtain global response features. This weight can be viewed as the salience of the response of each channel at each location; The calculation formula is: i is a control point.

6. The livestock segmentation method based on multi-view point cloud reconstruction and improved segmentation network model according to claim 1, characterized in that, Based on the segmented livestock point cloud, body size measurements were performed at corresponding locations. The detailed method is as follows: First, assume that the livestock point cloud has its head facing the positive x-axis, its back facing the positive y-axis, and its left side facing the positive z-axis, meaning it is standing with its head facing the right side. The chest point cloud is segmented in the segmentation network model, and slices are made with the maximum x value as the end point to obtain the chest circumference slice. The perimeter is calculated using the closed curve fitting method with control point constraints, which is the chest circumference. In the segmentation network model, the abdominal point cloud is segmented and projected onto the xz plane to obtain the top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain the abdominal cross-section point cloud set. The perimeter and abdominal circumference are calculated using the closed curve fitting method with control point constraints. Calculate the maximum y value in the chest point cloud, and calculate the distance between y and the ground y value, which is the body height; The hip point cloud is segmented in the segmentation network model. The top view of the hip point cloud is obtained by projecting it onto the xz plane. The protruding parts on the left and right sides, i.e. the y-points with the largest absolute value of z, are the outer edges of the two waist angles. The cross-shaped part is the center point of the two waist angles. The distance from the y-value of the cross-shaped part to the ground y-value is the height of the cross-shaped part. For the body size measurement process of each part mentioned above, let the coordinates of the corresponding body size measurement points A(x1,y1,z1) and B(x2,y2,z2) be set, and the length of the straight body size between the two points is calculated by the Euclidean distance formula; in order to achieve accurate measurement of the body size of livestock such as chest circumference and abdominal circumference, a geometric modeling method of point cloud slicing, closed curve fitting and B spline interpolation is used. First, the abdominal point cloud is segmented in the segmentation network model and projected onto the yz plane to obtain a top view of the abdomen. The x value corresponding to the largest absolute z value on the left and right sides of the livestock is found. The points are sliced ​​according to the x coordinate to obtain a cross-sectional point cloud set. Then, a closed curve fitting method with control point constraints is used to optimize the contour after initial fitting, followed by B-spline interpolation. The perimeter of the fitted curve is the livestock's body size. A B-spline is a smooth curve defined on multiple control points, and its expression is as follows: c i N is the control point; i,k (u) is the B-spline basis function of the i-th control point with order k, and C(u) represents the coordinates of the interpolation curve under index u.

Images