A three-dimensional reconstruction method for sheep body

Abstract

The present application provides a three-dimensional reconstruction method for a sheep body, which relates to the technical field of three-dimensional point cloud segmentation and registration. The method includes: obtaining a three-dimensional reconstruction diagram of the sheep body based on a trained background segmentation model and a registration model. The background segmentation model is obtained based on training samples, and the training samples are: obtaining sample point cloud data, subsampling the sample point cloud data by using a non-replacement random sampling method to obtain sampling points, and the sample point cloud data is obtained by filtering the initial sample point cloud data by using four set filtering methods simultaneously; for each sampling point, determining a local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the point pair feature; and obtaining the training samples according to the initial sample point cloud data and the local feature vector of each sampling point. In the present application, traditional algorithms are used for data annotation to reduce the amount of manual annotation, and deep learning algorithms are applied for training the model, thereby improving the generalization ability of the model.

Description

Technical Field

[0001] This application relates to the technical field of three-dimensional point cloud segmentation and registration. Specifically, this application relates to a method for three-dimensional reconstruction of sheep bodies. Background Art

[0002] In computer vision, three-dimensional reconstruction refers to the process of reconstructing three-dimensional information based on single-view or multi-view images. Compared with two-dimensional images, three-dimensional point clouds are less sensitive to color, texture, and light. Therefore, the universality of three-dimensional point cloud methods is better than that of two-dimensional. However, during the data acquisition process, the environment is usually complex, and the point cloud contains a large number of data points in non-target areas. Therefore, in order to obtain a relatively clean object for detection, different methods need to be applied to the original point cloud according to the acquisition characteristics, such as direct filtering, k-nearest neighbor algorithm, and other conditional filtering methods, and repeated filtering is carried out multiple times. In this process, since depth cameras with multiple different viewpoints are usually used when acquiring the object for detection, the data acquisition of the object for detection is completed. Therefore, it is necessary to register the point cloud data from different viewpoints after filtering. For this purpose, algorithms such as ICP, 4PCS (4-Points Congruent Sets) + ICP are applied to three-dimensional registration.

[0003] However, the above existing filtering and registration methods are all based on traditional algorithms. In terms of filtering, the robustness of traditional algorithms is insufficient. When the variety or equipment of the acquisition changes, it is necessary to find a new filtering algorithm according to the characteristics of the acquired data. In terms of registration, that is, three-dimensional reconstruction, traditional algorithms are sensitive to the position of the initial point cloud or the order of the point set, and are prone to falling into local optimal solutions, thus failing to achieve a good registration effect. Summary of the Invention

[0004] The purpose of this application aims to solve at least one of the above technical defects.

[0005] On the one hand, a method for three-dimensional reconstruction of sheep bodies is provided. The method includes:

[0006] Obtain the original point cloud data of the target sheep body and the background segmentation model. Based on the background segmentation model, segment the background data in the original point cloud data to obtain the sheep body point cloud data, which includes the left-view point cloud data, right-view point cloud data, and top-view point cloud data of the target sheep body;

[0007] Input the sheep body point cloud data into the registration model to obtain the upper-left-view point cloud data and upper-right-view point cloud data of the target sheep body, and merge the upper-left-view point cloud data and upper-right-view point cloud data to obtain the three-dimensional reconstruction diagram of the target sheep body;

[0008] Among them, the background segmentation model is trained based on at least one training sample, and each training sample is obtained through the following method:

[0009] Obtain sample point cloud data, and perform at least one downsampling on the sample point cloud data using the random sampling method without replacement to obtain at least one sampling point. The sample point cloud data is obtained by filtering the initial sample point cloud data using four set filtering methods simultaneously. The initial sample point cloud data includes the initial sample point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0010] For each sampling point, determine the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the point pair feature;

[0011] Obtain the training sample according to the initial sample point cloud data and the local feature vector of each sampling point.

[0012] Optionally, obtaining the sample point cloud data includes:

[0013] Obtain the initial sample point cloud data, and filter the initial sample point cloud data of the three views in the initial sample point cloud data using four set filtering methods simultaneously to obtain the target initial sample point cloud data;

[0014] Obtain the sample background point cloud data according to the initial sample point cloud data and the target initial sample point cloud data;

[0015] Mark the last column of the target initial sample point cloud data and the sample background point cloud data with different identifiers respectively, and merge the marked target initial sample point cloud data and the sample background point cloud data to obtain the sample point cloud data.

[0016] Optionally, the four set filtering methods include direct filtering, region growing method, radius filtering, and DBSCAN.

[0017] Optionally, determining the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the PPF feature includes:

[0018] Determine at least one sampling point group corresponding to the sampling point according to the set radius. The sampling point group includes the neighborhood data points within the spherical region determined by the sampling point with the set radius;

[0019] For each sampling point group, obtain the geometric feature corresponding to the sampling point group through multi-layer perception and max-pooling encoding operations according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point within the sampling point group, and the PPF feature;

[0020] Fuse the geometric features of at least one group of sampling points according to the set radius to obtain the local feature vector of the sampling points.

[0021] Optionally, the PPF features include the angle between the line connecting the sampling point and each neighborhood point, the angle between the normal vector of each neighborhood point and the line connecting the sampling points, the angle between the normal vectors of the two points of the sampling point and each neighborhood point, and the Euclidean distance between the sampling point and each neighborhood point.

[0022] Optionally, according to the initial sample point cloud data and the local feature vector of each sampling point, training samples are obtained, including:

[0023] Fuse the local feature vector of each sampling point into the initial sample point cloud data by upsampling to obtain the corresponding global feature vector;

[0024] Concatenate the global feature vector and the local feature vector of each sampling point to obtain the training samples.

[0025] Optionally, the background segmentation model is trained in the following way:

[0026] Obtain at least one training sample, the labeled classification result corresponding to each data point in each training sample, and the MLP model. The labeled classification result is the sheep body data point identifier or the background data point identifier;

[0027] Train the MLP model based on each training sample until the loss function corresponding to the MLP model converges, and use the MLP model at the end of training as the background segmentation model;

[0028] Among them, the input of the MLP model is the training sample, the output is the classification result of the training sample, the loss function is the cross-entropy loss function, and the value of the loss function is used to characterize the difference between the classification results of each training sample output by the model and the labeled classification results corresponding to each training sample.

[0029] Optionally, the registration model is trained by at least one training data set, and the training data set is obtained in the following way:

[0030] Obtain the registration parameters and the source point cloud data set. The source point cloud data set includes the initial training point cloud data of three views of the sample sheep body. The three views include the left view, the right view, and the top view;

[0031] Coarsely register the initial training point cloud data of the three views according to the registration parameters to obtain the training point cloud data after coarse registration of the three views;

[0032] Perform fine registration on the training point cloud data after rough registration of the three views according to the ICP algorithm to obtain a reference point cloud data set, which includes the training data of the upper left view, the training data of the upper right view, and the training data of the upper left and right views;

[0033] Use the source point cloud data set and the reference point cloud data set as the training data set.

[0034] Optionally, the registration model is trained in the following way:

[0035] Obtain the training data set and the initial rigid transformation matrix, and transform the source point cloud data set in the training data set according to the initial rigid transformation matrix to obtain a transformed source point cloud data set;

[0036] Determine the corresponding permutation matrix according to the transformed source point cloud data set and the reference point cloud data set in the training data set;

[0037] For each data point in the source point cloud data set, determine the weight of the data point according to the three-dimensional coordinates of the corresponding data point in the reference point cloud data set;

[0038] Obtain the covariance matrix according to the weights of each data point in the reference point cloud data set, and perform singular value decomposition on the covariance matrix with respect to the permutation matrix to obtain the rigid transformation matrix;

[0039] Use the rigid transformation matrix as the initial rigid transformation matrix for iterative training until the set number of times, and use the trained rigid transformation matrix as the registration model.

[0040] Optionally, determining the corresponding permutation matrix according to the transformed source point cloud data set and the reference point cloud data set in the training data set includes:

[0041] Extract features from the transformed source point cloud data set and the reference point cloud data set in the training data set, and aggregate and normalize the extracted features to obtain feature vectors;

[0042] Obtain the corresponding annealing parameter and outlier parameter according to the transformed source point cloud data set and the reference point cloud data set;

[0043] Obtain a matching matrix according to the transformed source point cloud data set, the reference point cloud data set, the annealing parameter and the outlier parameter, and perform normalization processing on the matching matrix to obtain the permutation matrix.

[0044] On the other hand, a device for three-dimensional reconstruction of a sheep body is provided, and the device includes:

[0045] A segmentation module, configured to obtain the original point cloud data of the target sheep body and a background segmentation model, and based on the background segmentation model, segment the background data in the original point cloud data to obtain the sheep body point cloud data, where the sheep body point cloud data includes the left view point cloud data, the right view point cloud data, and the top view point cloud data of the target sheep body;

[0046] A registration module, configured to input the sheep body point cloud data into a registration model to obtain the upper left view point cloud data and the upper right view point cloud data of the target sheep body, and merge the upper left view point cloud data and the upper right view point cloud data to obtain a three-dimensional reconstruction diagram of the target sheep body;

[0047] A training module, configured to train a background segmentation model based on at least one training sample, and each training sample is obtained through the following method:

[0048] Obtain sample point cloud data, and perform at least one downsampling on the sample point cloud data by using a random sampling method without replacement to obtain at least one sampling point. The sample point cloud data is obtained by filtering the initial sample point cloud data by using four preset filtering methods respectively at the same time. The initial sample point cloud data includes the initial sample point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0049] For each sampling point, determine the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the point pair feature;

[0050] Obtain a training sample according to the initial sample point cloud data and the local feature vector of each sampling point.

[0051] The beneficial effects brought by the technical solution provided by the embodiments of the present application are:

[0052] In the embodiment of the present application, the sample point cloud data for the background segmentation model is obtained after four types of filtering processes. At this time, the sample point cloud data obtained after filtering will include ten data representing sample characteristics, and the data quality will be greatly improved and the robustness will be enhanced, so that the subsequent model training will be more accurate. Further, the sample point cloud data will be downsampled at least once using the random sampling method without replacement for the sample point cloud data, that is, each data point can only be selected once. In this way, while improving the sampling efficiency, not only is the implementation method simple, but also the diversity and uniformity of sampling can be ensured. At the same time, PPF features are added to the training samples, enhancing the expression ability of the local features of the sampling points. And because the PPF features combine the ideas of global modeling and local matching, even when the sheep body data is in a state of noise, clutter, and partial occlusion, high recognition performance can still be shown. Moreover, traditional algorithms are used for data preprocessing in both the segmentation and registration stages for data annotation, reducing the workload of manual annotation. At the same time, deep learning algorithms are applied to train the segmentation and registration models, improving the generalization ability of the models and enabling them to quickly adapt to new scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for description in the embodiments of the present application.

[0054] Figure 1 Schematic flowchart of a three-dimensional reconstruction method for a sheep body provided by an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the original point cloud data of a target sheep body provided by an embodiment of the present application;

[0056] Figure 3 Schematic diagram of a three-dimensional reconstruction diagram of a sheep body provided by an embodiment of the present application;

[0057] Figure 4 Schematic diagram of the PPF feature provided by an embodiment of the present application;

[0058] Figure 5 Schematic diagram of the complete steps of the three-dimensional reconstruction method for a sheep body provided by an embodiment of the present application;

[0059] Figure 6 Schematic diagram of the classified sheep body and background point cloud data provided by an embodiment of the present application;

[0060] Figure 7 Schematic diagram of a point cloud registration provided by an embodiment of the present application;

[0061] Figure 8 Schematic diagram of the structure of a device for three-dimensional reconstruction of a sheep body provided by an embodiment of the present application;

[0062] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0063] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present invention.

[0064] Those skilled in the art of the present technology can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0065] To make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] An embodiment of the present application provides a three-dimensional reconstruction method for a sheep body, as Figure 1 shown, the method includes:

[0067] Step S101, obtaining the original point cloud data of the target sheep body and a background segmentation model, and based on the background segmentation model, segmenting the background data in the original point cloud data to obtain sheep body point cloud data, where the sheep body point cloud data includes the left view point cloud data, the right view point cloud data and the top view point cloud data of the target sheep body.

[0068] Optionally, the target sheep body refers to a sample that needs to be three-dimensionally reconstructed. The present application does not limit the method for obtaining the original point cloud data of the target sheep, and it can be obtained, for example, by taking pictures with a depth camera. Specifically, as Figure 2As shown, at this time, the original point cloud data obtained based on the depth camera (i.e., the original point cloud image in the figure) will also include other background point cloud data in addition to the point cloud data of the sheep body. At this time, the obtained original point cloud data can be input into the trained background segmentation model, which can determine whether each data point in the original point cloud data is a background data point or a target sheep body data point. Further, the background point cloud data can be removed to obtain the sheep body point cloud data, which includes the left view point cloud data, the right view point cloud data, and the top view point cloud data of the target sheep body.

[0069] Step S102: Input the sheep body point cloud data into the registration model to obtain the upper left view point cloud data and the upper right view point cloud data of the target sheep body, and merge the upper left view point cloud data and the upper right view point cloud data to obtain the three-dimensional reconstruction diagram of the target sheep body.

[0070] Optionally, as Figure 3 shown, after obtaining the sheep body point cloud data, the sheep body point cloud data can be input into the registration model, which will register based on the left view point cloud data and the top view point cloud data in the sheep body point cloud data to obtain the upper left view point cloud data (i.e., Figure 3 the left-top partial point cloud in (b)), and register based on the right view point cloud data and the top view point cloud data in the sheep body point cloud data to obtain the upper right view point cloud data (i.e., Figure 3 the right-top partial point cloud in (c)), and then merge the upper left view point cloud data and the upper right view point cloud data to obtain the three-dimensional reconstruction diagram of the target sheep body (i.e., Figure 3 the overall point cloud in (a)).

[0071] Among them, the above background segmentation model is trained based on at least one training sample, and each training sample is obtained through the following method:

[0072] Obtain sample point cloud data, and use the non-replacement random sampling method to perform at least one downsampling on the sample point cloud data to obtain at least one sampling point. The sample point cloud data is obtained by filtering the initial sample point cloud data using four preset filtering methods at the same time. The initial sample point cloud data includes the initial sample point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0073] For each sampling point, determine the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighboring data point, and the Point Pair Feature (PPF feature for short);

[0074] Obtain the training sample according to the initial sample point cloud data and the local feature vector of each sampling point.

[0075] Optionally, initial sample point cloud data including left-view initial sample point cloud data, right-view initial sample point cloud data, and top-view initial sample point cloud data of a sample sheep body can be obtained, and then the left-view initial sample point cloud data, right-view initial sample point cloud data, and top-view initial sample point cloud data are simultaneously filtered using four preset filtering methods respectively to obtain sample point cloud data. In the embodiments of the present application, the four preset filtering methods may include direct filtering, region growing method, radius filtering, and DBSCAN. At this time, the sample point cloud data obtained after filtering will include ten data representing sample features, and the data quality will be greatly improved and the robustness will be enhanced, so that subsequent model training will be more accurate.

[0076] In an alternative embodiment of the present application, obtaining the sample point cloud data may include:

[0077] Obtaining the initial sample point cloud data, and simultaneously filtering the initial sample point cloud data of the three views in the initial sample point cloud data using four preset filtering methods respectively to obtain target initial sample point cloud data;

[0078] Obtaining sample background point cloud data based on the initial sample point cloud data and the target initial sample point cloud data;

[0079] Identifying the last column of the target initial sample point cloud data and the sample background point cloud data respectively using different identifiers, and merging the identified target initial sample point cloud data and sample background point cloud data to obtain sample point cloud data.

[0080] Optionally, after the initial sample point cloud data obtained by a depth camera is simultaneously filtered using direct filtering, region growing method, radius filtering, and DBSCAN respectively, target initial sample point cloud data can be obtained. At this time, the target initial sample point cloud data can be sent into the initial sample point cloud data, and its corresponding sample background point cloud data can be obtained; further, since the obtained target initial sample point cloud data and sample background point cloud data are files in a table-like form, the last column of the target initial sample point cloud data and the sample background point cloud data can be identified at this time. For example, the last column of the target initial sample point cloud data is identified as 1, and the last column of the sample background point cloud data is identified as 0. Then, the data of the last column of the identified target initial sample point cloud data and sample background point cloud data are merged into a new point cloud file. At this time, the obtained new point cloud file is a sample point cloud data.

[0081] Furthermore, downsampling can be performed on each sample point cloud data. In this application, the farthest sampling method is modified to a random sampling method without replacement for downsampling, that is, each data point can only be selected once. In this way, while improving the sampling efficiency, the implementation method is not only simple, but also the diversity and uniformity of sampling can be ensured.

[0082] In an alternative embodiment of the present application, according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the PPF feature, the local feature vector of the sampling point is determined, including:

[0083] At least one sampling point group corresponding to the sampling point is determined according to a set radius. The sampling point group includes the neighborhood data points within the spherical region determined by the sampling point with the set radius;

[0084] For each sampling point group, according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point within the sampling point group, and the PPF feature, through multi-layer perception and max pooling encoding operations, the geometric feature corresponding to the sampling point group is obtained;

[0085] The geometric features of at least one sampling point group are fused according to the set radius to obtain the local feature vector of the sampling point.

[0086] Optionally, after sampling to obtain the sampling point, at least one sampling point group of the sampling point can be determined according to a preset radius. Specifically, after the sampling point is selected, a fixed number of points (i.e., neighborhood data points) within the spherical region with a radius r around it can be selected as a group. For example, when sampling for the first time, the selectable radii are [0.1, 0.2, 0.4], and the number of samples (i.e., the number of neighborhood data points) corresponding to the sampling point group at this time is [32, 64, 128].

[0087] Furthermore, for each sampling point group, the three-dimensional coordinates of the sampling point and each neighborhood data point within the sampling point group can be determined, and then according to the three-dimensional coordinates of the sampling point and each neighborhood data point within the sampling point group, the distance between the sampling point and each neighborhood data point within the sampling point group and the PPF feature can be determined.

[0088] In an alternative embodiment of the present application, the PPF feature includes the angle between the line connecting the sampling point and each neighborhood point, the angle between the normal vector of each neighborhood point and the line connecting the sampling points, the angle between the normal vectors of the two points of the sampling point and each neighborhood point, and the Euclidean distance between the sampling point and each neighborhood point.

[0089] For example, assuming that the three-dimensional coordinate point of the sampling point is xyz, the distance between the sampling point and each neighborhood data point within the group and the PPF feature between the sampling point and each neighborhood point within the group can be as Figure 4As shown. Each large circle in the figure is a sampling point group, the center point in the circle (such as point c in the figure) is the sampling point, and the points in the circle that are connected to the sampling point (such as points P1 to P8 in the figure) are the neighborhood points that need to calculate the distance and PPF features. c A neighborhood point x i Take the distance between the sampling point and each neighboring data point in the group as an example to illustrate the PPF characteristics between the sampling point and each neighboring point in the group. c and the neighboring point x i The Euclidean distance d is |x c -x i |, sampling point x c and the neighboring point x i The angle F1 between the point pairs and the sampling point x c and the neighboring point x i The angle F3 between the point pairs and the line connecting them, the sampling point x c and the neighboring point x i The normal vector n of the two points i and n c The angle F2 between them, at this time the PPF feature is PPF(x c ,x i )=(||d||,∠(n c -d),∠(n i -d),∠(n c -n i )).

[0090] Furthermore, the geometric features of the operation area can be encoded through multi-layer perception and maximum pooling, and then feature splicing can be performed by setting different radii, and these features can be aggregated into local feature vectors of the sampling points.

[0091] In this application, PPF features are added to the training samples, which can enhance the expressive ability of the local features of the sampling points. And because the PPF features combine the ideas of global modeling and local matching, they can still show high recognition performance when the sheep body data is noisy, cluttered and partially occluded.

[0092] In the optional example of the application, the training samples are obtained based on the initial sample point cloud data and the local feature vector of each sampling point, including:

[0093] The local feature vector of each sampling point is fused into the initial sample point cloud data by upsampling to obtain the corresponding global feature vector;

[0094] The global feature vector and the local feature vector of each sampling point are concatenated to obtain the training sample.

[0095] Optionally, for the obtained local features, the local feature vectors of each sampling point can be fused into the initial sample point cloud data by reverse interpolation, i.e., upsampling, to obtain the global feature vector corresponding to each sampling point, and then the global feature and the previous local feature are concatenated to obtain the training sample.

[0096] In an optional embodiment of the present application, the background segmentation model is trained in the following manner:

[0097] Obtain at least one training sample, the labeled classification result corresponding to each data point in each training sample, and an MLP model, where the labeled classification result is the sheep body data point identifier or the background data point identifier;

[0098] Train the MLP model based on each training sample until the loss function corresponding to the MLP model converges, and use the MLP model at the end of training as the background segmentation model;

[0099] Among them, the input of the MLP model is the training sample, the output is the classification result of the training sample, the loss function is the cross-entropy loss function, and the value of the loss function is used to characterize the difference between the classification results of each training sample output by the model and the labeled classification results corresponding to each training sample.

[0100] Optionally, after obtaining the training sample, each training sample can be input into the MLP model to train the MLP model until the loss function corresponding to the MLP model converges. At this time, the MLP model will output whether each data point in the training sample is a sheep body data point or a background data point. For example, for a data point, if 0 represents a background data point and 1 represents a sheep body data point, at this time, the probability of 0 or 1 can be output, and then it is determined whether the data point is a background data point or a sheep body data point according to the output probability of 0 or 1. In addition, the value of the above loss function characterizes the difference between the classification results of each training sample output by the model and the labeled classification results corresponding to each training sample. If the loss function converges, it means that the training can be ended, and at this time, the MLP model at the end of training is used as the background segmentation model.

[0101] Optionally, since the loss function in the present application is modified from the negative log-likelihood loss function NLLLoss to the cross-entropy loss with relatively stable derivatives, this helps to propagate effective gradients in the deep layers of the network, making the gradients in backpropagation relatively stable, which helps the convergence of the optimization algorithm and helps the network to better learn features.

[0102] In an optional embodiment of the present application, the registration model is trained by at least one training data set, and the training data set is obtained in the following manner:

[0103] Obtain the registration parameters and the source point cloud dataset, where the source point cloud dataset includes the initial training point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0104] Coarsely register the initial training point cloud data of the three views according to the registration parameters to obtain the training point cloud data after coarse registration of the three views;

[0105] Precisely register the training point cloud data after coarse registration of the three views according to the ICP algorithm to obtain the reference point cloud dataset, where the reference point cloud dataset includes the training data of the upper left view, the training data of the upper right view, and the training data of the upper left and right views;

[0106] Use the source point cloud dataset and the reference point cloud dataset as the training dataset.

[0107] Optionally, the registration model required during the data registration process can also be trained based on at least one training data set. At this time, the initial training point cloud data including the left view, the right view, and the top view of the sample sheep body can be obtained as the source point cloud dataset, and then, with the calibration box as the reference, its transformation matrix, that is, the registration parameters, can be obtained. Coarsely register the initial training point cloud data of the left view, the right view, and the top view of the sample sheep body after background segmentation, with the initial training point cloud data of the top view as the reference. Further, use the ICP algorithm to precisely register the initial training point cloud data of the left view, the right view, and the top view, and merge the results after registration to obtain three point cloud files, that is, the training data of the upper left view, the training data of the upper right view, and the training data of the upper left and right views as the reference point cloud dataset. Further, use the source point cloud dataset and the reference point cloud dataset as the training dataset and use them to train the registration model.

[0108] In the embodiment of the present application, the traditional method is used to register the point cloud data as the training data, and after coarse registration by the calibration method, precise registration is performed by the ICP method, which can prevent falling into the local optimal solution and improve the data quality, and at the same time reduce the manual annotation process.

[0109] In an optional example of the present application, the registration model is trained in the following manner:

[0110] Obtain the training dataset and the initial rigid transformation matrix, and transform the source point cloud dataset in the training dataset according to the initial rigid transformation matrix to obtain the transformed source point cloud dataset;

[0111] Determine the corresponding permutation matrix according to the transformed source point cloud dataset and the reference point cloud dataset in the training dataset;

[0112] For each data point in the source point cloud dataset, determine the weight of the data point according to the three-dimensional coordinates of the corresponding data point in the reference point cloud dataset;

[0113] Obtain the covariance matrix according to the weights of each data point in the reference point cloud dataset, and perform singular value decomposition on the covariance matrix with respect to the permutation matrix to obtain the rigid transformation matrix;

[0114] Use the rigid transformation matrix as the initial rigid transformation matrix to perform iterative training until the set number of times, and use the trained rigid transformation matrix as the registration model.

[0115] Optionally, for the obtained training dataset, the source point cloud dataset in the training dataset can be transformed based on the initial rigid transformation matrix to obtain a transformed source point cloud dataset. At this time, the corresponding permutation matrix can be determined according to the transformed source point cloud dataset and the reference point cloud dataset. Further, for each data point in the source point cloud dataset, calculate the three-dimensional coordinates of the corresponding data point in the reference point cloud dataset. Since not every data point in the source point cloud dataset can find a corresponding data point in the reference point cloud dataset, the weights can be obtained by summation, and then the covariance matrix can be calculated according to the weights of each obtained data point, and the permutation matrix can be subjected to singular value decomposition based on the obtained covariance matrix, so as to obtain the rigid transformation matrix.

[0116] Correspondingly, use the obtained rigid transformation matrix as the initial rigid transformation matrix to perform iterative training until the set number of times, and use the trained rigid transformation matrix as the registration model. At this time, the registration maps of the left view and the top view of the sheep body and the registration maps of the right view and the top view of the sheep body can be obtained respectively based on the obtained registration model, so that the final three-dimensional reconstruction map of the sheep body can be obtained.

[0117] Optionally, the set number of iterations can be set according to actual needs, and this application does not make any restrictions. For example, the number of iterations is set to 2 during the training of the registration model and 5 during the test.

[0118] In an optional embodiment of the present application, determining the corresponding permutation matrix according to the transformed source point cloud dataset and the reference point cloud dataset in the training dataset includes:

[0119] Extract features from the transformed source point cloud dataset and the reference point cloud dataset in the training dataset, and aggregate and normalize the extracted features to obtain feature vectors;

[0120] Obtain the corresponding annealing parameter and outlier parameter according to the transformed source point cloud dataset and the reference point cloud dataset;

[0121] A matching matrix is obtained based on the converted source point cloud dataset, the reference point cloud dataset, the annealing parameter, and the outlier parameter, and the matching matrix is normalized to obtain a permutation matrix.

[0122] Optionally, feature extraction is performed on the converted source point cloud dataset and the reference point cloud dataset respectively. The specific feature extraction method can be: extracting the features of 64 neighborhood points within the neighborhood sphere with a radius of 0.3 for each data point. At this time, the obtained features include the three-dimensional coordinates xyz of each data point, the distance between the data point and each neighborhood point within the neighborhood, and the PPF feature between the data point and the neighborhood point. Then, the geometric features of the local area are encoded through a shared multi-layer perceptron and a max pooling operation, and these features are aggregated and normalized to obtain a feature vector. Among them, for the method of obtaining the PPF feature, reference can be made to the description above, and it will not be elaborated in this embodiment of the present application.

[0123] Furthermore, a column of feature 0 can be added to the converted source point cloud dataset, and a column of feature 1 can be added to the reference point cloud dataset, and they are concatenated along the added column features. The concatenated data is processed by an MLP (multi-layer perceptron) with shared parameters, then a max pooling operation is performed, and then it is processed by an MLP and a Softplus function activation to obtain the annealing parameter β and the outlier parameter α respectively. Correspondingly, a matching matrix is calculated according to the features of the converted source point cloud dataset and the reference point cloud dataset, and the matching matrix is Sinkhorn-normalized to make it a permutation matrix.

[0124] In the embodiment of the present application, after rough registration using the calibration method, ICP is used for fine registration. While improving the quality of training data, the traditional data preprocessing method is combined with deep learning. Deep learning is used to automatically learn the complex features in the data. When the initial value is good, the model convergence is accelerated, and the accuracy of point cloud registration is improved.

[0125] To better understand the sheep body three-dimensional reconstruction method provided by the embodiments of the present application, the following will be described in detail with specific embodiments. Specifically, this example can specifically include three parts: data preprocessing, background segmentation, and point cloud registration. The following will be combined with Figure 5 to elaborate in detail.

[0126] (1) The original point cloud data of the sheep body: The original point cloud data is obtained through a depth camera (i.e., Figure 5(in the step of obtaining the original point cloud data), then, for the three point cloud files (i.e., point cloud data) of the left, upper, and right views, the four filtering methods of straight-through filtering, region growing method, radius filtering, and DBSCAN are respectively used to obtain the point cloud data of the left, upper, and right parts of the target detection object, the sheep body (i.e., the sheep body in the figure: the four filtering methods of straight-through filtering, region growing method, radius filtering, and DBSCAN are respectively used to obtain the target detection object, the sheep body).

[0127] (2) Background acquisition: The filtered sheep body point cloud data including three views is respectively sent into the corresponding original point cloud data, and the corresponding background data is obtained (i.e., obtaining the corresponding background data of the filtered sheep body in the original point cloud data in [description]).

[0128] (3) Sheep body identification: The last column of the sheep body point cloud data preprocessed in step (1) is filled and identified with 1 (i.e., Figure 5 filling and identifying the last column of the sheep body point cloud data with 1 in [description]).

[0129] (4) Background identification: The last column of the background point cloud data obtained in step (2) is filled and identified with 0 (i.e., Figure 5 filling and identifying the last column of the background point cloud data with 0 in [description]).

[0130] (5) Data merging: The identified sheep body and background data in steps (3) and (4) are merged along the last column into a new point cloud file (i.e., the sample point cloud data) for data training (i.e., Figure 5 merging the identified sheep body and background data into a new point cloud file for data training in [description]).

[0131] In this process, the sheep body point cloud data is obtained by using traditional geometric methods, the background data is obtained from the original point cloud file according to the point cloud data, and then the traditional preprocessing methods are used to remove noise and improve data quality, while reducing the manual annotation process.

[0132] (6) Training the segmentation model based on the training samples (i.e., the background segmentation in the figure):

[0133] (6-1) Sampling: The farthest sampling method is modified to the random sampling method without replacement for downsampling. Each data point can only be selected once. While improving the sampling efficiency, it is used to ensure the diversity and uniformity of sampling. The advantage of this step is that, compared with the farthest sampling method, the random sampling method without replacement has high sampling efficiency and simple implementation method.

[0134] (6-2) Grouping: After the sampling points are selected, the fixed number of points (i.e., neighborhood points) within a spherical region with a radius R around them are taken as a group. Optionally, during the first sampling, the selected radii are [0.1, 0.2, 0.4], and the corresponding number of samples (i.e., neighborhood points) in the clusters are [32, 64, 128].

[0135] (6-3) Feature Extraction: For each group, a small PointNet network is used to extract features, which are: the xyz coordinate points of the sampling points, the dyz and PPF of the neighborhood points. By establishing a multi-layer perceptron and max-pooling operations, the geometric features of the local region are encoded, and these features are aggregated into the feature vector of the sampling points. Among them, the PPF feature is the angle between the line connecting the sampling point and the neighborhood point; the angle between the normal vector of the neighborhood point and the line connecting the sampling points; the angle between the normal vectors of the two points of the sampling point and the neighborhood point; the Euclidean distance between the sampling point and the neighborhood point. For details, please refer to Figure 4 as shown.

[0136] It can be understood that the advantage of this step is that it adds the PPF feature, enhances the expression ability of the local features of the sampling points, and the PPF feature combines the ideas of global modeling and local matching. This method shows high recognition performance in dealing with noise, clutter, and partial occlusion.

[0137] Optionally, in this example, a downsampled point set can be obtained through two rounds of downsampling and grouping in steps (6-1), (6-2), and (6-3), and the global features of the point cloud are gradually extracted. These feature vectors are used for subsequent segmentation tasks.

[0138] (6-4) Decoding: The downsampled points output by the encoder are upsampled and feature-fused back to the original point cloud (i.e., the original point cloud data) through reverse interpolation.

[0139] (6-6) Segmentation: The global features and the point feature vectors output by the decoder are fed into one or more fully connected layers (i.e., MLP) to perform the prediction of the segmentation task. For each point, the output is the probability of 0 or 1, where 0 represents the background and 1 represents the sheep body. The result can be as Figure 6 shown. In the figure, the data points within the circle are the point cloud data of the sheep body, and the other are the point cloud data of the background.

[0140] Preferably, in the segmentation model training of step (6), during the output stage of the network, the original negative log-likelihood loss function NLLLoss is modified to the cross-entropy loss (Cross Entropy Loss) to calculate the classification error of each point. Its advantage lies in that the derivative of the cross-entropy loss is relatively stable, which helps to spread effective gradients in the deep layers of the network, making the gradients in backpropagation relatively stable, contributing to the convergence of the optimization algorithm and enabling the network to better learn features.

[0141] It can be understood that in this example, using traditional preprocessing methods can retain key features in the point cloud data and improve data quality. Combining with deep learning can automatically learn complex features in the data, thereby improving the accuracy of point cloud segmentation and enhancing the robustness of the segmentation model for data under different conditions, especially when dealing with complex scenes and noisy data.

[0142] (7) Source point cloud X: Define the left-view point cloud data and right-view point cloud data of the sheep body after segmentation in step (6) as the source point cloud X (i.e., the source point cloud dataset, the segmented left or right-view point cloud in the figure).

[0143] (8) Define the files of the upper-left view point cloud data, upper-right view point cloud data, and upper-left and right view point cloud data that apply calibration method for rough registration and ICP for fine registration as the reference point Y (i.e., the reference point cloud dataset), as Figure 7 shown. Specifically, it can be the following sub-steps:

[0144] (8-1) Taking the calibration box as a reference, obtain its transformation matrix, i.e., the registration parameters, and perform rough registration on the point clouds of the left, upper (also called top), and right views of the sheep body after background segmentation. Taking the top view as a reference, perform rough registration.

[0145] (8-2) Use the ICP algorithm to perform fine registration on the left and top, and right and top, and merge the results after registration to obtain three point cloud files respectively: the left-top view point cloud data, the right-top view point cloud data, and the three files of the left and right view reference point cloud dataset are used as the reference point cloud Y for the training data of the registration model. Among them, the right-top view point cloud + the top view point cloud + the left view point cloud = the complete point cloud image.

[0146] It can be understood that in this example, using the traditional method to register point cloud data as training data, after rough registration by the calibration method and then fine registration by the ICP method can prevent getting stuck in local optimal solutions and improve data quality while reducing the process of manual annotation.

[0147] (9) Use the RPMnet algorithm to perform registration on the left view, top view, and right view. The training of its registration model specifically includes the following sub-steps.

[0148] (9-1) Preliminary Rigid Transformation {R i-1 , t i-1} (i.e., the transformation matrix of the previous iteration); the transformed source point cloud is transformed from the source point cloud X by the transformation matrix of the previous iteration.

[0149] (9-2) Feature Extraction and Y Feature Extraction: Sample the xyz coordinate points, neighborhood points dyz and PPF. Encode the geometric features of the local area through the establishment of a multi-layer perceptron and max-pooling operations, and aggregate these features into the feature vectors of the sampling points. Among them, the PPF feature is the angle between the line connecting the sampling point and the neighborhood point; the angle between the normal vector of the neighborhood point and the line connecting the sampling points; the angle between the normal vectors of the two points of the sampling point and the neighborhood point; the Euclidean distance between the sampling point and the neighborhood point. For specific details, please refer to Figure 4 as shown, which will not be elaborated here.

[0150] (9-2) Parameter Prediction: Add a column of feature 0 to the point cloud, add a column of feature 1 to Y, and splice them along the added column features. Process the spliced data through an MLP with shared parameters, then perform max-pooling operations, and then process it through an MLP and a Softplus function activation (not shown in the figure) to obtain the annealing parameter β and the outlier parameter α.

[0151] (9-4) Calculate the Matching Matrix: Calculate the matching matrix M according to the features of i and Y, and the values of β and α, and perform Sinkhorn normalization on M i to make it a permutation matrix.

[0152] (9-5) Estimate the Rigid Transformation {R i , t i}: For each point in the X set, calculate its corresponding reference point in Y. Since not every x j can find its corresponding point y j , the weight w jk is obtained by summing m j , where m jk represents the matching degree between (x j , y j ), k refers to the number of all possible matching points, and the singular value decomposition (SVD) is used to calculate the singular values and singular vectors of the covariance matrix, so as to obtain the transformation matrix {R i , t i}.

[0153] (9-6) Return the transformation matrix obtained in step (9-5) to step (9-1) for a new round of iteration, where the number of iterations is 2 during model training and 5 during testing.

[0154] (9-7) Apply the finally obtained transformation matrix to obtain the registration maps of the left view and the top view, and the registration maps of the right view and the top view respectively. Finally, merge the two registered point cloud files (i.e., the registered sheep body point cloud in the figure, the left top view point cloud data and the right top view point cloud data in the above) to obtain the final three-dimensional reconstruction map of the sheep body.

[0155] In this example, after rough registration using the calibration method, ICP is used for fine registration. While improving the quality of training data, the traditional data preprocessing method is combined with deep learning. Deep learning is used to automatically learn the complex features in the data. When the initial value is good, it can accelerate the convergence of the model and improve the accuracy of point cloud registration. In addition, after testing, the accuracy of the segmentation algorithm based on this application can reach 0.98, the IoU can reach 0.955, the average transformation error of the registration algorithm is 0.01004, and the chamfer error is 0.0004339. Its transformation accuracy is much higher than that of the traditional algorithm.

[0156] The embodiment of this application provides a device for three-dimensional reconstruction of a sheep body, as Figure 8 shown. The device 80 may include: a segmentation module 801, a registration module 802, and a training module 803, where

[0157] The segmentation module is used to obtain the original point cloud data of the target sheep body and a background segmentation model, and based on the background segmentation model, segment the background data in the original point cloud data to obtain sheep body point cloud data, and the sheep body point cloud data includes the left view point cloud data, the right view point cloud data, and the top view point cloud data of the target sheep body;

[0158] The registration module is used to input the sheep body point cloud data into the registration model to obtain the left top view point cloud data and the right top view point cloud data of the target sheep body, and merge the left top view point cloud data and the right top view point cloud data to obtain the three-dimensional reconstruction map of the target sheep body;

[0159] The training module is used to train a background segmentation model based on at least one training sample, and each training sample is obtained by the following method:

[0160] Obtain sample point cloud data, and perform at least one downsampling on the sample point cloud data using the non-replacement random sampling method to obtain at least one sampling point. The sample point cloud data is obtained by filtering the initial sample point cloud data using four set filtering methods simultaneously. The initial sample point cloud data includes the initial sample point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0161] For each sampling point, determine the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the point pair feature.

[0162] Obtain training samples based on the initial sample point cloud data and the local feature vector of each sampling point.

[0163] Optionally, when the training module obtains the sample point cloud data, it is specifically used for:

[0164] Obtain the initial sample point cloud data, and simultaneously filter the initial sample point cloud data of the three views in the initial sample point cloud data respectively using four preset filtering methods to obtain the target initial sample point cloud data.

[0165] Obtain the sample background point cloud data based on the initial sample point cloud data and the target initial sample point cloud data.

[0166] Use different identifiers to identify the last column of the target initial sample point cloud data and the sample background point cloud data respectively, and merge the identified target initial sample point cloud data and the sample background point cloud data to obtain the sample point cloud data.

[0167] Optionally, the four preset filtering methods include direct filtering, region growing method, radius filtering, and DBSCAN.

[0168] Optionally, when the training module determines the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the PPF feature, it is specifically used for:

[0169] Determine at least one sampling point group corresponding to the sampling point according to the set radius, and the sampling point group includes the neighborhood data points within the spherical region determined by the sampling point with the set radius.

[0170] For each sampling point group, obtain the geometric feature corresponding to the sampling point group through multi-layer perception and max pooling encoding operations according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point within the sampling point group, and the PPF feature.

[0171] Fuse the geometric features of at least one sampling point group according to the set radius to obtain the local feature vector of the sampling point.

[0172] Optionally, the PPF feature includes the angle between the point pair connection lines between the sampling point and each neighborhood point, the angle between the normal vector of each neighborhood point and the sampling point pair connection line, the angle between the normal vectors of the two points of the sampling point and each neighborhood point, and the Euclidean distance between the sampling point and each neighborhood point.

[0173] Optionally, when obtaining the training samples based on the initial sample point cloud data and the local feature vectors of each sampling point, the training module is specifically configured to:

[0174] Fuse the local feature vectors of each sampling point into the initial sample point cloud data through upsampling to obtain the corresponding global feature vector;

[0175] Concatenate the global feature vector and the local feature vectors of each sampling point to obtain the training samples.

[0176] Optionally, the background segmentation model is trained through the following method:

[0177] Obtain at least one training sample, the labeled classification result corresponding to each data point in each training sample, and an MLP model, where the labeled classification result is the sheep body data point identifier or the background data point identifier;

[0178] Train the MLP model based on each training sample until the loss function corresponding to the MLP model converges, and use the MLP model at the end of the training as the background segmentation model;

[0179] Wherein, the input of the MLP model is the training sample, the output is the classification result of the training sample, the loss function is the cross-entropy loss function, and the value of the loss function is used to characterize the difference between the classification results of each training sample output by the model and the labeled classification results corresponding to each training sample.

[0180] Optionally, the registration model is trained through at least one training data set, and the training data set is obtained through the following method:

[0181] Obtain the registration parameters and the source point cloud data set, where the source point cloud data set includes the initial training point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view;

[0182] Coarsely register the initial training point cloud data of the three views according to the registration parameters to obtain the coarsely registered training point cloud data of the three views;

[0183] Precisely register the coarsely registered training point cloud data of the three views according to the ICP algorithm to obtain the reference point cloud data set, where the reference point cloud data set includes the training data of the upper left view, the training data of the upper right view, and the training data of the upper left and right views;

[0184] Use the source point cloud data set and the reference point cloud data set as the training data set.

[0185] Optionally, the registration model is trained through the following method:

[0186] Obtain a training data set and an initial rigid transformation matrix, and transform the source point cloud data set in the training data set according to the initial rigid transformation matrix to obtain a transformed source point cloud data set;

[0187] Determine a corresponding permutation matrix according to the transformed source point cloud data set and the reference point cloud data set in the training data set;

[0188] For each data point in the source point cloud data set, determine the weight of the data point according to the three-dimensional coordinates of the corresponding data point in the reference point cloud data set;

[0189] Obtain a covariance matrix according to the weights of each data point in the reference point cloud data set, and perform singular value decomposition on the covariance matrix with respect to the permutation matrix to obtain a rigid transformation matrix;

[0190] Use the rigid transformation matrix as the initial rigid transformation matrix for iterative training until a set number of times, and use the trained rigid transformation matrix as the registration model.

[0191] Optionally, when the training module determines a corresponding permutation matrix according to the transformed source point cloud data set and the reference point cloud data set in the training data set, it is specifically used for:

[0192] Extract features from the transformed source point cloud data set and the reference point cloud data set in the training data set, and aggregate and normalize the extracted features to obtain feature vectors;

[0193] Obtain corresponding annealing parameters and outlier parameters according to the transformed source point cloud data set and the reference point cloud data set;

[0194] Obtain a matching matrix according to the transformed source point cloud data set, the reference point cloud data set, the annealing parameters and the outlier parameters, and perform normalization processing on the matching matrix to obtain a permutation matrix.

[0195] The device for three-dimensional reconstruction of a sheep body in this embodiment can execute the method for three-dimensional reconstruction of a sheep body shown in the embodiments of the present application, and its implementation principle is similar, which will not be elaborated here.

[0196] The embodiments of the present application provide an electronic device, such as Figure 9 shown, Figure 9 The electronic device 2000 shown includes: a processor 2001 and a memory 2003. Among them, the processor 2001 and the memory 2003 are connected, such as connected through a bus 2002. Optionally, the electronic device 2000 may further include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation to the embodiments of the present application.

[0197] Among them, the processor 2001 is applied in the embodiments of the present application to be used for implementingFigure 6 The function of the AA module shown.

[0198] The processor 2001 can be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of the present application. The processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0199] The bus 2002 can include a path for transmitting information between the above components. The bus 2002 can be a PCI bus or an EISA bus, etc. The bus 2002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, Figure 9 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0200] The memory 2003 can be a ROM or other types of static storage devices that can store static information and instructions, a RAM, or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM, a CD-ROM, or other optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0201] The memory 2003 is used to store the application program code for implementing the solution of the present application, and is controlled by the processor 2001 for execution. The processor 2001 is used to execute the application program code stored in the memory 2003 to implement Figure 8 the actions of the device provided by the shown embodiment.

[0202] The embodiments of the present application provide a computer-readable storage medium for storing computer instructions. When the computer instructions run on a computer, the computer can execute to implement the three-dimensional reconstruction method of a sheep body.

[0203] The terms and implementation principles related to a computer-readable storage medium in the present application can specifically refer to a calling method for three-dimensional reconstruction of a sheep body in the embodiments of the present application, and will not be elaborated herein.

[0204] It should be understood that although the steps in the flowchart of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least some of the sub-steps or stages of other steps or other steps.

[0205] The above are only some embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for a sheep body, characterized in that, Including: Obtain the original point cloud data of the target sheep body and the background segmentation model. Based on the background segmentation model, segment the background data in the original point cloud data to obtain the sheep body point cloud data, where the sheep body point cloud data includes the left view point cloud data, the right view point cloud data, and the top view point cloud data of the target sheep body; Input the sheep body point cloud data into the registration model to obtain the upper left view point cloud data and the upper right view point cloud data of the target sheep body, and merge the upper left view point cloud data and the upper right view point cloud data to obtain the three-dimensional reconstruction diagram of the target sheep body; Among them, the background segmentation model is trained based on at least one training sample, and each training sample is obtained through the following method: Obtain the sample point cloud data, and use the non-replacement random sampling method to perform at least one downsampling on the sample point cloud data to obtain at least one sampling point. The sample point cloud data is obtained by filtering the initial sample point cloud data simultaneously using four preset filtering methods respectively. The initial sample point cloud data includes the initial sample point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view; For each sampling point, determine the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the Point Pair Feature (PPF); Obtain the training sample according to the initial sample point cloud data and the local feature vector of each sampling point.

2. The method according to claim 1, wherein The obtaining of the sample point cloud data includes: Obtain the initial sample point cloud data, and filter the initial sample point cloud data of the three views in the initial sample point cloud data simultaneously using four preset filtering methods respectively to obtain the target initial sample point cloud data; Obtain the sample background point cloud data according to the initial sample point cloud data and the target initial sample point cloud data; Use different identifiers to identify the last column of the target initial sample point cloud data and the sample background point cloud data respectively, and merge the identified target initial sample point cloud data and sample background point cloud data to obtain the sample point cloud data.

3. The method according to claim 1 or 2, characterized in that, The four preset filtering methods include straight-through filtering, region growing method, radius filtering, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

4. The method according to claim 1, characterized in that, The determining of the local feature vector of the sampling point according to the three-dimensional coordinates of the sampling point, the distance between the sampling point and each neighborhood data point, and the PPF feature includes: Determine at least one sampling point group corresponding to the sampling point according to the set radius, where the sampling point group includes the neighborhood data points within the spherical region determined by the sampling point with the set radius; For each of the sampling point groups, according to the three-dimensional coordinates of the sampling points, the distances between the sampling points and each neighborhood data point within the sampling point group, and the PPF features, through multi-layer perception and max pooling encoding operations, geometric features corresponding to the sampling point group are obtained; According to the set radius, the geometric features of the at least one sampling point group are fused to obtain the local feature vector of the sampling points.

5. The method according to claim 4, wherein The PPF features include the angle between the line connecting the sampling point and each neighborhood point, the angle between the normal vector of each neighborhood point and the line connecting the sampling points, the angle between the normal vectors of the two points of the sampling point and each neighborhood point, and the Euclidean distance between the sampling point and each neighborhood point.

6. The method according to claim 1, characterized in that, The obtaining of the training samples according to the initial sample point cloud data and the local feature vector of each sampling point includes: By means of upsampling, the local feature vector of each sampling point is fused into the initial sample point cloud data to obtain the corresponding global feature vector; The global feature vector and the local feature vector of each sampling point are concatenated to obtain the training samples.

7. The method according to claim 1, characterized in that, The background segmentation model is trained in the following manner: Obtain at least one of the training samples, the labeled classification result corresponding to each data point in each training sample, and a multi-layer perceptron model (MultiLayer Perceptron, MLP), where the labeled classification result is the sheep body data point identifier or the background data point identifier; Based on each of the training samples, the MLP model is trained until the loss function corresponding to the MLP model converges, and the MLP model at the end of training is used as the background segmentation model; Wherein, the input of the MLP model is the training sample, the output is the classification result of the training sample, the loss function is the cross-entropy loss function, and the value of the loss function is used to characterize the difference between the classification results of each training sample output by the model and the labeled classification results corresponding to each training sample.

8. The method according to claim 1, characterized in that The registration model is trained by at least one training data set, and the training data set is obtained in the following manner: Obtain registration parameters and a source point cloud data set, where the source point cloud data set includes the initial training point cloud data of three views of the sample sheep body, and the three views include the left view, the right view, and the top view; According to the registration parameters, the initial training point cloud data of the three views are roughly registered to obtain the training point cloud data after rough registration of the three views; According to the ICP algorithm, the training point cloud data after rough registration of the three views are finely registered to obtain a reference point cloud data set, where the reference point cloud data set includes the training data of the upper left view, the training data of the upper right view, and the training data of the upper left and right views; The source point cloud data set and the reference point cloud data set are used as the training data set.

9. The method according to claim 8, wherein The registration model is trained in the following manner: Obtain the training data set and an initial rigid transformation matrix, and according to the initial rigid transformation matrix, the source point cloud data set in the training data set is transformed to obtain a transformed source point cloud data set; Determine the corresponding permutation matrix according to the transformed source point cloud dataset and the reference point cloud dataset in the training dataset; For each data point in the source point cloud dataset, determine the weight of the data point according to the three-dimensional coordinates of the corresponding data point in the reference point cloud dataset; Obtain the covariance matrix according to the weights of each data point in the reference point cloud dataset, and perform singular value decomposition on the covariance matrix with respect to the permutation matrix to obtain the rigid transformation matrix; Use the rigid transformation matrix as the initial rigid transformation matrix for iterative training until the set number of times, and use the trained rigid transformation matrix as the registration model.

10. The method according to claim 9, wherein The step of determining the corresponding permutation matrix according to the transformed source point cloud dataset and the reference point cloud dataset in the training dataset includes: Extract features from the transformed source point cloud dataset and the reference point cloud dataset in the training dataset, and aggregate and normalize the extracted features to obtain feature vectors; Obtain the corresponding annealing parameter and outlier parameter according to the transformed source point cloud dataset and the reference point cloud dataset; Obtain the matching matrix according to the transformed source point cloud dataset, the reference point cloud dataset, the annealing parameter and the outlier parameter, and normalize the matching matrix to obtain the permutation matrix.

Images