Soybean plant three-dimensional point cloud model reconstruction method and device, medium and equipment

Abstract

The invention discloses a soybean plant three-dimensional point cloud model reconstruction method and device, a medium and equipment, and relates to the field of plant phenotype analysis, and the method comprises the steps: obtaining a plurality of multi-angle soybean plant images; inputting the soybean plant image into a semantic segmentation model based on a deep learning network Unet for training, and separating the soybean plant from the background; and based on an SAVANT algorithm, according to the soybean plant image after background separation processing, obtaining contour information and plant surface information of the soybean plant, extracting direction information by using the contour information of the soybean plant, and dividing the plant surface information by using a triangular grid to determine direction detail features of the soybean plant image. Direction detail features are enhanced through texture mapping, a soybean three-dimensional plant model is constructed, registration is performed on the generated soybean plant three-dimensional point cloud model based on an RANSAC algorithm and an ICP algorithm, and three-dimensional reconstruction is performed on soybean plants; and inputting the soybean plant image into the soybean three-dimensional plant model, and determining the phenotypic characteristics of the soybean plant.

Description

Technical Field

[0001] The present invention relates to the field of plant phenotyping, and in particular to a method, device, medium and equipment for reconstructing a three-dimensional point cloud model of a soybean plant. Background Art

[0002] As one of the most important legume seed crops in the world, soybean plays an irreplaceable role in the supply of edible oil and animal feed. It is rich in nutritional value, helps prevent a variety of diseases, and is of great significance to both human and animal consumption. As one of the world's major soybean producers, China urgently needs to improve soybean varieties and increase yields. The phenotypic characteristics of soybeans are key indicators for evaluating their yield and quality. Traditional phenotypic analysis methods mainly rely on manual measurement, which is not only time-consuming and labor-intensive, but also limited in accuracy and repeatability due to the involvement of human factors.

[0003] With the development of computer vision and deep learning technology, automated plant phenotyping has become possible. Automated phenotyping technology can extract various biological characteristics of plants through image processing and pattern recognition technology. However, existing automated phenotyping technology still faces challenges when dealing with crops such as soybeans that have complex structures and variable growth patterns. For example, traditional image processing methods are difficult to provide rich geometric information during image recognition, resulting in low recognition accuracy. Summary of the invention

[0004] The present invention provides a method, device, medium and equipment for reconstructing a three-dimensional point cloud model of a soybean plant, which is used to solve the above-mentioned problem existing in the prior art, that is, how to improve the accuracy of soybean plant phenotype recognition in the prior art. The present invention provides a method for reconstructing a three-dimensional point cloud model of a soybean plant, and the method comprises:

[0005] Acquire multiple soybean plant images from multiple angles;

[0006] The soybean plant image is input into the semantic segmentation model based on the deep learning network Unet for training to separate the soybean plant from the background;

[0007] Based on the SAVANT algorithm, the contour information and plant surface information of the soybean plant are obtained according to the soybean plant image after background separation. The directional information is extracted using the contour information of the soybean plant, and the plant surface information is divided by using triangular meshes to determine the directional detail features of the soybean plant image. The directional detail features are enhanced by texture mapping, and a soybean three-dimensional plant model is constructed. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for point cloud coarse alignment, and the iterative closest point ICP algorithm is used for fine alignment to reconstruct the soybean plant in three dimensions.

[0008] Optionally, the point cloud rough registration is performed by using a random sample consistency RANSAC algorithm, specifically including:

[0009] Select three non-collinear data points from the source cloud and the corresponding point set from the target cloud;

[0010] The source cloud is transformed into a new 3D point cloud dataset using a transformation matrix, the new 3D point cloud dataset is compared with the target cloud, all points whose distance deviation is less than a preset threshold are extracted, a consistent point cloud dataset is determined, and the number of inliers is recorded;

[0011] Repeat the above steps, and when a consistent point cloud set is obtained, the point cloud set with the largest number of internal points is selected.

[0012] Optionally, the acquiring of multiple multi-angle soybean plant images specifically includes:

[0013] A multi-view stereo vision method is used to acquire multiple soybean plant images.

[0014] Optionally, the soybean plant profile information specifically includes:

[0015] Information on key nodes in soybean plant outline.

[0016] The present invention provides a soybean plant three-dimensional point cloud model reconstruction device, comprising:

[0017] An acquisition module, used for acquiring multiple soybean plant images at multiple angles;

[0018] A training module is used to input the soybean plant image into a semantic segmentation model based on the deep learning network Unet for training, so as to separate the soybean plant from the background;

[0019] The reconstruction module is used to obtain the contour information and plant surface information of the soybean plant based on the soybean plant image after background separation processing based on the SAVANT algorithm, extract the direction information using the contour information of the soybean plant, and divide the plant surface information by using a triangular grid to determine the directional detail features of the soybean plant image, enhance the directional detail features through texture mapping, and construct a soybean three-dimensional plant model. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for point cloud coarse alignment, and the iterative closest point ICP algorithm is used for fine alignment to reconstruct the soybean plant in three dimensions.

[0020] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned soybean plant three-dimensional point cloud model reconstruction method is implemented.

[0021] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned soybean plant three-dimensional point cloud model reconstruction method is implemented.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method for reconstructing a three-dimensional point cloud model of a soybean plant. The method is based on the SAVANT algorithm, and obtains the surface geometric information of the soybean plant according to the soybean plant image after background separation processing. By dividing the plant surface information by using a triangular grid, the directional detail information of the soybean plant image can be obtained, and then a more accurate three-dimensional soybean plant model is constructed, thereby improving the phenotypic recognition accuracy of the soybean plant; the directional detail features are enhanced by texture mapping, thereby further enhancing the phenotypic recognition accuracy of the soybean plant; by using the random sample consistency RANSAC algorithm and the ICP algorithm to perform coarse and fine registration on the point cloud, a more accurate three-dimensional point cloud model can be obtained, and high-precision three-dimensional reconstruction can be achieved, thereby effectively improving the phenotypic recognition accuracy of the soybean plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 A flowchart of a method for reconstructing a three-dimensional point cloud model of a soybean plant provided by an embodiment of the present invention;

[0025] Figure 2 A network structure diagram of the deep learning network Unet used in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a computer device for a method for reconstructing a three-dimensional point cloud model of a soybean plant provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0029] Figure 1 FIG. 1 is a flowchart of a method for reconstructing a three-dimensional point cloud model of a soybean plant provided by an embodiment of the present invention. Figure 1 As shown, this embodiment shows a method for reconstructing a soybean plant three-dimensional point cloud model, comprising:

[0030] S1: Acquire multiple soybean plant images from multiple angles.

[0031] Optionally, a multi-view stereo vision method is used to acquire multiple soybean plant images.

[0032] S2: Input the soybean plant image into the semantic segmentation model based on the deep learning network Unet for training to separate the soybean plant from the background.

[0033] For example, the data annotation tool LabelMe can be used to annotate the soybean plant images, and then a data set for semantic segmentation is constructed based on the annotated soybean plant images, and the data set is divided into a training set and a test set.

[0034] After the training set and the test set can be divided into 8:2 ratios by the deep learning algorithm Unet, the Unet network is trained based on the training set, and the segmented images obtained by semantic segmentation prediction are processed to remove irrelevant backgrounds, such as Figure 2 shown.

[0035] S3: Based on the SAVANT algorithm, the contour information and plant surface information of the soybean plant are obtained according to the soybean plant image after background separation. The directional information is extracted using the contour information of the soybean plant, and the plant surface information is divided by using a triangular mesh to determine the directional detail features of the soybean plant image. The directional detail features are enhanced by texture mapping, and a soybean three-dimensional plant model is constructed. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for coarse point cloud registration, and the iterative closest point ICP algorithm is used for fine registration to reconstruct the soybean plant in three dimensions.

[0036] For example, several polygonal approximations of the contour can be obtained based on partial information of the soybean plant in the multi-angle image, a number is assigned to each approximation, and three vertices are calculated based on the polygonal contour, and information of each vertex is recorded.

[0037] By using triangular meshes, the complete surface is divided, the surface details of the soybean plant are determined, and the basic skeleton of the soybean three-dimensional plant model is obtained; finally, based on the directional information extracted from the three-dimensional surface contour model of the soybean plant and combined with the directional details from various multi-angle images, texture mapping is used to enhance the visualization features of the surface.

[0038] After constructing the soybean 3D plant model, its point cloud data needs to be registered.

[0039] Exemplarily, point cloud registration refers to the process of finding the transformation relationship between two sets of 3D data points from different coordinate systems. The process of point cloud data registration can adopt the following formula:

[0040] p t =R·q s +t

[0041] Among them, p t and q s are the corresponding points of the target point cloud and the source point cloud, that is, the points with the closest Euclidean distance, R represents the rotation matrix, and t represents the translation vector.

[0042] Optionally, a random sample consistency RANSAC algorithm is used for coarse registration of point clouds, and an iterative closest point ICP algorithm is used for fine registration.

[0043] For example, the RANSAC algorithm can be used for rough registration of point clouds. The specific process is as follows:

[0044] Select at least three corresponding pairs of points. Randomly select three non-collinear data points {q 1 ,q 2 ,q 3}, select the corresponding point set {p 1 , p 2 , p 3};

[0045] Use the minimum square method to calculate the rotation and translation matrices H for these two point sets;

[0046] The source cloud Q is transformed into a new 3D point cloud dataset Q' using the transformation matrix H. Q' is compared with P to extract all points whose distance deviation is less than a given threshold k to form a consistent point cloud dataset.

[0047] Set the threshold K and repeat the above process. After performing K operations, if a consistent point cloud set cannot be obtained, the model parameter estimation fails. If a consistent point cloud set is obtained, the point cloud set with the largest number of internal points is selected. At this time, the corresponding rotation and translation matrix H is the optimal model parameter.

[0048] The initial coarse registration approximately aligns the source cloud Q and the target cloud P, but in order to improve the accuracy of point cloud registration, fine registration is required.

[0049] In this embodiment, an iterative closest point ICP algorithm can be used to achieve fine registration of point clouds. The purpose of the ICP algorithm is to minimize the difference between two point clouds by finding the closest points in the two point clouds.

[0050] The ICP algorithm works as follows:

[0051] Based on the approximate parameter values ​​of R and t obtained by rough registration of point clouds, the corresponding points are directly searched by identifying the closest points in the two point clouds;

[0052] The least squares method is used to construct the objective function and iteratively minimize the overall distance between corresponding points until the termination condition is met, that is, the number of iterations reaches the maximum or the error is lower than the threshold, and finally the rigid transformation matrix is ​​obtained.

[0053] The core of the ICP algorithm is to minimize the objective function, which is essentially the sum of the squares of the Euclidean distances between all corresponding points. The objective function is obtained using the following formula:

[0054]

[0055] where E is the Euclidean norm, R is the rotation matrix, t is the translation vector, n is the number of point pairs between the two point clouds, and pt and qs are a pair of corresponding points from the target cloud and the source cloud, respectively.

[0056] Therefore, the ICP problem can be described as finding the values ​​of R and t that minimize E(R,t).

[0057] For example, the ICP problem can be solved using linear algebra SVD, and the specific process is as follows:

[0058] The centroid of two sets of corresponding points is obtained using the following formula;

[0059]

[0061] in, is the centroid of the corresponding points of the target cloud, is the centroid of the corresponding points of the source cloud, and n is the number of point pairs between the two point clouds.

[0062] The following formula is used to obtain the point set without centroid;

[0063]

[0064] The 3×3 matrix H is obtained using the following formula;

[0065] H=XYT

[0066] Among them, X and Y are the source point cloud matrix and target point cloud matrix of centroid removal, respectively.

[0067] Perform SVD decomposition on H;

[0068] H=U∑V T

[0069] Where U is an m×m matrix, Σ is an m×n matrix with all elements zero except the main diagonal, and V is an n×n matrix.

[0070] The optimal rotation matrix is ​​obtained using the following formula;

[0071] R=UV T

[0072] The optimal translation vector is obtained using the following formula:

[0073]

[0074] Of course, in addition to using the RANSAC algorithm and the ICP algorithm, some other point cloud registration methods are also provided under the existing technology, and these technologies provided under the existing technology can also be used.

[0075] The above is a soybean plant three-dimensional point cloud model reconstruction method provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding soybean plant three-dimensional point cloud model reconstruction device including:

[0076] An acquisition module, used for acquiring multiple soybean plant images at multiple angles;

[0077] A training module is used to input the soybean plant image into a semantic segmentation model based on the deep learning network Unet for training, so as to separate the soybean plant from the background;

[0078] The reconstruction module is used to obtain the contour information and plant surface information of the soybean plant based on the soybean plant image after background separation processing based on the SAVANT algorithm, extract the direction information using the contour information of the soybean plant, and divide the plant surface information by using a triangular grid to determine the directional detail features of the soybean plant image, enhance the directional detail features through texture mapping, and construct a soybean three-dimensional plant model. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for point cloud coarse alignment, and the iterative closest point ICP algorithm is used for fine alignment to reconstruct the soybean plant in three dimensions.

[0079] For the specific limitations on the soybean plant three-dimensional point cloud model reconstruction device, please refer to the limitations on the soybean plant three-dimensional point cloud model reconstruction method above, which will not be repeated here. Each module in the above-mentioned soybean plant three-dimensional point cloud model reconstruction device can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the reconstruction of the three-dimensional point cloud model of the soybean plant provided above.

[0081] The present invention also provides Figure 3 The structural diagram of the computer device shown in FIG. Figure 3 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the soybean plant three-dimensional point cloud model reconstruction method provided in the above embodiment.

[0082] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for reconstructing a three-dimensional point cloud model of a soybean plant, characterized in that: include: Acquire multiple soybean plant images from multiple angles; The soybean plant image is input into the semantic segmentation model based on the deep learning network Unet for training to separate the soybean plant from the background; Based on the SAVANT algorithm, the contour information and plant surface information of the soybean plant are obtained according to the soybean plant image after background separation. The directional information is extracted using the contour information of the soybean plant, and the plant surface information is divided by using triangular meshes to determine the directional detail features of the soybean plant image. The directional detail features are enhanced by texture mapping, and a soybean three-dimensional plant model is constructed. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for point cloud coarse alignment, and the iterative closest point ICP algorithm is used for fine alignment to reconstruct the soybean plant in three dimensions.

2. The soybean plant three-dimensional point cloud model reconstruction method according to claim 1, characterized in that: The point cloud rough registration is performed by using the random sample consistency RANSAC algorithm, specifically including: Select three non-collinear data points from the source cloud and the corresponding point set from the target cloud; The source cloud is transformed into a new 3D point cloud dataset using a transformation matrix, the new 3D point cloud dataset is compared with the target cloud, all points whose distance deviation is less than a preset threshold are extracted, a consistent point cloud dataset is determined, and the number of inliers is recorded; Repeat the above steps, and when a consistent point cloud set is obtained, the point cloud set with the largest number of internal points is selected.

3. The method for reconstructing a soybean plant three-dimensional point cloud model according to claim 1, characterized in that: The acquiring of multiple multi-angle soybean plant images specifically includes: A multi-view stereo vision method is used to acquire multiple soybean plant images.

4. The method for reconstructing a soybean plant three-dimensional point cloud model according to claim 1, characterized in that: The soybean plant profile information specifically includes: Information on key nodes in soybean plant outline.

5. A soybean plant three-dimensional point cloud model reconstruction device, characterized in that: include: An acquisition module, used for acquiring multiple soybean plant images at multiple angles; A training module is used to input the soybean plant image into a semantic segmentation model based on the deep learning network Unet for training, so as to separate the soybean plant from the background; The reconstruction module is used to obtain the contour information and plant surface information of the soybean plant based on the soybean plant image after background separation processing based on the SAVANT algorithm, extract the direction information using the contour information of the soybean plant, and divide the plant surface information by using a triangular grid to determine the directional detail features of the soybean plant image, enhance the directional detail features through texture mapping, and construct a soybean three-dimensional plant model. Based on the soybean three-dimensional plant model, the random sample consistency RANSAC algorithm is used for point cloud coarse alignment, and the iterative closest point ICP algorithm is used for fine alignment to reconstruct the soybean plant in three dimensions.

6. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method for reconstructing a three-dimensional point cloud model of a soybean plant according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for reconstructing a three-dimensional point cloud model of a soybean plant as claimed in any one of claims 1 to 4 is implemented.

Images