Soybean plant phenotype analysis method, system and device based on three-dimensional reconstruction

Abstract

This invention relates to the fields of computer vision and soybean plant phenotyping technology, and discloses a method, system, and apparatus for soybean plant phenotyping analysis based on 3D reconstruction. The method includes the following steps: The invention uses a device for multi-view image acquisition of soybean plants to collect images from multiple perspectives, and constructs an end-to-end soybean plant leaf phenotyping model based on PointNet Transform. This model can directly predict key phenotypic features such as leaf area and leaf perimeter from high-precision leaf point cloud data. Before constructing the model, discrete point cloud voxelization processing is required to provide conditions for semantic segmentation. This invention achieves automated acquisition of soybean plant point clouds, automated segmentation of soybean plant point clouds, and automated measurement of soybean plant leaf point cloud phenotyping, providing a data foundation for soybean plant phenotyping analysis and realizing high-throughput, intelligent soybean plant phenotyping analysis.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and soybean plant phenotyping technology, and in particular to a method, system and apparatus for soybean plant phenotyping based on three-dimensional reconstruction. Background Technology

[0002] High-throughput phenotypic analysis of soybean plants helps to deepen the understanding of the relationship between genotype and phenotype and accelerates the selection process for desirable heritable traits. However, traditional manual phenotypic acquisition methods are not only time-consuming and labor-intensive, but also prone to measurement errors due to subjective factors. Therefore, achieving efficient and automated phenotypic data acquisition has become a key challenge in the field of plant phenotyping research. In recent years, computer vision technology has been widely used in plant phenotyping analysis due to its ability to quickly and accurately analyze acquired data and reduce human error, greatly improving the efficiency of plant phenotyping analysis and reducing errors.

[0003] For decades, computer vision methods based on two-dimensional images have provided strong support for high-throughput plant phenotypic analysis. However, these two-dimensional methods are limited by their inherent limitations, such as lack of depth information, difficulty in handling occlusion problems, and inability to accurately describe plant structure, making it difficult to effectively measure many phenotypic parameters in high-throughput. Therefore, applying three-dimensional reconstruction technology to plant phenotypic research can provide richer three-dimensional data. In particular, some new technologies developed in recent years, such as PGSR, can generate high-quality plant point cloud data. Applying this three-dimensional data can make the phenotypic measurement of soybean plants more accurate. Therefore, building a high-throughput multi-view data acquisition device for soybean plants will greatly assist in the phenotypic extraction of soybean plants.

[0004] Furthermore, with the increasing prevalence of 3D data acquisition technologies such as LiDAR, depth cameras, and multi-view stereo reconstruction (MVS), deep learning-based point cloud processing neural networks such as PointNet, PointNet++, SparseConvUnet, and Transform are rapidly developing. Applying these advanced models to the processing of soybean plant point clouds and constructing corresponding phenotypic analysis algorithms and systems is a key technical challenge that urgently needs to be addressed. Therefore, this paper proposes a method, system, and device for soybean plant phenotypic analysis based on 3D reconstruction. Summary of the Invention

[0005] This invention overcomes the shortcomings of the prior art and provides a method, system and device for soybean plant phenotypic analysis based on three-dimensional reconstruction.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] The first aspect of this invention provides a method for soybean plant phenotypic analysis based on three-dimensional reconstruction, comprising the following steps:

[0008] A device is set up to acquire multi-view images of soybean plants, and multi-view images of soybean plants are taken to obtain multi-view images of the target soybean plants.

[0009] Image preprocessing is performed on multi-view images of the target soybean plant in the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and image feature data is extracted to construct a three-dimensional model of the soybean plant.

[0010] The three-dimensional model of soybean plant is processed into discrete point cloud voxels. The feature vectors of the voxelized soybean plant point cloud are extracted by combining the Sparse ConvUnet network and semantic segmentation is performed to obtain the leaf point cloud and stem point cloud of the target soybean plant.

[0011] Based on the leaf point cloud and stem point cloud of the target soybean plant, and combined with the PointNet Transform backbone network, the phenotypic information of the leaves and stems of the target soybean plant is measured.

[0012] Furthermore, in a preferred embodiment of the present invention, the device for setting up multi-view image acquisition of soybean plants and taking multi-view pictures of soybean plants to obtain multi-view images of the target soybean plants specifically includes:

[0013] A device for acquiring multi-view images of soybean plants, calibrated as a target image acquisition device, wherein the target image acquisition device includes an outer layer device and an inner layer device;

[0014] The outer layer of the target image acquisition device consists of an outer support frame and a light-shielding cloth, while the inner layer consists of 40 cameras, 40 industrial control computers, a switch, and a light source.

[0015] One camera is connected to an industrial control computer. The industrial control computer receives and temporarily stores the images captured by the camera. At the same time, the industrial control computer is connected to a switch via a network cable to form a local area network between the industrial control computer and the switch, which is designated as the target local area network.

[0016] Obtain a data analysis terminal device, connect the data analysis terminal device to the target local area network, so that the data analysis terminal device can extract and store photos taken by the camera that are temporarily stored in the industrial control computer;

[0017] Acquire soybean plants that need to be photographed from multiple perspectives, mark them as target soybean plants, place the target soybean plants in the target image acquisition device, and control all cameras to take multi-view images of the target soybean plants to obtain multi-view images of the target soybean plants. At the same time, store the obtained multi-view images of the target soybean plants in the data analysis terminal device.

[0018] In the process of controlling all cameras to capture multi-view images of the target soybean plant, it is necessary to close the shading cloth and ensure that 40 cameras are shooting at the same time.

[0019] Furthermore, in a preferred embodiment of the present invention, the step of performing image preprocessing on the multi-view images of the target soybean plant within the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and simultaneously extracting image feature data to construct a three-dimensional model of the soybean plant, specifically involves:

[0020] Within the data analysis terminal device, the acquired multi-view images of the target soybean plant are subjected to color space conversion. The color space conversion involves grayscale processing of the multi-view images of the target soybean plant to obtain grayscale multi-view images of the target soybean plant.

[0021] Interpolation and size cropping are performed on the grayscale target soybean plant multi-view image to retain the soybean plant portion in the grayscale target soybean plant multi-view image, resulting in a grayscale cropped soybean plant multi-view image.

[0022] Perspective and geometric corrections are performed on the grayscale cropped soybean plant multi-view images, and wavelet transform algorithm is introduced to perform noise reduction filtering on the grayscale cropped soybean plant multi-view images.

[0023] Among them, the wavelet transform algorithm is to perform wavelet transform on the pixels in the grayscale cropped soybean plant multi-view image, filter out the pixels whose wavelet values ​​are not within the preset range after wavelet transform, and reconstruct the pixels to obtain the preprocessed target soybean plant multi-view image.

[0024] A threshold segmentation algorithm is introduced to perform threshold segmentation on the preprocessed target soybean plant multi-view images to obtain image feature data of the preprocessed target soybean plant multi-view images. Then, 3D reconstruction software is introduced to perform 3D reconstruction on the image feature data of the preprocessed target soybean plant multi-view images to obtain a 3D model of the soybean plant.

[0025] Furthermore, in a preferred embodiment of the present invention, the process of discretizing the three-dimensional model of the soybean plant into voxels, extracting feature vectors from the voxelized soybean plant point cloud using the Sparse ConvUnet network, and performing semantic segmentation to obtain the leaf point cloud and stem point cloud of the target soybean plant, specifically involves:

[0026] Within the 3D reconstruction software, a 3D space is acquired, and a soybean plant point cloud voxelization processing module is obtained. Based on the soybean plant point cloud voxelization processing module, the 3D model of the soybean plant is voxelized.

[0027] The module for voxelizing the three-dimensional model of soybean plants converts the three-dimensional model of soybean plants into discrete voxel data points and introduces a Sparse ConvUnet network. Through sparse convolution, it performs up-and-down sampling on all discrete voxel data points to obtain feature values ​​of different discrete voxel data points.

[0028] Among them, the feature values ​​of different discrete voxel data points include the voxel coordinates of the discrete voxel data points in three-dimensional space and the maximum value;

[0029] In the 3D reconstruction software, a semantic segmentation module is introduced to predict the label of different discrete voxel data points. The label is the semantic probability distribution of different discrete voxel data points in soybean plants, and the instance center of different discrete voxel data points is determined at the same time.

[0030] Among them, the instance center of different discrete voxel data points is the true semantics of different discrete voxel data points. Based on the instance center of different discrete voxel data points, the offset vector from each discrete voxel data point to its corresponding instance center is calculated.

[0031] By combining the offset vector of each discrete voxel data point to its corresponding instance center and the feature values ​​of different discrete voxel data points, instance generation processing is performed on different discrete voxel data points. The instance generation processing requires filtering background points in the discrete voxel data points, and offset correction is performed on different discrete voxel data points according to the offset vector of each discrete voxel data point to its corresponding instance center. At the same time, feature similarity is calculated on different discrete voxel data points according to the feature values ​​of different discrete voxel data points to achieve clustering and grouping processing.

[0032] After clustering and grouping, leaf point clouds and stem point clouds of the target soybean plant were obtained.

[0033] Furthermore, in a preferred embodiment of the present invention, the step of measuring the phenotypic information of the leaves and stems of the target soybean plant based on the leaf point cloud and stem point cloud of the target soybean plant, combined with the PointNet Transform backbone network, specifically involves:

[0034] In the 3D reconstruction software, the PointNet Transform backbone network is introduced. Based on the PointNet Transform backbone network, combined with the leaf point cloud and stem point cloud of the target soybean plant, a point cloud measurement network architecture of the target soybean plant is constructed and calibrated as the target network architecture.

[0035] The method for constructing the target network architecture is to translate the center point of the leaf point cloud and stem point cloud of the target soybean plant to the origin of the three-dimensional space, and to perform point cloud coordinate alignment and point cloud coordinate transformation on the leaf point cloud and stem point cloud of the target soybean plant in the three-dimensional space.

[0036] Within the target network architecture, the leaf point cloud and stem point cloud of the target soybean plant are pooled. Through a self-attention mechanism, the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are captured and aggregated.

[0037] By combining the global feature vectors of leaf point clouds and stem point clouds of the target soybean plant, a neural network is introduced to train on multi-sample point cloud data and predict the phenotypic information of the leaves and stems of the target soybean plant.

[0038] Furthermore, in a preferred embodiment of the present invention, the step of introducing a neural network for training on multi-sample point cloud data and predicting the phenotypic information of the leaves and stems of the target soybean plant specifically involves:

[0039] The leaf point cloud and stem point cloud of 4000 target soybean plants were extracted to construct the target network architecture. In the target network architecture, the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plants were simulated and trained through a neural network.

[0040] The simulated training involves performing data regularization and convolution training on the global feature vector, and monitoring the vector value change amplitude of the global feature vector in real time during the convolution training process, and setting a maximum change amplitude.

[0041] If, during the convolution training process, the change in the vector value of the global feature vector does not exceed the maximum change for five consecutive rounds, then the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are simulated and trained, and the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are output.

[0042] The preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are adjusted a second time. The method of the second adjustment is to search the historical data network for the true phenotypic information range of the leaves and stems of the target soybean plant, and adjust the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant to the true phenotypic information range, so as to obtain the target predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant.

[0043] A second aspect of the present invention also provides a soybean plant phenotypic analysis system based on three-dimensional reconstruction. The soybean plant phenotypic analysis system includes a memory and a processor. The memory stores a soybean plant phenotypic analysis method. When the soybean plant phenotypic analysis method is executed by the processor, it performs the following steps:

[0044] A device is set up to acquire multi-view images of soybean plants, and multi-view images of soybean plants are taken to obtain multi-view images of the target soybean plants.

[0045] Image preprocessing is performed on multi-view images of the target soybean plant in the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and image feature data is extracted to construct a three-dimensional model of the soybean plant.

[0046] The three-dimensional model of soybean plant is processed into discrete point cloud voxels. The feature vectors of the voxelized soybean plant point cloud are extracted by combining the Sparse ConvUnet network and semantic segmentation is performed to obtain the leaf point cloud and stem point cloud of the target soybean plant.

[0047] Based on the leaf point cloud and stem point cloud of the target soybean plant, and combined with the PointNet Transform backbone network, the phenotypic information of the leaves and stems of the target soybean plant is measured.

[0048] This invention addresses the technical deficiencies in the prior art and offers the following advantages: By establishing a device for multi-view image acquisition of soybean plants, this invention acquires images of soybean plants from multiple perspectives and constructs an end-to-end soybean plant leaf phenotypic measurement model based on PointNet Transform. This model can directly predict key phenotypic features such as leaf area and leaf perimeter from high-precision leaf point cloud data. Before constructing the model, discrete point cloud voxelization processing is required to provide conditions for semantic segmentation. This invention achieves automated acquisition of soybean plant point clouds, automated segmentation of soybean plant point clouds, and automated measurement of soybean plant leaf point cloud phenotypic characteristics, providing a data foundation for soybean plant phenotypic analysis and enabling high-throughput, intelligent soybean plant phenotypic analysis. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained from these drawings without creative effort.

[0050] Figure 1 A flowchart of a soybean plant phenotypic analysis method based on three-dimensional reconstruction is shown;

[0051] Figure 2 A flowchart illustrating the method for obtaining phenotypic information of leaves and stems of a target soybean plant is shown.

[0052] Figure 3 A program view of a soybean plant phenotypic analysis system based on 3D reconstruction is shown.

[0053] Figure 4 A diagram of a soybean plant phenotypic analysis device based on three-dimensional reconstruction is shown. Detailed Implementation

[0054] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0055] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0056] Figure 1 A flowchart illustrating a soybean plant phenotypic analysis method based on 3D reconstruction is shown, including the following steps:

[0057] S102: Set up a device for acquiring multi-view images of soybean plants, and take multi-view pictures of soybean plants to obtain multi-view images of the target soybean plants.

[0058] S104: Perform image preprocessing on the multi-view images of the target soybean plant in the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and extract image feature data to construct a three-dimensional model of the soybean plant.

[0059] S106: Discrete point cloud voxelization is performed on the three-dimensional model of soybean plants. Feature vectors are extracted from the voxelized soybean plant point cloud using the Sparse ConvUnet network, and semantic segmentation is performed to obtain the leaf point cloud and stem point cloud of the target soybean plant.

[0060] S108: Based on the leaf point cloud and stem point cloud of the target soybean plant, and combined with the PointNet Transform backbone network, measure the phenotypic information of the leaves and stems of the target soybean plant.

[0061] Furthermore, in a preferred embodiment of the present invention, the device for setting up multi-view image acquisition of soybean plants and taking multi-view pictures of soybean plants to obtain multi-view images of the target soybean plants specifically includes:

[0062] A device for acquiring multi-view images of soybean plants, calibrated as a target image acquisition device, wherein the target image acquisition device includes an outer layer device and an inner layer device;

[0063] The outer layer of the target image acquisition device consists of an outer support frame and a light-shielding cloth, while the inner layer consists of 40 cameras, 40 industrial control computers, a switch, and a light source.

[0064] One camera is connected to an industrial control computer. The industrial control computer receives and temporarily stores the images captured by the camera. At the same time, the industrial control computer is connected to a switch via a network cable to form a local area network between the industrial control computer and the switch, which is designated as the target local area network.

[0065] Obtain a data analysis terminal device, connect the data analysis terminal device to the target local area network, so that the data analysis terminal device can extract and store photos taken by the camera that are temporarily stored in the industrial control computer;

[0066] Acquire soybean plants that need to be photographed from multiple perspectives, mark them as target soybean plants, place the target soybean plants in the target image acquisition device, and control all cameras to take multi-view images of the target soybean plants to obtain multi-view images of the target soybean plants. At the same time, store the obtained multi-view images of the target soybean plants in the data analysis terminal device.

[0067] In the process of controlling all cameras to capture multi-view images of the target soybean plant, it is necessary to close the shading cloth and ensure that 40 cameras are shooting at the same time.

[0068] It should be noted that this is a multi-view image acquisition device for soybean plants. The device mainly consists of an outer support frame and a light-blocking cloth; an inner layer containing 40 cameras, 40 industrial control computers, a switch, and lighting sources. The entire outer structure is constructed with a metal support frame, providing high strength, stability, and pressure resistance. The light-blocking cloth eliminates interference from external light and wind, ensuring an undisturbed shooting environment for the soybean plants inside. Each camera is connected to a Linux-based industrial control computer, which can receive shooting commands and temporarily store images. All the industrial control computers form a local area network (LAN) via network cables and a switch. A laptop connects to this LAN to control all the cameras for concurrent shooting. After shooting, the data is temporarily stored in the Linux industrial control computers and then transmitted back to the laptop's storage used for device control. Testing has shown that this device can capture 40 multi-view images of soybean plants within 0.02 seconds. It is crucial to ensure that all 40 cameras capture images simultaneously to prevent subsequent 3D reconstruction failures due to soybean plant movement. The user places the soybean plant to be photographed into the device, closes the light-blocking cloth, and issues a shooting command to the device via control software. Upon receiving the shooting command, 40 industrial control computers control their respective cameras to capture images and temporarily store the results in the industrial control computers. The results are then returned to the user's PC via the local area network. The obtained multi-view images of the soybean plant are then used for subsequent PGSR 3D reconstruction to obtain soybean plant point cloud data, thus completing the data acquisition process.

[0069] Furthermore, in a preferred embodiment of the present invention, the step of performing image preprocessing on the multi-view images of the target soybean plant within the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and simultaneously extracting image feature data to construct a three-dimensional model of the soybean plant, specifically involves:

[0070] Within the data analysis terminal device, the acquired multi-view images of the target soybean plant are subjected to color space conversion. The color space conversion involves grayscale processing of the multi-view images of the target soybean plant to obtain grayscale multi-view images of the target soybean plant.

[0071] Interpolation and size cropping are performed on the grayscale target soybean plant multi-view image to retain the soybean plant portion in the grayscale target soybean plant multi-view image, resulting in a grayscale cropped soybean plant multi-view image.

[0072] Perspective and geometric corrections are performed on the grayscale cropped soybean plant multi-view images, and wavelet transform algorithm is introduced to perform noise reduction filtering on the grayscale cropped soybean plant multi-view images.

[0073] Among them, the wavelet transform algorithm is to perform wavelet transform on the pixels in the grayscale cropped soybean plant multi-view image, filter out the pixels whose wavelet values ​​are not within the preset range after wavelet transform, and reconstruct the pixels to obtain the preprocessed target soybean plant multi-view image.

[0074] A threshold segmentation algorithm is introduced to perform threshold segmentation on the preprocessed target soybean plant multi-view images to obtain image feature data of the preprocessed target soybean plant multi-view images. Then, 3D reconstruction software is introduced to perform 3D reconstruction on the image feature data of the preprocessed target soybean plant multi-view images to obtain a 3D model of the soybean plant.

[0075] It should be noted that after acquiring multi-view images of the target soybean plant, preprocessing is necessary to prevent image blurring and excessive irrelevant features during the acquisition process. This includes adjusting image sharpness and cropping the image size to maximize the clarity of the multi-view images of the target soybean plant. This includes using wavelet transform algorithms for noise reduction filtering. Before constructing the 3D model, feature extraction is required to obtain the feature data of the target soybean plant. Therefore, threshold segmentation is used for feature extraction, and the 3D model of the soybean plant is constructed using 3D modeling software.

[0076] Furthermore, in a preferred embodiment of the present invention, the process of discretizing the three-dimensional model of the soybean plant into voxels, extracting feature vectors from the voxelized soybean plant point cloud using the Sparse ConvUnet network, and performing semantic segmentation to obtain the leaf point cloud and stem point cloud of the target soybean plant, specifically involves:

[0077] Within the 3D reconstruction software, a 3D space is acquired, and a soybean plant point cloud voxelization processing module is obtained. Based on the soybean plant point cloud voxelization processing module, the 3D model of the soybean plant is voxelized.

[0078] The module for voxelizing the three-dimensional model of soybean plants converts the three-dimensional model of soybean plants into discrete voxel data points and introduces a Sparse ConvUnet network. Through sparse convolution, it performs up-and-down sampling on all discrete voxel data points to obtain feature values ​​of different discrete voxel data points.

[0079] Among them, the feature values ​​of different discrete voxel data points include the voxel coordinates of the discrete voxel data points in three-dimensional space and the maximum value;

[0080] In the 3D reconstruction software, a semantic segmentation module is introduced to predict the label of different discrete voxel data points. The label is the semantic probability distribution of different discrete voxel data points in soybean plants, and the instance center of different discrete voxel data points is determined at the same time.

[0081] Among them, the instance center of different discrete voxel data points is the true semantics of different discrete voxel data points. Based on the instance center of different discrete voxel data points, the offset vector from each discrete voxel data point to its corresponding instance center is calculated.

[0082] By combining the offset vector of each discrete voxel data point to its corresponding instance center and the feature values ​​of different discrete voxel data points, instance generation processing is performed on different discrete voxel data points. The instance generation processing requires filtering background points in the discrete voxel data points, and offset correction is performed on different discrete voxel data points according to the offset vector of each discrete voxel data point to its corresponding instance center. At the same time, feature similarity is calculated on different discrete voxel data points according to the feature values ​​of different discrete voxel data points to achieve clustering and grouping processing.

[0083] After clustering and grouping, leaf point clouds and stem point clouds of the target soybean plant were obtained.

[0084] It should be noted that the soybean point cloud segmentation module aims to segment the leaf and stem point clouds of soybean plants. First, the point cloud is voxelized. Then, features are extracted from the soybean point cloud using the Sparse ConvUnet network. The feature vectors are then fed into the offset vector and semantic label predicted by the semantic segmentation module, respectively, to obtain the instance segmentation result of the soybean plant point cloud. The Sparse ConvUnet network is a deep learning architecture specifically designed for efficiently processing sparse data (especially 3D point cloud data). It is used to extract feature values ​​from discrete voxel data points in the soybean point cloud. The Sparse ConvUnet network replaces the convolutional layers in the traditional feature extraction module with coefficient convolutional layers to improve the efficiency of processing large-scale 3D data and effectively learn local and global features in 3D space. Semantic segmentation involves splitting the soybean plant to determine the positions of the stem and leaves. The semantic segmentation step involves determining the bias of each discrete voxel data point to its corresponding instance center by analyzing its offset vector. Then, the corresponding discrete voxel data points are connected to achieve semantic determination for different data points. First, the point cloud is voxelized to facilitate feature extraction from the Sparse ConvUnet network. The voxelization process is illustrated below: Assuming the maximum coordinate of the point cloud is... The minimum coordinates are The side length of the voxel is For each point in the point cloud The index in the voxel grid can be calculated using the following formula. :

[0085] ,

[0086] ,

[0087]

[0088] The voxelized points are then fed into Sparse ConvUnet to extract their features. The outputs are then fed into the semantic segmentation module to predict the label and offset vector of each point from its instance center. The loss function of the entire network is calculated as follows:

[0089]

[0090] in The calculation formula is as follows: It is the number of points. It is the predicted semantic label of the i-th point. It is the true semantic label of the i-th point. Represents the cross-entropy loss function:

[0091]

[0092] in The calculation formula is as follows: It is the offset vector predicted by the model, representing the vector from the point to the geometric center of its instance. This is the actual offset corresponding to that point. This represents the L1 norm (sum of absolute values). It is an indicator function, when The value is 1 if it belongs to a certain instance, and 0 otherwise.

[0093]

[0094] Figure 2 A flowchart illustrating a method for obtaining phenotypic information of leaves and stems of a target soybean plant is shown, including the following steps:

[0095] S202: Based on the leaf point cloud and stem point cloud of the target soybean plant, combined with the PointNet Transform backbone network, measure the phenotypic information of the leaves and stems of the target soybean plant.

[0096] S204: Introduce a neural network for training on multi-sample point cloud data and predict the phenotypic information of the leaves and stems of the target soybean plant.

[0097] Furthermore, in a preferred embodiment of the present invention, the step of measuring the phenotypic information of the leaves and stems of the target soybean plant based on the leaf point cloud and stem point cloud of the target soybean plant, combined with the PointNet Transform backbone network, specifically involves:

[0098] In the 3D reconstruction software, the PointNet Transform backbone network is introduced. Based on the PointNet Transform backbone network, combined with the leaf point cloud and stem point cloud of the target soybean plant, a point cloud measurement network architecture of the target soybean plant is constructed and calibrated as the target network architecture.

[0099] The method for constructing the target network architecture is to translate the center point of the leaf point cloud and stem point cloud of the target soybean plant to the origin of the three-dimensional space, and to perform point cloud coordinate alignment and point cloud coordinate transformation on the leaf point cloud and stem point cloud of the target soybean plant in the three-dimensional space.

[0100] Within the target network architecture, the leaf point cloud and stem point cloud of the target soybean plant are pooled. Through a self-attention mechanism, the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are captured and aggregated.

[0101] By combining the global feature vectors of leaf point clouds and stem point clouds of the target soybean plant, a neural network is introduced to train on multi-sample point cloud data and predict the phenotypic information of the leaves and stems of the target soybean plant.

[0102] It should be noted that in the soybean leaf phenotypic analysis module, an end-to-end deep point cloud learning framework is constructed based on Point Transformer. This framework can directly predict key phenotypic features such as leaf area from point cloud data. The network model designed here uses Point Transformer as the backbone to build an efficient regression model for predicting leaf phenotypic traits. This model introduces a feature multidimensionality module to extract local features, while simultaneously capturing global features through a self-attention mechanism, thereby effectively combining and associating global and local information. Furthermore, basic phenotypic traits such as leaf length and width are used as important prior information, and this information is used to constrain the model to improve the accuracy of phenotypic trait prediction. Point Transformer is mainly used to solve the problem of geometric transformation invariance of point cloud data. If traditional networks directly process the original coordinates, they may misjudge different poses of the same object due to coordinate changes. The Point Transformer network can ensure that the features learned by the network do not depend on the absolute spatial coordinates of the input point cloud.

[0103] Furthermore, in a preferred embodiment of the present invention, the step of introducing a neural network for training on multi-sample point cloud data and predicting the phenotypic information of the leaves and stems of the target soybean plant specifically involves:

[0104] The leaf point cloud and stem point cloud of 4000 target soybean plants were extracted to construct the target network architecture. In the target network architecture, the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plants were simulated and trained through a neural network.

[0105] The simulated training involves performing data regularization and convolution training on the global feature vector, and monitoring the vector value change amplitude of the global feature vector in real time during the convolution training process, and setting a maximum change amplitude.

[0106] If, during the convolution training process, the change in the vector value of the global feature vector does not exceed the maximum change for five consecutive rounds, then the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are simulated and trained, and the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are output.

[0107] The preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are adjusted a second time. The method of the second adjustment is to search the historical data network for the true phenotypic information range of the leaves and stems of the target soybean plant, and adjust the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant to the true phenotypic information range, so as to obtain the target predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant.

[0108] It should be noted that this invention uses PointNet Transform as the backbone network to predict phenotypic information such as leaf point cloud area and perimeter. Considering that training the neural network requires a large amount of data, the network can be pre-trained by simulating the generation of 4000 leaf point cloud data with random perturbations, and then fine-tuned using real point cloud data. This allows for the measurement of phenotypic information such as leaf area, leaf length, leaf width, and leaf perimeter. Simulated training uses convolutional training of the neural network. If the variation amplitude of the global feature vector values ​​does not exceed the maximum variation amplitude within the specified number of epochs, it proves that the simulated training is in a stable state. At this point, the preliminary predicted values ​​of the phenotypic information of the leaf and stem point clouds of the target soybean plant are output, and fine-tuning is performed to ensure they are within a suitable and acceptable range, thus obtaining the target predicted values ​​of the phenotypic information of the leaf and stem point clouds of the target soybean plant.

[0109] like Figure 3 As shown, a second aspect of the present invention also provides a soybean plant phenotypic analysis system based on three-dimensional reconstruction. The soybean plant phenotypic analysis system includes a memory 31 and a processor 32. The memory 31 stores a soybean plant phenotypic analysis method. When the soybean plant phenotypic analysis method is executed by the processor 32, the following steps are implemented:

[0110] A device is set up to acquire multi-view images of soybean plants, and multi-view images of soybean plants are taken to obtain multi-view images of the target soybean plants.

[0111] Image preprocessing is performed on multi-view images of the target soybean plant in the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and image feature data is extracted to construct a three-dimensional model of the soybean plant.

[0112] The three-dimensional model of soybean plant is processed into discrete point cloud voxels. The feature vectors of the voxelized soybean plant point cloud are extracted by combining the Sparse ConvUnet network and semantic segmentation is performed to obtain the leaf point cloud and stem point cloud of the target soybean plant.

[0113] Based on the leaf point cloud and stem point cloud of the target soybean plant, and combined with the PointNet Transform backbone network, the phenotypic information of the leaves and stems of the target soybean plant is measured.

[0114] like Figure 4 As shown, the third aspect of the present invention also provides a device diagram of a soybean plant phenotypic analysis device based on three-dimensional reconstruction.

[0115] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for phenotyping soybean plants based on three-dimensional reconstruction, characterized in that, Includes the following steps: A device is set up to acquire multi-view images of soybean plants, and multi-view images of soybean plants are taken to obtain multi-view images of the target soybean plants. Image preprocessing is performed on multi-view images of the target soybean plant in the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and image feature data is extracted to construct a three-dimensional model of the soybean plant. The three-dimensional model of soybean plants is processed into discrete point cloud voxels. The feature vectors of the voxelized soybean plant point cloud are extracted by combining the Sparse ConvUnet network and semantic segmentation is performed. The semantic segmentation process includes instance generation for different discrete voxel data points. This involves filtering background points from the discrete voxel data points, correcting the offset of each discrete voxel data point to its corresponding instance center based on the offset vector, and calculating the feature similarity of different discrete voxel data points based on their feature values ​​to achieve clustering and grouping. After clustering and grouping, leaf point clouds and stem point clouds of the target soybean plant were obtained. Based on the leaf point cloud and stem point cloud of the target soybean plant, and combined with the PointNet Transform backbone network, a point cloud measurement network architecture for the target soybean plant is constructed. This involves translating the center point of the leaf point cloud and stem point cloud of the target soybean plant to the origin of the three-dimensional space, and performing point cloud coordinate alignment and point cloud coordinate transformation on the leaf point cloud and stem point cloud of the target soybean plant in the three-dimensional space. Meanwhile, within the target network architecture, the leaf point cloud and stem point cloud of the target soybean plant are pooled, and the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are captured and aggregated through a self-attention mechanism. By combining the global feature vectors of leaf point clouds and stem point clouds of the target soybean plant, a neural network is introduced to train on multi-sample point cloud data and predict the phenotypic information of the leaves and stems of the target soybean plant.

2. The soybean plant phenotyping method based on three-dimensional reconstruction according to claim 1, characterized in that, The device is configured to acquire multi-view images of soybean plants and capture multi-view images of the soybean plants to obtain multi-view images of the target soybean plants, specifically as follows: A device for acquiring multi-view images of soybean plants, calibrated as a target image acquisition device, wherein the target image acquisition device includes an outer layer device and an inner layer device; The outer layer of the target image acquisition device consists of an outer support frame and a light-shielding cloth, while the inner layer consists of 40 cameras, 40 industrial control computers, a switch, and a light source. One camera is connected to an industrial control computer. The industrial control computer receives and temporarily stores the images captured by the camera. At the same time, the industrial control computer is connected to a switch via a network cable to form a local area network between the industrial control computer and the switch, which is designated as the target local area network. Obtain a data analysis terminal device, connect the data analysis terminal device to the target local area network, so that the data analysis terminal device can extract and store photos taken by the camera that are temporarily stored in the industrial control computer; Acquire soybean plants that need to be photographed from multiple perspectives, mark them as target soybean plants, place the target soybean plants in the target image acquisition device, and control all cameras to take multi-view images of the target soybean plants to obtain multi-view images of the target soybean plants. At the same time, store the obtained multi-view images of the target soybean plants in the data analysis terminal device. In the process of controlling all cameras to capture multi-view images of the target soybean plant, it is necessary to close the shading cloth and ensure that 40 cameras are shooting at the same time.

3. The soybean plant phenotyping method based on three-dimensional reconstruction according to claim 1, characterized in that, The step involves preprocessing multi-view images of the target soybean plant within the data analysis terminal device to obtain preprocessed multi-view images of the target soybean plant, and simultaneously extracting image feature data to construct a three-dimensional model of the soybean plant. Specifically: Within the data analysis terminal device, the acquired multi-view images of the target soybean plant are subjected to color space conversion. The color space conversion involves grayscale processing of the multi-view images of the target soybean plant to obtain grayscale multi-view images of the target soybean plant. Interpolation and size cropping are performed on the grayscale target soybean plant multi-view image to retain the soybean plant portion in the grayscale target soybean plant multi-view image, resulting in a grayscale cropped soybean plant multi-view image. Perspective distortion geometric correction is performed on the grayscale cropped soybean plant multi-view image, and wavelet transform algorithm is introduced to perform noise reduction filtering on the grayscale cropped soybean plant multi-view image. Among them, the wavelet transform algorithm is to perform wavelet transform on the pixels in the grayscale cropped soybean plant multi-view image, filter out the pixels whose wavelet values ​​are not within the preset range after wavelet transform, and reconstruct the pixels to obtain the preprocessed target soybean plant multi-view image. A threshold segmentation algorithm is introduced to perform threshold segmentation on the preprocessed target soybean plant multi-view images to obtain image feature data of the preprocessed target soybean plant multi-view images. Then, 3D reconstruction software is introduced to perform 3D reconstruction on the image feature data of the preprocessed target soybean plant multi-view images to obtain a 3D model of the soybean plant.

4. The soybean plant phenotyping method based on three-dimensional reconstruction according to claim 1, characterized in that, The process involves discretizing the three-dimensional model of the soybean plant into voxels, extracting feature vectors from the voxelized soybean plant point cloud using the Sparse ConvUnet network, and performing semantic segmentation to obtain the leaf and stem point clouds of the target soybean plant. Specifically: Within the 3D reconstruction software, a 3D space is acquired, and a soybean plant point cloud voxelization processing module is obtained. Based on the soybean plant point cloud voxelization processing module, the 3D model of the soybean plant is voxelized. The module for voxelizing the three-dimensional model of soybean plants converts the three-dimensional model of soybean plants into discrete voxel data points and introduces a Sparse ConvUnet network. Through sparse convolution, it performs up-and-down sampling on all discrete voxel data points to obtain feature values ​​of different discrete voxel data points. Among them, the feature values ​​of different discrete voxel data points include the voxel coordinates of the discrete voxel data points in three-dimensional space and the maximum value; In the 3D reconstruction software, a semantic segmentation module is introduced to predict the label of different discrete voxel data points. The label is the semantic probability distribution of different discrete voxel data points in soybean plants, and the instance center of different discrete voxel data points is determined at the same time. Among them, the instance center of different discrete voxel data points is the true semantics of different discrete voxel data points. Based on the instance center of different discrete voxel data points, the offset vector from each discrete voxel data point to its corresponding instance center is calculated. By combining the offset vector from each discrete voxel data point to its corresponding instance center, and the feature values ​​of different discrete voxel data points, instance generation processing is performed on different discrete voxel data points to obtain the leaf point cloud and stem point cloud of the target soybean plant.

5. The soybean plant phenotyping method based on three-dimensional reconstruction according to claim 1, wherein, The step involves measuring the phenotypic information of the leaves and stems of the target soybean plant based on the leaf point cloud and stem point cloud, combined with the PointNet Transform backbone network. Specifically: Within the 3D reconstruction software, the PointNet Transform backbone network is introduced. Based on the PointNet Transform backbone network, and combined with the leaf point cloud and stem point cloud of the target soybean plant, a point cloud measurement network architecture for the target soybean plant is constructed and calibrated as the target network architecture.

6. The soybean plant phenotyping method based on three-dimensional reconstruction according to claim 1, wherein, The process of introducing a neural network for training on multi-sample point cloud data and predicting the phenotypic information of the leaves and stems of the target soybean plant is as follows: The leaf point cloud and stem point cloud of 4000 target soybean plants were extracted to construct the target network architecture. In the target network architecture, the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plants were simulated and trained through a neural network. The simulated training involves performing data regularization and convolution training on the global feature vector, and monitoring the vector value change amplitude of the global feature vector in real time during the convolution training process, and setting a maximum change amplitude. If, during the convolution training process, the change in the vector value of the global feature vector does not exceed the maximum change for five consecutive rounds, then the global feature vectors of the leaf point cloud and stem point cloud of the target soybean plant are simulated and trained, and the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are output. The preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant are adjusted a second time. The method of the second adjustment is to search the historical data network for the true phenotypic information range of the leaves and stems of the target soybean plant, and adjust the preliminary predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant to the true phenotypic information range, so as to obtain the target predicted values ​​of the phenotypic information of the leaf point cloud and stem point cloud of the target soybean plant.

7. A soybean plant phenotypic analysis system based on three-dimensional reconstruction, characterized in that, The soybean plant phenotypic analysis system includes a memory and a processor. The memory stores a soybean plant phenotypic analysis method program. When the soybean plant phenotypic analysis method program is executed by the processor, the steps of the soybean plant phenotypic analysis method as described in any one of claims 1-6 are implemented.

8. A soybean plant phenotypic analysis device based on three-dimensional reconstruction, characterized in that, When the soybean plant phenotyping device is running, it implements the method steps as described in any one of claims 1-6, specifically as follows: The image capturing module is used to capture multi-view images of soybean plants through a camera in the soybean plant multi-view image acquisition device. The data storage module is used to store multi-view images of soybean plants captured by the computer and switch into the data processing terminal equipment. The model building module is used to build a three-dimensional model of the preprocessed soybean plant multi-view image and to perform instance segmentation of the three-dimensional model of the soybean plant through semantic segmentation to obtain the leaf point cloud and stem point cloud of the soybean plant. The model training module is used to predict leaf phenotypic information from the leaf point cloud and stem point cloud of soybean plants.

Images