Phenotype acquisition method and device based on neural radiation field

Abstract

The present invention provides a phenotypic acquisition method and device based on neural radiation fields, relating to the field of agricultural information technology. The method comprises: obtaining a sequence of images of the plant to be reconstructed, captured by a camera at multiple viewing angles; generating internal and external parameters of the camera based on the image sequence using a large image segmentation model and a motion structure recovery algorithm, which serve as a training dataset; using the training dataset to train a 3D plant reconstruction model based on a neural radiation field network framework; the trained 3D plant reconstruction model generates an implicit 3D scene representation based on the image sequence; the model includes a multi-resolution hash coding module, a volume density prediction unit based on an ECA attention mechanism, a color prediction unit based on a multi-layer perceptron, and a volume rendering module; and extracting phenotypic parameters corresponding to the plant to be reconstructed based on the implicit 3D scene representation. The present invention achieves high crop reconstruction accuracy and significantly reduces crop reconstruction time.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information technology, and in particular to a phenotype acquisition method and device based on neural radiation fields. Background Art

[0002] The three-dimensional structure of plants is the most intuitive way to obtain plant phenotypic information and is crucial for assessing plant growth dynamics, functional performance, and resource utilization efficiency. Therefore, the ability to quickly and accurately obtain plant three-dimensional structure is of great research significance for conducting high-throughput plant phenotyping analysis, promoting genetic research, and optimizing breeding strategies.

[0003] Current methods for reconstructing plant 3D structures are primarily categorized as sensor-based and photogrammetry-based. Sensor-based methods primarily utilize devices such as 3D scanners to directly acquire high-precision 3D point clouds of plants. However, this approach is prohibitively expensive, and the reconstruction process is susceptible to environmental influences. Photogrammetry-based methods, based on camera geometry, employ structure-from-motion-multiview stereo (SFM-MVS) algorithms to gradually restore the 3D scene. While inexpensive, this approach suffers from a lengthy dense reconstruction process and poor reconstruction of low-texture areas within the 3D scene, which, to some extent, limits its application in 3D plant phenotyping. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a phenotype acquisition method and device based on neural radiation field, which has high crop reconstruction accuracy and can significantly shorten the crop reconstruction time.

[0005] In a first aspect, an embodiment of the present invention provides a method for acquiring a phenotype based on a neural radiation field, comprising:

[0006] Obtain an image sequence of the plant to be reconstructed, captured by a camera at multiple viewing angles. Generate internal and external parameters of the camera based on the image sequence using a large image segmentation model and a structure-from-motion algorithm. The combination of these parameters and the image sequence is used as a training dataset.

[0007] A training dataset is used to train a 3D plant reconstruction model based on a neural radiation field network framework. The trained 3D plant reconstruction model generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence. The 3D plant reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron.

[0008] The phenotypic parameters corresponding to the plant to be reconstructed are extracted based on the implicit three-dimensional scene representation.

[0009] In one embodiment, the internal and external parameters of the camera are generated based on the image sequence using a large image segmentation model and a structure-from-motion algorithm, including:

[0010] The target plant area in each real plant image in the image sequence is obtained through the image segmentation model to obtain an image mask sequence;

[0011] The image mask sequence is processed using the motion structure recovery algorithm to obtain the internal and external parameters of the shooting device.

[0012] In one embodiment, the plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence, including:

[0013] Through the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point;

[0014] Through the volume density and color extraction module, based on the high-dimensional vector of the pixel points, the volume density prediction value and color prediction value corresponding to each pixel point are extracted;

[0015] Through the volume rendering module, the volume density prediction value and the color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0016] In one embodiment, the volume density and color extraction module extracts the volume density prediction value and color prediction value corresponding to each pixel based on the high-dimensional vector of the pixel point, including:

[0017] Through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel point;

[0018] A color prediction unit based on a multi-layer perceptron is used to generate a color prediction value corresponding to each pixel based on the volume density prediction value corresponding to each pixel.

[0019] In one embodiment, the volume density prediction unit includes a plurality of fully connected multi-layer perceptrons and an ECA attention network, wherein the ECA attention network is embedded after the first fully connected multi-layer perceptron;

[0020] The color prediction unit consists of multiple fully connected multi-layer perceptrons.

[0021] In one embodiment, extracting phenotypic parameters corresponding to a plant to be reconstructed based on an implicit three-dimensional scene representation includes:

[0022] Using voxel-level reconstruction algorithms, the implicit 3D scene representation is converted into a visible 3D mesh representation corresponding to the plant to be reconstructed.

[0023] A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0024] In one embodiment, extracting a region of interest corresponding to the plant to be reconstructed from the three-dimensional mesh expression, and extracting phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest, includes:

[0025] Perform coordinate correction and scale restoration on the three-dimensional mesh expression to obtain the target three-dimensional mesh expression;

[0026] Extracting a region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression; wherein the region of interest includes the plant point cloud and the leaf organ point cloud corresponding to each leaf of the plant to be reconstructed;

[0027] The difference between the highest and lowest points of the plant point cloud in the vertical direction is taken as the plant height; and the maximum distance between any two points in the horizontal direction of the plant point cloud is taken as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is taken as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is taken as the leaf width;

[0028] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

[0029] In a second aspect, an embodiment of the present invention further provides a phenotype acquisition device based on a neural radiation field, comprising:

[0030] The data acquisition module is used to obtain a sequence of images of the plant to be reconstructed, which is captured by a camera at multiple viewing angles. Based on the image sequence, the internal and external parameters of the camera are generated using a large image segmentation model and a motion structure recovery algorithm. The combination of the internal and external parameters and the image sequence is used as a training dataset.

[0031] A 3D reconstruction module is used to train a 3D plant reconstruction model based on a neural radiation field network framework using a training dataset. The trained 3D plant reconstruction model generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence. The 3D plant reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron.

[0032] The phenotypic acquisition module is used to extract the phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0035] An embodiment of the present invention provides a phenotypic acquisition method and device based on a neural radiation field. First, an image sequence of a plant to be reconstructed, which is captured by a shooting device in multiple viewing directions, is acquired. Internal and external parameters of the shooting device are generated based on the image sequence through a large image segmentation model and a motion structure recovery algorithm. The combination of the internal and external parameters and the image sequence is used as a training data set. Then, a three-dimensional reconstruction model of a plant based on a neural radiation field network framework is trained using the training data set. The three-dimensional reconstruction model of the plant obtained through training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence. The three-dimensional reconstruction model of the plant includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron. Finally, the phenotypic parameters corresponding to the plant to be reconstructed are extracted based on the implicit three-dimensional scene representation. The above method designs a plant 3D reconstruction model based on the neural radiation field network framework, trains the plant 3D reconstruction model using a training data set constructed from image sequences, and uses the trained plant 3D reconstruction model to generate an implicit 3D mesh expression of the plant to be reconstructed, and further extracts the corresponding phenotypic parameters of the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the crop reconstruction time while ensuring the accuracy of crop reconstruction.

[0036] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic flow chart of a method for acquiring a phenotype based on a neural radiation field provided in an embodiment of the present invention;

[0040] Figure 2 An overall flow chart of a phenotype acquisition method based on neural radiation fields provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the principle of NeRF-Plant plant reconstruction provided by an embodiment of the present invention;

[0042] Figure 4 A NeRF-Plant network architecture diagram provided by an embodiment of the present invention;

[0043] Figure 5 A plant comparison image reconstructed based on NeRF-Plant and SFM-MVS methods provided in an embodiment of the present invention;

[0044] Figure 6 A time comparison chart based on NeRF-Plant and SFM-MVS methods provided in an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of phenotypic parameter extraction provided by an embodiment of the present invention;

[0046] Figure 8 A comparison chart of phenotypic parameters extracted from plants reconstructed based on NeRF-Plant and manually measured values ​​provided in an embodiment of the present invention;

[0047] Figure 9 A schematic structural diagram of a phenotype acquisition device based on neural radiation field provided by an embodiment of the present invention;

[0048] Figure 10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] Currently, using 3D scanners and other devices to obtain 3D structural information about plants is plagued by high costs and low reconstruction efficiency. Although structure-from-motion (SMO) multi-view stereo vision algorithms are widely used for 3D plant reconstruction, they still suffer from issues such as lengthy dense reconstruction processes and poor reconstruction of low-texture areas within 3D scenes, limiting their application in subsequent phenotyping. Therefore, a non-contact, fast, and accurate 3D plant reconstruction method is urgently needed.

[0051] Neural Radiance Fields (NeRF) is an implicit 3D scene representation based on neural networks. It synthesizes new perspective information from existing image sequences through learning methods, boasting fast inference speed and realistic reconstruction. Currently, NeRF research focuses primarily on large-scale scene reconstruction and visual effects generation, but research on capturing the 3D structure of plants is scarce. Therefore, embodiments of the present invention utilize improved NeRFs to achieve 3D plant reconstruction and phenotypic extraction.

[0052] Based on this, the present invention provides a phenotype acquisition method and device based on neural radiation field, which has high crop reconstruction accuracy and can significantly shorten the crop reconstruction time.

[0053] To facilitate understanding of this embodiment, a phenotype acquisition method based on neural radiation field disclosed in an embodiment of the present invention is first described in detail. Figure 1 The flowchart of a method for acquiring a phenotype based on a neural radiation field is shown, and the method mainly includes the following steps S102 to S106:

[0054] Step S102: Obtain an image sequence of the plant to be reconstructed captured by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through an image segmentation model and a motion structure recovery algorithm, and use the combination of the internal and external parameters and the image sequence as a training data set.

[0055] Among them, the shooting device is a device with image acquisition function, such as a camera, and the image sequence contains multiple real plant images. The training data set includes the image sequence and the internal and external parameters of the shooting device, which are determined based on the image sequence.

[0056] In one example, the plant to be reconstructed can be placed on a grid calibration paper with a side length of 2 cm. The plant to be reconstructed can be recorded multiple times around it. During the recording process, uniform circular motion should be maintained as much as possible, and the viewfinder should always contain the complete plant to be reconstructed. The captured video is then framed to obtain an image sequence.

[0057] In one example, a large image segmentation model was used to obtain the target plant area in each real plant image in the image sequence, and an image mask sequence was obtained. The image mask sequence was processed using SFM (Structure from Motion) to obtain the internal and external parameters of the shooting device. The combination of the internal and external parameters and the image sequence was used as a training data set for a plant 3D reconstruction model based on the neural radiation field network framework.

[0058] In step S104, the training data set is used to train the plant 3D reconstruction model based on the neural radiation field network framework, and the plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence.

[0059] The plant 3D reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron. In one example, sampling point information is generated based on a training dataset and input into the multi-resolution hash module. Each sampling point and viewing direction in the three-dimensional space is encoded into a high-dimensional vector through a hash coding function, which serves as the input to the volume density and color extraction module. The volume density and color extraction module mainly includes two MLP (Multilayer Perceptron) networks, which are used to extract the volume density prediction value and color prediction value of the sampling point, respectively. The volume rendering module generates a synthetic image corresponding to each viewing direction based on the volume density prediction value and color prediction value. The difference between the synthetic object and the corresponding real plant image is then used to train the plant 3D reconstruction model. The image sequence is input into the trained plant 3D reconstruction model to obtain an implicit 3D scene representation corresponding to the plant to be reconstructed.

[0060] Step S106 : extracting phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0061] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width. In one example, an implicit 3D scene representation can be converted into a displayed 3D mesh representation. After coordinate correction and scale restoration of the 3D mesh representation, a region of interest corresponding to the plant to be reconstructed is extracted. The region of interest includes a plant point cloud and a leaf organ point cloud corresponding to each leaf in the plant to be reconstructed. The plant height and plant crown width are then determined based on the plant point cloud, and the leaf length and leaf width corresponding to each leaf are determined based on the leaf organ point cloud corresponding to each leaf.

[0062] The phenotypic acquisition method based on neural radiation field provided by the embodiment of the present invention designs a plant three-dimensional reconstruction model based on the neural radiation field network framework, trains the plant three-dimensional reconstruction model using a training data set constructed using image sequences, and uses the trained plant three-dimensional reconstruction model to generate an implicit three-dimensional mesh expression of the plant to be reconstructed, and further extracts the corresponding phenotypic parameters of the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the time of crop reconstruction while ensuring the accuracy of crop reconstruction.

[0063] For ease of understanding, the embodiments of the present invention provide Figure 2 The overall flow chart of a phenotypic acquisition method based on neural radiation field is shown, including: data set preparation: surround video acquisition, video frame processing, camera parameter acquisition; NeRF-Plant (i.e., a 3D plant reconstruction model based on the neural radiation field network framework) training: hash coding, feature extraction, volume rendering; coordinate correction: coordinate transformation, scale recovery; phenotypic parameter extraction: plant height, crown width, leaf length, leaf width; accuracy and efficiency evaluation: point-to-point distance, reconstruction time, phenotypic parameters.

[0064] Among them, see Figure 3 A schematic diagram of the principle of NeRF-Plant plant reconstruction is shown. Figure 3 (a) shows how the camera position map is obtained by 3D reconstruction of the image sequence, which is related to the internal and external parameters of the camera. NeRF-Plant is used to output the synthetic image at each viewing direction, and the Marching Cube (voxel-level reconstruction algorithm) is used to obtain the displayed 3D mesh representation. Figure 3 (b) in the figure illustrates the training process of NeRF-Plant.

[0065] Based on this, an embodiment of the present invention provides a specific implementation of a phenotype acquisition method based on neural radiation field.

[0066] Regarding the aforementioned step S102, the embodiment of the present invention provides a specific process for preparing a data set for neural radiation field training, which specifically includes the following sub-steps 1a to 1c:

[0067] Step 1a: Place the plant to be reconstructed on a 2cm square of calibration paper. Hold your phone and record three circles around the subject, maintaining a constant circular motion and ensuring the entire plant remains in the frame. Each circle should be recorded at angles of approximately 15°, 45°, and 75° to the ground.

[0068] Step 1b: Frame the three videos and save one image every 1 second. The final number of images obtained is about 80-130. The collection of these images is the aforementioned image sequence.

[0069] Step 1c: Use the Segment Anything Model (SAM) to automatically capture the target plant region in the image sequence and obtain the corresponding image mask sequence. This reduces the amount of surrounding scene information in the subsequently generated plant point cloud, reducing the burden on subsequent point cloud processing.

[0070] Step 1c: Process the image mask sequence using the SFM module to obtain the camera's intrinsic and extrinsic parameters. The combination of the camera's intrinsic and extrinsic parameters and the image sequence is the dataset used for neural radiance field training.

[0071] Before explaining the aforementioned step S104, the embodiment of the present invention first provides a specific implementation of a plant 3D reconstruction model (NeRF-Plant) based on a neural radiation field network framework. Figure 4 The NeRF-Plant network architecture diagram shown in the figure shows that the NeRF-Plant network consists of three parts: a multi-resolution hash encoding module, a volume density and color extraction module (including a volume density prediction unit density network and a color prediction unit color network), and a volume rendering module.

[0072] Among them, (1) Multi-resolution hash coding module: Each sampling point and viewing direction in the three-dimensional space is encoded into a high-dimensional vector through a hash coding function. This method is more compact and has higher parallelism than the position coding in NeRF, and the training and inference speed of the NeRF network will also be faster. (2) Volume density and color extraction module: It is mainly used to predict the volume density (density) and color (color) of each sampling point. Volume density represents the degree of occupation of the point in space, and color represents the RGB value of the point. The volume density and color extraction module mainly contains two MLP networks. The first network is a volume density prediction unit that integrates the ECA (Efficient Channel Attention) attention mechanism, and the other is a color prediction unit based on MLP. Among them, the ECA attention module enables the model to focus more on information useful for volume density prediction by adaptively adjusting the weight of each channel. (3) Volume rendering module: The predicted volume density and color are integrated along the ray to synthesize the image at each viewpoint. Specifically, through ray projection, the volume density and color of each sampling point are calculated, and the final pixel color value is calculated using the volume rendering equation. The calculation formula of the volume rendering module is as follows:

[0073]

[0074] Where C(r) is the color value of the pixel on the ray, T(t) is the transmittance, σ(t) is the volume density, and c(t) is the color value.

[0075] Based on the aforementioned network structure, an embodiment of the present invention provides a specific implementation method for training a plant 3D reconstruction model, as shown in steps 2a to 2c below:

[0076] Step 2a: Generate sampling point information based on the training data set. In one example, the sampling point information is generated based on the viewing angle of the shooting device and the collected image sequence information.

[0077] Step 2b: Output a first synthetic image corresponding to each viewing direction based on the sampling point information through a plant three-dimensional reconstruction model based on a neural radiation field network framework.

[0078] In the specific implementation, it includes:

[0079] 1) Using a multi-resolution hash coding module, the sampling point information is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point. In one example, using a multi-resolution hash coding module to map the sampling point information to a high-dimensional space, the coordinate information of the sampling point is increased to 63 dimensions, and the camera view angle information is increased to 27 dimensions.

[0080] 2) Through the volume density and color extraction module, based on the high-dimensional vector of the sampling points, the volume density prediction value and color prediction value corresponding to each sampling point are extracted. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron. In one embodiment, the volume density prediction unit includes multiple fully connected multi-layer perceptrons and an ECA attention network, and the ECA attention network is embedded after the first fully connected multi-layer perceptron; the color prediction unit includes multiple fully connected multi-layer perceptrons. Please continue to refer to Figure 4 , Figure 4 The figure also shows that the volume density prediction network consists of five fully connected MLP networks and one ECA attention network, where the ECA attention network is embedded after the first fully connected MLP network. The color prediction network consists of three fully connected MLP networks.

[0081] Based on this, first, a volume density prediction unit based on the ECA attention mechanism is used to generate a volume density prediction value corresponding to each sampling point based on the high-dimensional vector of the sampling point; then, a color prediction unit based on the multi-layer perceptron is used to generate a color prediction value corresponding to each sampling point based on the volume density prediction value corresponding to each sampling point.

[0082] In practical applications, the ECA attention network calculates global statistics for each channel in the feature map and uses adaptive one-dimensional convolution kernels to capture inter-channel dependencies. Sigmoid activation is then used to generate channel attention weights. These weights are then applied to the original feature map, enhancing important feature channels and suppressing unimportant channels, thereby improving the network's ability to extract key information. The final network output is the volume density prediction and RGB color prediction corresponding to the sampling point.

[0083] 3) The volume density prediction value and the color prediction value are integrated along the ray through the volume rendering module to synthesize a first synthetic image corresponding to each viewing angle direction.

[0084] In step 2c, the network parameters of the 3D plant reconstruction model are adjusted based on the first synthetic image and the real plant image corresponding to the first synthetic image in the image sequence, thereby training the 3D plant reconstruction model. In one example, the difference between the synthetic image and the real plant image is calculated, and the parameters of the MLP network are continuously adjusted using a backpropagation algorithm to minimize the loss function and gradually optimize the 3D scene representation. The loss function optimization is iteratively performed until the model converges and an implicit 3D scene representation is obtained.

[0085] After training the plant 3D reconstruction model, the trained plant 3D reconstruction model can be used to generate a 3D mesh representation of the plant to be reconstructed based on the image sequence. In practice, the plant 3D reconstruction model, based on a neural radiation field network framework, can output a second synthetic image corresponding to each viewing direction in the image sequence based on the real crop image corresponding to each viewing direction. This second synthetic image can then describe an implicit 3D scene representation.

[0086] In one example, the process of generating an implicit three-dimensional scene representation corresponding to the plant to be reconstructed is as follows: through the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point; through the volume density and color extraction module, based on the high-dimensional vector of the pixel point, the volume density prediction value and color prediction value corresponding to each pixel point are extracted; through the volume rendering module, the volume density prediction value and color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0087] In one example, the process of extracting the volume density prediction value and color prediction value corresponding to each pixel is as follows: through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel; through the color prediction unit based on the multi-layer perceptron, the color prediction value corresponding to each pixel is generated based on the volume density prediction value corresponding to each pixel.

[0088] Specifically, the detailed process of the plant three-dimensional reconstruction model obtained through training generating the implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence can be referred to the data processing process in the training stage of the plant three-dimensional reconstruction model, and the embodiment of the present invention will not go into details about this.

[0089] Furthermore, the voxel-level reconstruction (Marching Cubes) algorithm is first used to convert the implicit three-dimensional scene representation into a displayed three-dimensional grid expression corresponding to the plant to be reconstructed; then the region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract the phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0090] In this embodiment of the present invention, the SFM-MVS method is used to perform 3D plant reconstruction based on the same image sequence data. Then, using the SFM-MVS reconstructed point cloud as a reference, the point pair distance is used to evaluate the similarity between the two reconstructed point clouds. This is shown below:

[0091]

[0092]

[0093] Among them, (x s ,y s ,z s ) is a point in the plant point cloud reconstructed using SFM-MVS, (x n ,y n ,z n ) is a point in the plant point cloud reconstructed using NeRF-Plant, and m is the number of points in the point cloud reconstructed by the latter.

[0094] See also Figure 5 A comparative image of plants reconstructed based on NeRF-Plant and SFM-MVS methods is shown. Figure 5 It shows that the reconstruction effect of NeRF-Plant method is significantly better than that of SFM-MVS method; and, see Figure 6 A time comparison chart based on NeRF-Plant and SFM-MVS methods is shown. Figure 6 It shows that the time consumed by the NeRF-Plant method is less than that consumed by the SFM-MVS method.

[0095] After determining the 3D mesh representation of the plant to be reconstructed, the phenotypic parameters of the plant to be reconstructed can be extracted. Figure 7 The following figure shows a schematic diagram of phenotypic parameter extraction, where the phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width. The specific process of extracting the phenotypic parameters of the plant to be reconstructed can be seen in steps 3a to 3c below:

[0096] Step 3a: Perform coordinate correction and scale restoration on the 3D mesh representation to obtain the target 3D mesh representation. The 3D mesh representation is used as the scene point cloud. In one example, the area where the calibration paper is located is segmented from the reconstructed scene point cloud, and the RANSAC algorithm is used to fit the plane point cloud equation where the calibration paper is located. The translation and rotation matrix between the fitted plane and the xy plane is calculated and applied to the entire scene point cloud for coordinate correction. The ratio between the distance of the calibration paper grid side in the point cloud and the actual distance is calculated, and the scale of the scene point cloud is restored to keep it consistent with the actual physical size.

[0097] Step 3b extracts the region of interest (ROI) corresponding to the plant to be reconstructed from the target 3D mesh representation. The ROI includes the plant point cloud and the leaf organ point clouds corresponding to each leaf in the plant to be reconstructed. The target 3D mesh representation is also the resized scene point cloud. In one example, the entire plant and leaf organ point clouds are segmented from the resized scene point cloud. For each leaf point cloud, the leaf tip, leaf base, and two leaf edge points corresponding to the leaf width are recorded.

[0098] Step 3c, taking the difference between the highest point and the lowest point of the plant point cloud in the vertical direction as the plant height; and, taking the maximum distance between any two points in the horizontal direction of the plant point cloud as the plant crown width; and, for each leaf, taking the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf as the leaf length; and, for each leaf, taking the shortest distance between the two leaf edge points in the leaf organ point cloud corresponding to the leaf as the leaf width.

[0099] In one example, the plant height is calculated as the difference between the highest point and the lowest point in the z-axis direction.

[0100] In one example, the convex hull point set of the point cloud on the xy plane is calculated, and the maximum distance between any two points in the traversal point set is the crown width of the plant.

[0101] In one example, the leaf point cloud used for calculation is converted into a directed graph data structure, and the Floyd algorithm is used to calculate the shortest distance from the leaf tip to the leaf base and the shortest distance between the two recorded leaf edge points, which are recorded as leaf length and leaf width respectively.

[0102] In this embodiment of the present invention, R2 and RMSE are used to evaluate the degree of agreement between the phenotypic parameters extracted based on the neural radiation field method and the manually measured values. The calculation formula is as follows:

[0103]

[0104] where y i represents the phenotypic parameter value obtained by measurement, Represents the calculated value obtained through the Nerf-Plant point cloud, represents the average value of the measured values, and n represents the number of measured data.

[0105] See also Figure 8 A comparison diagram of phenotypic parameters extracted from plants reconstructed based on NeRF-Plant and manually measured values ​​is shown. It can be seen that the phenotypic parameters extracted by the embodiment of the present invention have higher accuracy.

[0106] In summary, the embodiments of the present invention propose a phenotyping method based on an improved neural radiation field, which, compared with the SFM-MVS reconstruction method, has a lower reconstruction time while ensuring the reconstruction quality.

[0107] Based on the above embodiments, the present invention provides a phenotype acquisition device based on neural radiation field, see Figure 9 The schematic diagram of the structure of a phenotype acquisition device based on neural radiation field is shown, and the device mainly includes the following parts:

[0108] Data acquisition module 902 is used to obtain a sequence of images of the plant to be reconstructed, captured by a camera at multiple viewing angles. Based on the image sequence, internal and external parameters of the camera are generated using a large image segmentation model and a structure-from-motion algorithm. The combination of the internal and external parameters and the image sequence is used as a training dataset.

[0109] A 3D reconstruction module 904 is configured to train a 3D plant reconstruction model based on a neural radiation field network framework using a training dataset. The trained 3D plant reconstruction model generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence. The 3D plant reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron.

[0110] The phenotype acquisition module 906 is used to extract phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0111] The phenotypic acquisition device based on neural radiation field provided by an embodiment of the present invention designs a plant three-dimensional reconstruction model based on the neural radiation field network framework, trains the plant three-dimensional reconstruction model using a training data set constructed using an image sequence, and uses the trained plant three-dimensional reconstruction model to generate an implicit three-dimensional mesh expression of the plant to be reconstructed, and further extracts the corresponding phenotypic parameters of the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the time of crop reconstruction while ensuring the accuracy of crop reconstruction.

[0112] In one embodiment, the data acquisition module 902 is specifically configured to:

[0113] The target plant area in each real plant image in the image sequence is obtained through the image segmentation model to obtain an image mask sequence;

[0114] The image mask sequence is processed using the motion structure recovery algorithm to obtain the internal and external parameters of the shooting device.

[0115] In one embodiment, the 3D reconstruction module 904 is specifically configured to:

[0116] Through the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point;

[0117] Through the volume density and color extraction module, based on the high-dimensional vector of the pixel points, the volume density prediction value and color prediction value corresponding to each pixel point are extracted;

[0118] Through the volume rendering module, the volume density prediction value and the color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0119] In one embodiment, the 3D reconstruction module 904 is specifically configured to:

[0120] Through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel point;

[0121] A color prediction unit based on a multi-layer perceptron is used to generate a color prediction value corresponding to each pixel based on the volume density prediction value corresponding to each pixel.

[0122] In one embodiment, the volume density prediction unit includes a plurality of fully connected multi-layer perceptrons and an ECA attention network, wherein the ECA attention network is embedded after the first fully connected multi-layer perceptron;

[0123] The color prediction unit consists of multiple fully connected multi-layer perceptrons.

[0124] In one embodiment, the phenotype acquisition module 906 is specifically configured to:

[0125] Using voxel-level reconstruction algorithms, the implicit 3D scene representation is converted into a visible 3D mesh representation corresponding to the plant to be reconstructed.

[0126] A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0127] In one embodiment, the phenotype acquisition module 906 is specifically configured to:

[0128] Perform coordinate correction and scale restoration on the three-dimensional mesh expression to obtain the target three-dimensional mesh expression;

[0129] Extracting a region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression; wherein the region of interest includes the plant point cloud and the leaf organ point cloud corresponding to each leaf of the plant to be reconstructed;

[0130] The difference between the highest and lowest points of the plant point cloud in the vertical direction is taken as the plant height; and the maximum distance between any two points in the horizontal direction of the plant point cloud is taken as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is taken as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is taken as the leaf width;

[0131] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

[0132] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0133] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0134] Figure 10 A structural diagram of an electronic device provided in an embodiment of the present invention, wherein the electronic device 100 includes: a processor 10, a memory 11, a bus 12 and a communication interface 13, wherein the processor 10, the communication interface 13 and the memory 11 are connected via the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0135] The memory 11 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 13 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0136] The bus 12 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0137] Among them, the memory 11 is used to store programs, and the processor 10 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0138] The processor 10 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 10. The processor 10 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 11 , and the processor 10 reads the information in the memory 11 and completes the steps of the above method in combination with its hardware.

[0139] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A phenotype acquisition method based on neural radiation field, characterized in that: include: Obtain an image sequence of the plant to be reconstructed captured by a camera at multiple viewing angles, generate intrinsic and extrinsic parameters of the camera based on the image sequence using a large image segmentation model and a structure-from-motion algorithm, and use the combination of the intrinsic and extrinsic parameters and the image sequence as a training dataset; The training data set is used to train a plant three-dimensional reconstruction model based on a neural radiation field network framework, and the plant three-dimensional reconstruction model obtained by training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein, the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron, the volume density prediction unit includes multiple fully connected multi-layer perceptrons and an ECA attention network, and the ECA attention network is embedded after the first fully connected multi-layer perceptron; the color prediction unit includes multiple fully connected multi-layer perceptrons; Extracting phenotypic parameters corresponding to the plant to be reconstructed based on an implicit three-dimensional scene representation; The plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence, including: Mapping the pixel information of each real plant image in the image sequence to a high-dimensional space to obtain a high-dimensional vector of the pixel through the multi-resolution hash coding module; Extracting the volume density prediction value and color prediction value corresponding to each pixel based on the high-dimensional vector of the pixel point through the volume density and color extraction module, including: generating the volume density prediction value corresponding to each pixel point based on the high-dimensional vector of the pixel point through the volume density prediction unit based on the ECA attention mechanism; generating the color prediction value corresponding to each pixel point based on the volume density prediction value corresponding to each pixel point through the color prediction unit based on the multi-layer perceptron; The volume density prediction value and the color prediction value are integrated along the ray by the volume rendering module to synthesize a synthetic image corresponding to each viewing direction, so as to describe an implicit three-dimensional scene representation by the synthetic image.

2. The method for acquiring phenotypes based on neural radiation fields according to claim 1, characterized in that: Generating internal and external parameters of the shooting device based on the image sequence using an image segmentation model and a motion structure recovery algorithm, including: Obtaining the target plant region in each real plant image in the image sequence through the image segmentation model to obtain an image mask sequence; The image mask sequence is processed using a structure-from-motion algorithm to obtain the intrinsic and extrinsic parameters of the shooting device.

3. The method for acquiring phenotypes based on neural radiation fields according to claim 1, characterized in that: Extracting phenotypic parameters corresponding to the plant to be reconstructed based on an implicit three-dimensional scene representation includes: Using a voxel-level reconstruction algorithm, the implicit three-dimensional scene representation is converted into a displayed three-dimensional mesh expression corresponding to the plant to be reconstructed; A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

4. The method for acquiring phenotypes based on neural radiation fields according to claim 3, characterized in that: Extracting a region of interest corresponding to the plant to be reconstructed from the three-dimensional grid expression, and extracting phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest, comprising: performing coordinate correction and scale restoration processing on the three-dimensional mesh expression to obtain a target three-dimensional mesh expression; Extracting a region of interest corresponding to the to-be-reconstructed plant from the target three-dimensional mesh expression; wherein the region of interest includes a plant point cloud and a leaf organ point cloud corresponding to each leaf of the to-be-reconstructed plant; The difference between the highest point and the lowest point of the plant point cloud in the vertical direction is used as the plant height; and the maximum distance between any two points in the horizontal direction of the plant point cloud is used as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is used as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is used as the leaf width; The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

5. A phenotype acquisition device based on neural radiation field, characterized in that: include: A data acquisition module is configured to acquire a sequence of images of the plant to be reconstructed, captured by a camera at multiple viewing angles, generate intrinsic and extrinsic parameters of the camera based on the image sequence using a large image segmentation model and a structure-from-motion algorithm, and use the combination of the intrinsic and extrinsic parameters and the image sequence as a training dataset; A three-dimensional reconstruction module is used to train a plant three-dimensional reconstruction model based on a neural radiation field network framework using the training data set, and the plant three-dimensional reconstruction model obtained by training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron, the volume density prediction unit includes multiple fully connected multi-layer perceptrons and an ECA attention network, and the ECA attention network is embedded after the first fully connected multi-layer perceptron; the color prediction unit includes multiple fully connected multi-layer perceptrons; A phenotypic acquisition module, configured to extract phenotypic parameters corresponding to the plant to be reconstructed based on an implicit three-dimensional scene representation; The three-dimensional reconstruction module is specifically used for: Mapping the pixel information of each real plant image in the image sequence to a high-dimensional space to obtain a high-dimensional vector of the pixel through the multi-resolution hash coding module; Extracting the volume density prediction value and color prediction value corresponding to each pixel based on the high-dimensional vector of the pixel point through the volume density and color extraction module, including: generating the volume density prediction value corresponding to each pixel point based on the high-dimensional vector of the pixel point through the volume density prediction unit based on the ECA attention mechanism; generating the color prediction value corresponding to each pixel point based on the volume density prediction value corresponding to each pixel point through the color prediction unit based on the multi-layer perceptron; The volume density prediction value and the color prediction value are integrated along the ray by the volume rendering module to synthesize a synthetic image corresponding to each viewing direction, so as to describe an implicit three-dimensional scene representation by the synthetic image.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 4.

Images