Neural radiation field-based phenotype acquisition method and device

Abstract

The invention provides a phenotype acquisition method and device based on a neural radiation field, and relates to the technical field of agricultural informationization, and the method comprises the steps: obtaining an image sequence of a to-be-reconstructed plant collected by shooting equipment in multiple view angle directions, generating internal and external parameters of the shooting equipment based on the image sequence through an image segmentation large model and a motion structure recovery algorithm, and obtaining a phenotype of the to-be-reconstructed plant; taking as a training data set; training the plant three-dimensional reconstruction model based on the neural radiation field network framework by using the training data set, and generating implicit three-dimensional scene representation based on the image sequence through the trained plant three-dimensional reconstruction model; the model comprises a multi-resolution hash coding module, a volume density prediction unit based on an ECA attention mechanism, a color prediction unit based on a multi-layer perceptron and a volume rendering module. And extracting phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation. The method has relatively high crop reconstruction precision, and meanwhile, the crop reconstruction time can be greatly shortened.

Description

Technical Field

[0001] The present invention relates to the field of agricultural information technology, and in particular to a phenotype acquisition method and device based on a neural radiation field. Background Art

[0002] The three-dimensional structure of plants is the most intuitive way to obtain plant phenotypic information, and is essential for evaluating plant growth dynamics, functional performance, and resource utilization efficiency. Therefore, being able to quickly and accurately obtain the three-dimensional structure of plants is of great research significance for conducting high-throughput plant phenotyping, promoting genetic research, and optimizing breeding strategies.

[0003] At present, the methods for plant 3D structure are mainly divided into two types: sensor-based and photogrammetry-based methods. The sensor-based method mainly uses 3D scanners and other equipment to directly obtain high-precision plant 3D point clouds, but the cost of this method is too high and the reconstruction process is easily affected by the environment. The photogrammetry-based method is mainly based on the principle of camera geometry, using the motion recovery structure-multi-view stereo vision (SFM-MVS) algorithm to gradually restore the 3D scene. Although this method is low-cost, it takes too long in the dense reconstruction process, and the reconstruction effect of low-texture areas in the 3D scene is not good, which to some extent restricts its application in plant 3D phenotyping. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a phenotype acquisition method and device based on neural radiation field, which has high crop reconstruction accuracy and can significantly shorten the crop reconstruction time.

[0005] In a first aspect, an embodiment of the present invention provides a method for acquiring a phenotype based on a neural radiation field, comprising:

[0006] Obtain an image sequence of the plant to be reconstructed collected by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through a large image segmentation model and a motion structure recovery algorithm, and use the combination of the internal and external parameters and the image sequence as a training data set;

[0007] The plant 3D reconstruction model based on the neural radiation field network framework is trained using a training data set. The plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence. The plant 3D reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module, and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron.

[0008] The phenotypic parameters corresponding to the plants to be reconstructed are extracted based on the implicit three-dimensional scene representation.

[0009] In one embodiment, the internal and external parameters of the shooting device are generated based on the image sequence through the image segmentation model and the motion structure recovery algorithm, including:

[0010] The target plant region in each real plant image in the image sequence is obtained through the image segmentation model to obtain an image mask sequence;

[0011] The image mask sequence is processed using the motion structure recovery algorithm to obtain the internal and external parameters of the shooting device.

[0012] In one embodiment, the plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence, including:

[0013] Through the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to the high-dimensional space to obtain the high-dimensional vector of the sampling point;

[0014] Through the volume density and color extraction module, based on the high-dimensional vector of the pixel point, the volume density prediction value and color prediction value corresponding to each pixel point are extracted;

[0015] Through the volume rendering module, the volume density prediction value and the color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0016] In one embodiment, the volume density and color extraction module is used to extract the volume density prediction value and color prediction value corresponding to each pixel based on the high-dimensional vector of the pixel point, including:

[0017] Through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel point;

[0018] Through a color prediction unit based on a multi-layer perceptron, a color prediction value corresponding to each pixel is generated based on the volume density prediction value corresponding to each pixel.

[0019] In one embodiment, the volume density prediction unit includes a plurality of fully connected multi-layer perceptrons and an ECA attention network, wherein the ECA attention network is embedded after the first fully connected multi-layer perceptron;

[0020] The color prediction unit consists of multiple fully connected multi-layer perceptrons.

[0021] In one embodiment, extracting phenotypic parameters corresponding to the plant to be reconstructed based on an implicit three-dimensional scene representation includes:

[0022] Using a voxel-level reconstruction algorithm, the implicit three-dimensional scene representation is converted into a displayed three-dimensional mesh expression corresponding to the plant to be reconstructed;

[0023] A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0024] In one embodiment, extracting a region of interest corresponding to the plant to be reconstructed from the three-dimensional mesh expression, and extracting phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest, includes:

[0025] Performing coordinate correction and scale restoration processing on the three-dimensional mesh expression to obtain the target three-dimensional mesh expression;

[0026] Extracting a region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression; wherein the region of interest includes a plant point cloud and a leaf organ point cloud corresponding to each leaf of the plant to be reconstructed;

[0027] The difference between the highest point and the lowest point of the plant point cloud in the vertical direction is taken as the plant height; and the maximum distance between any two points in the plant point cloud in the horizontal direction is taken as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is taken as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is taken as the leaf width;

[0028] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

[0029] In a second aspect, an embodiment of the present invention further provides a phenotype acquisition device based on a neural radiation field, comprising:

[0030] A data acquisition module is used to acquire an image sequence of the plant to be reconstructed collected by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through an image segmentation model and a motion structure recovery algorithm, and use the combination of the internal and external parameters and the image sequence as a training data set;

[0031] A three-dimensional reconstruction module is used to train a plant three-dimensional reconstruction model based on a neural radiation field network framework using a training data set, and the plant three-dimensional reconstruction model obtained through training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on an image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, and the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron;

[0032] The phenotype acquisition module is used to extract the phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0035] The embodiment of the present invention provides a phenotype acquisition method and device based on neural radiation field. First, an image sequence of a plant to be reconstructed collected by a shooting device in multiple viewing directions is obtained, and the internal and external parameters of the shooting device are generated based on the image sequence through a large image segmentation model and a motion structure recovery algorithm, and a combination of the internal and external parameters and the image sequence is used as a training data set; then, a plant three-dimensional reconstruction model based on a neural radiation field network framework is trained using the training data set, and the plant three-dimensional reconstruction model obtained through training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, and the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron; finally, the phenotypic parameters corresponding to the plant to be reconstructed are extracted based on the implicit three-dimensional scene representation. The above method designs a plant 3D reconstruction model based on a neural radiation field network framework, trains the plant 3D reconstruction model using a training data set constructed using an image sequence, and uses the trained plant 3D reconstruction model to generate an implicit 3D mesh expression of the plant to be reconstructed, and further extracts phenotypic parameters corresponding to the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the time of crop reconstruction while ensuring the accuracy of crop reconstruction.

[0036] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of a flow chart of a method for acquiring a phenotype based on a neural radiation field provided by an embodiment of the present invention;

[0040] Figure 2 An overall flow chart of a phenotype acquisition method based on neural radiation field provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the principle of NeRF-Plant plant reconstruction provided by an embodiment of the present invention;

[0042] Figure 4 A NeRF-Plant network architecture diagram provided for an embodiment of the present invention;

[0043] Figure 5 A plant comparison map reconstructed based on NeRF-Plant and SFM-MVS methods provided in an embodiment of the present invention;

[0044] Figure 6 A time comparison diagram based on NeRF-Plant and SFM-MVS methods provided in an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of phenotypic parameter extraction provided by an embodiment of the present invention;

[0046] Figure 8 A comparison chart of phenotypic parameters extracted from plants reconstructed based on NeRF-Plant and manually measured values ​​provided in an embodiment of the present invention;

[0047] Fig. 9 A schematic diagram of the structure of a phenotype acquisition device based on neural radiation field provided in an embodiment of the present invention;

[0048] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] At present, the use of 3D scanners and other equipment to obtain the three-dimensional structural information of plants has problems such as high cost and low reconstruction efficiency. Although the structure from motion-multi-view stereo vision algorithm is widely used in the three-dimensional reconstruction of plants, there are still problems such as the dense reconstruction process taking too long and poor reconstruction effect on low-texture areas in three-dimensional scenes, which limits its application in subsequent phenotypic analysis. Therefore, a non-contact, fast and accurate method for three-dimensional reconstruction of plants is urgently needed.

[0051] Neural Radiance Field (NeRF) is an implicit 3D scene expression based on neural networks. It synthesizes new perspective information from existing image sequences through learning methods. It has the characteristics of fast reasoning speed and realistic reconstruction effect. At present, the research on NeRF is mainly focused on the fields of large scene reconstruction and visual effects generation, but there are few reports on the research related to the acquisition of plant 3D structure. Therefore, the embodiment of the present invention will realize the 3D reconstruction and phenotypic extraction of plants based on the improved neural radiation field.

[0052] Based on this, the present invention provides a phenotype acquisition method and device based on neural radiation field, which has high crop reconstruction accuracy and can significantly shorten the crop reconstruction time.

[0053] To facilitate understanding of this embodiment, a phenotype acquisition method based on neural radiation field disclosed in an embodiment of the present invention is first described in detail, see Figure 1 The flowchart of a method for acquiring a phenotype based on a neural radiation field is shown, and the method mainly includes the following steps S102 to S106:

[0054] Step S102, obtain an image sequence of the plant to be reconstructed captured by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through an image segmentation model and a motion structure recovery algorithm, and use the combination of the internal and external parameters and the image sequence as a training data set.

[0055] Among them, the shooting device is a device with image acquisition function, such as a camera, the image sequence contains multiple real plant images, and the training data set includes the image sequence and the internal and external parameters of the shooting device, which are determined based on the image sequence.

[0056] In one example, the plant to be reconstructed can be placed on a grid calibration paper with a side length of 2 cm. The plant to be reconstructed can be filmed multiple times in a circle. During the recording process, try to maintain uniform circular motion and ensure that the complete plant to be reconstructed is always included in the viewfinder. The captured video is then framed to obtain an image sequence.

[0057] In one example, a large image segmentation model is used to obtain the target plant area in each real plant image in the image sequence to obtain an image mask sequence. The image mask sequence is processed using SFM (Structure from motion) to obtain the internal and external parameters of the shooting device. The combination of the internal and external parameters and the image sequence is used as a training data set for a plant 3D reconstruction model based on a neural radiation field network framework.

[0058] Step S104, using the training data set to train the plant 3D reconstruction model based on the neural radiation field network framework, and the plant 3D reconstruction model obtained by the training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence.

[0059] The plant 3D reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron. In one example, sampling point information is generated based on a training data set, and the sampling point information is input into the multi-resolution hash module. Each sampling point and viewing direction in the three-dimensional space is encoded into a high-dimensional vector through a hash coding function as the input of the volume density and color extraction module; the volume density and color extraction module mainly includes two MLP (Multilayer Perceptron, multi-layer perceptron) networks, which are used to extract the volume density prediction value and color prediction value of the sampling point respectively; the volume rendering module generates a synthetic image corresponding to each viewing direction based on the volume density prediction value and the color prediction value, and then uses the difference between the synthetic object and the corresponding real plant image to realize the training of the plant 3D reconstruction model, and inputs the image sequence into the trained plant 3D reconstruction model to obtain the implicit 3D scene representation corresponding to the plant to be reconstructed.

[0060] Step S106: extracting phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0061] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width. In one example, the implicit three-dimensional scene representation can be converted into a displayed three-dimensional grid expression, and after the three-dimensional grid expression is subjected to coordinate correction and scale restoration processing, the region of interest corresponding to the plant to be reconstructed is extracted, and the region of interest includes the plant point cloud and the leaf organ point cloud corresponding to each leaf in the plant to be reconstructed, and then the plant height and plant crown width are determined based on the plant point cloud, and the leaf length and leaf width corresponding to each leaf are determined based on the leaf organ point cloud corresponding to each leaf.

[0062] The phenotype acquisition method based on neural radiation field provided in the embodiment of the present invention designs a plant 3D reconstruction model based on the neural radiation field network framework, trains the plant 3D reconstruction model using a training data set constructed using an image sequence, and uses the trained plant 3D reconstruction model to generate an implicit 3D mesh expression of the plant to be reconstructed, and further extracts the phenotypic parameters corresponding to the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the time of crop reconstruction while ensuring the accuracy of crop reconstruction.

[0063] For ease of understanding, the embodiments of the present invention provide Figure 2 The overall flow chart of a phenotypic acquisition method based on neural radiation field is shown, including: data set production: surround video acquisition, video frame processing, camera parameter acquisition; NeRF-Plant (that is, a plant 3D reconstruction model based on the neural radiation field network framework) training: hash coding, feature extraction, volume rendering; coordinate correction: coordinate transformation, scale recovery; phenotypic parameter extraction: plant height, crown width, leaf length, leaf width; accuracy and efficiency evaluation: point-to-point distance, reconstruction time, phenotypic parameters.

[0064] Among them, see Figure 3 A schematic diagram of the principle of NeRF-Plant plant reconstruction is shown. Figure 3 (a) in the figure shows that a camera position map is obtained by performing three-dimensional reconstruction on the image sequence, and the camera position map is related to the internal and external parameters of the shooting device; NeRF-Plant is used to output the synthetic image in each viewing direction, and Marching Cube (voxel-level reconstruction algorithm) is used to obtain the displayed three-dimensional grid expression; Figure 3 (b) in the figure illustrates the training process of NeRF-Plant.

[0065] Based on this, an embodiment of the present invention provides a specific implementation of a phenotype acquisition method based on a neural radiation field.

[0066] For the aforementioned step S102, the embodiment of the present invention provides a specific process for preparing a data set for neural radiation field training, which specifically includes the following sub-steps 1a to 1c:

[0067] Step 1a: Place the plant to be reconstructed on a 2cm square calibration paper, hold the phone and circle the subject three times, try to maintain a uniform circular motion, and ensure that the entire plant is always in the frame. The angles between the device and the ground are approximately 15°, 45°, and 75° for each circle.

[0068] Step 1b: The three videos are divided into frames and one picture is saved every 1 second. The final number of pictures obtained is about 80-130. The collection of these pictures is the aforementioned image sequence.

[0069] Step 1c: Use the image segmentation model SAM (Segment Anything Model) to automatically obtain the target plant area in the image sequence and obtain the corresponding image mask sequence. By obtaining the image mask sequence, the amount of surrounding scene information in the subsequently generated plant point cloud can be reduced, reducing the pressure of subsequent point cloud processing.

[0070] Step 1c: Process the image mask sequence based on the SFM module to obtain the intrinsic and extrinsic parameters of the camera. The combination of the intrinsic and extrinsic parameters of the camera and the image sequence is the data set for neural radiation field training.

[0071] Before explaining the aforementioned step S104, the embodiment of the present invention first provides a specific implementation of a plant three-dimensional reconstruction model (NeRF-Plant) based on a neural radiation field network framework. Figure 4 The NeRF-Plant network architecture diagram shown in the figure shows that the NeRF-Plant network consists of three parts in total, namely a multi-resolution hash encoding module, a volume density and color extraction module (including a volume density prediction unit density network and a color prediction unit color network) and a volume rendering module.

[0072] Among them, (1) Multi-resolution hash coding module: Each sampling point and viewing direction in the three-dimensional space is encoded into a high-dimensional vector through a hash coding function. This method is more compact and has a higher degree of parallelism than the position coding in NeRF, and the training and inference speed of the NeRF network will also be faster. (2) Volume density and color extraction module: It is mainly used to predict the volume density (density) and color (color) of each sampling point. The volume density indicates the degree of occupation of the point in space, and the color indicates the RGB value of the point. The volume density and color extraction module mainly includes two MLP networks. The first network is a volume density prediction unit that integrates the ECA (Efficient Channel Attention) attention mechanism, and the other is a color prediction unit based on MLP. Among them, the ECA attention module enables the model to focus more on information useful for volume density prediction by adaptively adjusting the weight of each channel. (3) Volume rendering module: The predicted volume density and color are integrated along the ray to synthesize the image at each viewpoint. Specifically, through ray projection, the volume density and color of each sampling point are calculated, and the final pixel color value is calculated using the volume rendering equation. The calculation formula of the volume rendering module is as follows:

[0073]

[0074] Where C(r) is the color value of the pixel on the ray, T(t) is the transmittance, σ(t) is the volume density, and c(t) is the color value.

[0075] Based on the aforementioned network structure, the embodiment of the present invention provides a specific implementation method for training a plant 3D reconstruction model, see the following steps 2a to 2c:

[0076] Step 2a, generating sampling point information according to the training data set. In one example, the sampling point information is generated according to the viewing angle of the shooting device and the collected image sequence information.

[0077] Step 2b, through the plant three-dimensional reconstruction model based on the neural radiation field network framework, output the first synthetic image corresponding to each viewing direction based on the sampling point information.

[0078] In the specific implementation, it includes:

[0079] 1) Through the multi-resolution hash coding module, the sampling point information is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point. In one example, the sampling point information is mapped to a high-dimensional space using a multi-resolution hash coding module, where the coordinate information of the sampling point is increased to 63 dimensions and the camera's viewing angle information is increased to 27 dimensions.

[0080] 2) Through the volume density and color extraction module, based on the high-dimensional vector of the sampling points, the volume density prediction value and color prediction value corresponding to each sampling point are extracted. The volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron. In one embodiment, the volume density prediction unit includes multiple fully connected multi-layer perceptrons and an ECA attention network, and the ECA attention network is embedded after the first fully connected multi-layer perceptron; the color prediction unit includes multiple fully connected multi-layer perceptrons. Please continue to refer to Figure 4 , Figure 4 It also shows that the volume density prediction network includes 5 fully connected MLP networks and one ECA attention network, where the ECA attention network is embedded after the first fully connected MLP network. The color prediction network includes 3 fully connected MLP networks.

[0081] Based on this, first, a volume density prediction unit based on the ECA attention mechanism is used to generate a volume density prediction value corresponding to each sampling point based on the high-dimensional vector of the sampling point; then, a color prediction unit based on a multi-layer perceptron is used to generate a color prediction value corresponding to each sampling point based on the volume density prediction value corresponding to each sampling point.

[0082] In practical applications, in the ECA attention network, the feature map calculates the global statistics of each channel and uses an adaptive one-dimensional convolution kernel to capture the dependencies between channels. Then, the channel attention weights are generated through Sigmoid activation, and these weights are finally applied to the original feature map to enhance important feature channels and suppress unimportant channels, thereby improving the network's ability to extract key information. The final network output is the volume density prediction value and RGB color prediction value corresponding to the sampling point.

[0083] 3) The volume density prediction value and the color prediction value are integrated along the ray through the volume rendering module to synthesize a first synthetic image corresponding to each viewing angle direction.

[0084] Step 2c, based on the first synthetic image and the real plant image corresponding to the first synthetic image in the image sequence, the network parameters of the plant 3D reconstruction model are adjusted to realize the training of the plant 3D reconstruction model. In one example, the difference between the synthetic image and the real plant image is calculated, and the parameters of the MLP network are continuously adjusted through the back propagation algorithm to minimize the loss function and gradually optimize the 3D representation of the scene. The optimization of the loss function is performed iteratively until the model converges and an implicit 3D scene representation is obtained.

[0085] After the plant 3D reconstruction model is trained, the trained plant 3D reconstruction model can be used to generate a 3D mesh expression corresponding to the plant to be reconstructed based on the image sequence. In specific implementation, the plant 3D reconstruction model based on the neural radiation field network framework can output a second synthetic image corresponding to each viewing direction based on the real crop image corresponding to each viewing direction in the image sequence, so as to describe an implicit 3D scene representation through the second synthetic image.

[0086] In one example, the process of generating an implicit three-dimensional scene representation corresponding to the plant to be reconstructed is as follows: through a multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to a high-dimensional space to obtain a high-dimensional vector of the sampling point; through a volume density and color extraction module, based on the high-dimensional vector of the pixel point, the volume density prediction value and color prediction value corresponding to each pixel point are extracted; through a volume rendering module, the volume density prediction value and the color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0087] In one example, the process of extracting the volume density prediction value and color prediction value corresponding to each pixel is as follows: through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel; through the color prediction unit based on the multi-layer perceptron, the color prediction value corresponding to each pixel is generated based on the volume density prediction value corresponding to each pixel.

[0088] Specifically, the detailed process of the plant three-dimensional reconstruction model obtained through training generating an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence can be referred to the data processing process in the training phase of the plant three-dimensional reconstruction model, and the embodiments of the present invention will not go into details about this.

[0089] Furthermore, the implicit three-dimensional scene representation is first converted into a displayed three-dimensional grid expression corresponding to the plant to be reconstructed using a voxel-level reconstruction (Marching Cubes) algorithm; then the region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract the phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0090] The embodiment of the present invention uses the SFM-MVS method to perform 3D reconstruction of plants based on the same image sequence data. Then, the point cloud reconstructed by SFM-MVS is used as a reference, and the point pair distance is used to evaluate the similarity between the point clouds reconstructed by the two methods. As shown below:

[0091]

[0092]

[0093] Among them, (x s ,y s ,z s ) is a point in the plant point cloud reconstructed using SFM-MVS, (x n ,y n ,z n ) is a point in the plant point cloud reconstructed using NeRF-Plant, and m is the number of points in the point cloud reconstructed by the latter.

[0094] See also Figure 5 A comparison of plants reconstructed based on NeRF-Plant and SFM-MVS methods is shown. Figure 5 It shows that the reconstruction effect of NeRF-Plant method is significantly better than that of SFM-MVS method; and, see Figure 6 A time comparison chart based on NeRF-Plant and SFM-MVS methods is shown. Figure 6 It shows that the time consumed by the NeRF-Plant method is less than that consumed by the SFM-MVS method.

[0095] After determining the 3D mesh expression of the plant to be reconstructed, the phenotypic parameters of the plant to be reconstructed can be extracted. Figure 7 A schematic diagram of phenotypic parameter extraction is shown, where the phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width. The specific process of extracting the phenotypic parameters of the plant to be reconstructed can be seen in the following steps 3a to 3c:

[0096] Step 3a, coordinate correction and scale restoration processing are performed on the three-dimensional mesh expression to obtain the target three-dimensional mesh expression. The three-dimensional mesh expression is used as the scene point cloud. In one example, the area where the calibration paper is located is segmented from the reconstructed scene point cloud, and the plane point cloud equation where the calibration paper is located is fitted using the RANSAC algorithm; the translation and rotation matrix between the fitted plane and the xy plane is calculated, and applied to the entire scene point cloud for coordinate correction; the ratio between the distance of the calibration paper square edge in the point cloud and the actual distance is calculated, and the scale of the scene point cloud is restored to keep it consistent with the actual physical size.

[0097] Step 3b, extracting the region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression. The region of interest includes the plant point cloud and the leaf organ point cloud corresponding to each leaf in the plant to be reconstructed. The target three-dimensional mesh expression is also the scene point cloud after size restoration. In one example, the whole plant and the leaf organ point cloud are segmented from the scene point cloud after size restoration, and for each leaf point cloud, the two leaf edge points corresponding to the leaf tip, leaf base and leaf width are recorded respectively.

[0098] Step 3c, taking the difference between the highest point and the lowest point of the plant point cloud in the vertical direction as the plant height; and, taking the maximum distance between any two points in the horizontal direction of the plant point cloud as the plant crown width; and, for each leaf, taking the shortest distance between the leaf tip point and the leaf base point in the leaf organ point cloud corresponding to the leaf as the leaf length; and, for each leaf, taking the shortest distance between the two leaf edge points in the leaf organ point cloud corresponding to the leaf as the leaf width.

[0099] In one example, the calculation of the plant height is defined as the difference between the highest point and the lowest point in the z-axis direction.

[0100] In one example, the convex hull point set of the point cloud on the xy plane is calculated, and the maximum distance between any two points in the traversal point set is the crown width of the plant.

[0101] In one example, the leaf point cloud used for calculation is converted into a directed graph data structure, and the Floyd algorithm is used to calculate the shortest distance from the leaf tip to the leaf base and the shortest distance between two recorded leaf edge points, which are recorded as leaf length and leaf width, respectively.

[0102] In the embodiment of the present invention, R2 and RMSE are used to evaluate the degree of agreement between the phenotypic parameters extracted based on the neural radiation field method and the manually measured values. The calculation formula is as follows:

[0103]

[0104] where y i represents the phenotypic parameter value obtained by measurement, Represents the calculated value obtained through the Nerf-Plant point cloud, represents the average value of the measured values, and n represents the number of measured data.

[0105] See also Figure 8 A comparison chart showing phenotypic parameters extracted from plants reconstructed based on NeRF-Plant and manually measured values ​​shows that the phenotypic parameters extracted by the embodiment of the present invention have higher accuracy.

[0106] In summary, the embodiment of the present invention proposes a phenotyping method based on an improved neural radiation field, which has a lower reconstruction time while ensuring the reconstruction quality compared to the SFM-MVS reconstruction method.

[0107] Based on the above embodiments, the present invention provides a phenotype acquisition device based on neural radiation field, see Fig. 9 The schematic diagram of the structure of a phenotype acquisition device based on neural radiation field is shown, and the device mainly includes the following parts:

[0108] The data acquisition module 902 is used to acquire an image sequence of the plant to be reconstructed collected by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through an image segmentation model and a motion structure recovery algorithm, and use the combination of the internal and external parameters and the image sequence as a training data set;

[0109] The three-dimensional reconstruction module 904 is used to train the plant three-dimensional reconstruction model based on the neural radiation field network framework using the training data set, and the plant three-dimensional reconstruction model obtained by the training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, and the volume density and color extraction module includes a volume density prediction unit based on the ECA attention mechanism and a color prediction unit based on a multi-layer perceptron;

[0110] The phenotype acquisition module 906 is used to extract phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

[0111] The phenotype acquisition device based on neural radiation field provided in the embodiment of the present invention designs a plant 3D reconstruction model based on the neural radiation field network framework, trains the plant 3D reconstruction model using a training data set constructed using an image sequence, and uses the trained plant 3D reconstruction model to generate an implicit 3D mesh expression of the plant to be reconstructed, and further extracts the phenotypic parameters corresponding to the plant to be reconstructed. Compared with the traditional motion recovery structure-multi-view stereo vision algorithm, the embodiment of the present invention can significantly shorten the time of crop reconstruction while ensuring the accuracy of crop reconstruction.

[0112] In one implementation, the data acquisition module 902 is specifically used to:

[0113] The target plant region in each real plant image in the image sequence is obtained through the image segmentation model to obtain an image mask sequence;

[0114] The image mask sequence is processed using the motion structure recovery algorithm to obtain the internal and external parameters of the shooting device.

[0115] In one implementation, the 3D reconstruction module 904 is specifically configured to:

[0116] Through the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to the high-dimensional space to obtain the high-dimensional vector of the sampling point;

[0117] Through the volume density and color extraction module, based on the high-dimensional vector of the pixel point, the volume density prediction value and color prediction value corresponding to each pixel point are extracted;

[0118] Through the volume rendering module, the volume density prediction value and the color prediction value are integrated along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe the implicit three-dimensional scene representation through the synthetic image.

[0119] In one implementation, the 3D reconstruction module 904 is specifically configured to:

[0120] Through the volume density prediction unit based on the ECA attention mechanism, the volume density prediction value corresponding to each pixel is generated based on the high-dimensional vector of the pixel point;

[0121] Through a color prediction unit based on a multi-layer perceptron, a color prediction value corresponding to each pixel is generated based on the volume density prediction value corresponding to each pixel.

[0122] In one embodiment, the volume density prediction unit includes a plurality of fully connected multi-layer perceptrons and an ECA attention network, wherein the ECA attention network is embedded after the first fully connected multi-layer perceptron;

[0123] The color prediction unit consists of multiple fully connected multi-layer perceptrons.

[0124] In one embodiment, the phenotype acquisition module 906 is specifically used to:

[0125] Using a voxel-level reconstruction algorithm, the implicit three-dimensional scene representation is converted into a displayed three-dimensional mesh expression corresponding to the plant to be reconstructed;

[0126] A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

[0127] In one embodiment, the phenotype acquisition module 906 is specifically used to:

[0128] Performing coordinate correction and scale restoration processing on the three-dimensional mesh expression to obtain the target three-dimensional mesh expression;

[0129] Extracting a region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression; wherein the region of interest includes a plant point cloud and a leaf organ point cloud corresponding to each leaf of the plant to be reconstructed;

[0130] The difference between the highest point and the lowest point of the plant point cloud in the vertical direction is taken as the plant height; and the maximum distance between any two points in the plant point cloud in the horizontal direction is taken as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is taken as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is taken as the leaf width;

[0131] The phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

[0132] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0133] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0134] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 10, a memory 11, a bus 12 and a communication interface 13, wherein the processor 10, the communication interface 13 and the memory 11 are connected via the bus 12; the processor 10 is used to execute an executable module stored in the memory 11, such as a computer program.

[0135] The memory 11 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 13 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0136] The bus 12 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0137] Among them, the memory 11 is used to store programs, and the processor 10 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 10 or implemented by the processor 10.

[0138] The processor 10 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 10. The above processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 11, and the processor 10 reads the information in the memory 11 and completes the steps of the above method in combination with its hardware.

[0139] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0140] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0141] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A phenotype acquisition method based on neural radiation field, characterized in that: include: Acquire an image sequence of the plant to be reconstructed captured by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through a large image segmentation model and a motion structure recovery algorithm, and use a combination of the internal and external parameters and the image sequence as a training data set; The training data set is used to train a plant three-dimensional reconstruction model based on a neural radiation field network framework, and the plant three-dimensional reconstruction model obtained by training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, and the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron; The phenotypic parameters corresponding to the plant to be reconstructed are extracted based on the implicit three-dimensional scene representation.

2. The method for acquiring phenotype based on neural radiation field according to claim 1, characterized in that: Generate the internal and external parameters of the shooting device based on the image sequence through the image segmentation model and the motion structure recovery algorithm, including: Acquire the target plant region in each real plant image in the image sequence through the image segmentation model to obtain an image mask sequence; The image mask sequence is processed using a motion structure recovery algorithm to obtain the internal and external parameters of the shooting device.

3. The method for acquiring phenotype based on neural radiation field according to claim 1, characterized in that: The plant 3D reconstruction model obtained through training generates an implicit 3D scene representation corresponding to the plant to be reconstructed based on the image sequence, including: By means of the multi-resolution hash coding module, the pixel information of each real plant image in the image sequence is mapped to a high-dimensional space to obtain a high-dimensional vector of sampling points; By means of the volume density and color extraction module, based on the high-dimensional vector of the pixel point, a volume density prediction value and a color prediction value corresponding to each pixel point are extracted; The volume rendering module integrates the volume density prediction value and the color prediction value along the ray to synthesize a synthetic image corresponding to each viewing direction, so as to describe an implicit three-dimensional scene representation through the synthetic image.

4. The method for acquiring phenotype based on neural radiation field according to claim 3, characterized in that: The volume density and color extraction module extracts the volume density prediction value and color prediction value corresponding to each pixel based on the pixel high-dimensional vector, including: Generate a volume density prediction value corresponding to each pixel based on the high-dimensional vector of the pixel point through the volume density prediction unit based on the ECA attention mechanism; The color prediction unit based on the multi-layer perceptron generates a color prediction value corresponding to each pixel based on the volume density prediction value corresponding to each pixel.

5. The method for acquiring phenotype based on neural radiation field according to claim 4, characterized in that: The volume density prediction unit includes a plurality of fully connected multi-layer perceptrons and an ECA attention network, wherein the ECA attention network is embedded after the first fully connected multi-layer perceptron; The color prediction unit includes a plurality of fully connected multi-layer perceptrons.

6. The method for acquiring phenotype based on neural radiation field according to claim 1, characterized in that: Extracting phenotypic parameters corresponding to the plant to be reconstructed based on an implicit three-dimensional scene representation includes: Using a voxel-level reconstruction algorithm, the implicit three-dimensional scene representation is converted into a displayed three-dimensional grid expression corresponding to the plant to be reconstructed; A region of interest corresponding to the plant to be reconstructed is extracted from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest.

7. The method for acquiring phenotype based on neural radiation field according to claim 6, characterized in that: Extracting a region of interest corresponding to the plant to be reconstructed from the three-dimensional grid expression, so as to extract phenotypic parameters corresponding to the plant to be reconstructed based on the region of interest, comprising: Performing coordinate correction and scale restoration processing on the three-dimensional grid expression to obtain a target three-dimensional grid expression; Extracting a region of interest corresponding to the plant to be reconstructed from the target three-dimensional mesh expression; wherein the region of interest includes a plant point cloud and a leaf organ point cloud corresponding to each leaf in the plant to be reconstructed; The difference between the highest point and the lowest point of the plant point cloud in the vertical direction is taken as the plant height; and the maximum distance between any two points of the plant point cloud in the horizontal direction is taken as the plant crown width; and for each leaf, the shortest distance between the leaf tip and the leaf base in the leaf organ point cloud corresponding to the leaf is taken as the leaf length; and for each leaf, the shortest distance between two leaf edge points in the leaf organ point cloud corresponding to the leaf is taken as the leaf width; Wherein, the phenotypic parameters include one or more of plant height, plant crown width, leaf length, and leaf width.

8. A phenotype acquisition device based on neural radiation field, characterized in that: include: A data acquisition module is used to acquire an image sequence of the plant to be reconstructed collected by a shooting device in multiple viewing directions, generate internal and external parameters of the shooting device based on the image sequence through an image segmentation model and a motion structure recovery algorithm, and use a combination of the internal and external parameters and the image sequence as a training data set; A three-dimensional reconstruction module, used to train a plant three-dimensional reconstruction model based on a neural radiation field network framework using the training data set, and the plant three-dimensional reconstruction model obtained by the training generates an implicit three-dimensional scene representation corresponding to the plant to be reconstructed based on the image sequence; wherein the plant three-dimensional reconstruction model includes a multi-resolution hash coding module, a volume density and color extraction module and a volume rendering module, and the volume density and color extraction module includes a volume density prediction unit based on an ECA attention mechanism and a color prediction unit based on a multi-layer perceptron; The phenotype acquisition module is used to extract the phenotypic parameters corresponding to the plant to be reconstructed based on the implicit three-dimensional scene representation.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images