New perspective synthesis method of wheat population based on improved neural radiation field

Abstract

The present invention discloses a method for synthesizing a new perspective of group wheat based on an improved neural radiation field, and relates to the field of computer vision technology. The method comprises the following steps: taking an image of group wheat and constructing a group wheat data set; constructing a neural radiation field of a group wheat scene, producing rays in the neural radiation field, acquiring sampling points according to a neural radiation field sampling strategy, and obtaining the 3D position and direction of the sampling points; inputting the 3D position x of the sampling point into a multi-plane hash encoder, and then inputting the encoded data into a neural radiation field network, performing rendering loss training, and predicting the color and density of the sampling; after the training is completed, a new image with a new perspective can be obtained. The beneficial effects of the present invention are: the present invention adopts a reference image-guided strategy, and improves the multi-plane hash coding of the NeRF network, and can fit complex and dense scene reconstruction very efficiently and with high quality.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a new perspective synthesis method for wheat in a group based on an improved neural radiation field. Background Art

[0002] With the development of agricultural informatization and intelligence, accurate monitoring and evaluation of crop growth status has become a key demand in agricultural production. As an important food crop, the growth status of wheat groups directly affects food yield and quality. However, due to the complexity of wheat groups, such as leaves covering each other and unclear features, it is often difficult to accurately obtain complete information about wheat groups with just a few two-dimensional images, which limits the comprehensive evaluation of their growth status. If a comprehensive shooting is required, too many images are required, and the shooting position is difficult. Therefore, it is of great significance to synthesize wheat groups from a new perspective.

[0003] In recent years, the rapid development of deep learning technology, especially in the field of computer vision, has provided new ideas for crop monitoring. Among them, Neural Radiance Field (NeRF), as an emerging three-dimensional scene representation method, has attracted widespread attention due to its ability to reconstruct high-quality new perspectives from a small number of input images. However, when NeRF technology is directly applied to the synthesis of new perspectives of group wheat, it faces many challenges, such as poor adaptability to complex scenes and difficulty in predicting the middle area of ​​wheat. Summary of the invention

[0004] In order to ensure high synthesis quality and to provide a new image of group wheat from a new perspective quickly and conveniently, the present invention provides a convenient method for synthesizing group wheat from a new perspective based on an improved neural radiation field. Specifically, a group of relatively sparse wheat group images and a global reference image are taken, a neural radiation field is constructed to generate rays, the position and direction of the sampling point are obtained according to the sampling strategy, the position of the sampling point is put into a multi-plane hash encoder, the multi-dimensional features after the sampling point encoding are obtained, and the direction of the same point is input into the NeRF network for training. After each cycle step of training, a new image with the same perspective as the reference image is rendered, and the loss fitting of the two images is performed so that the middle occlusion position of the group wheat can also be inferred. After the training is completed, new images of group wheat from other perspectives can be obtained.

[0005] In order to achieve the above-mentioned invention object, the present invention provides a convenient group wheat new perspective synthesis method based on improved neural radiation field, and the synthesis method comprises the following steps:

[0006] Step S1: take images of group wheat and construct a group wheat dataset;

[0007] Step S2: construct the neural radiation field of the group wheat scene, generate rays in the neural radiation field, obtain sampling points according to the neural radiation field sampling strategy, and obtain the 3D position and direction of the sampling points;

[0008] Step S3: Input the 3D position x of the sampling point into the multi-plane hash encoder, and then input the encoded data into the neural radiance field network for rendering loss training to predict the sampled color and density;

[0009] Step S4: After the training is completed, a new image with a new perspective can be obtained.

[0010] Wherein, the step S1 is specifically as follows:

[0011] A set of wheat images from different peripheral perspectives were taken using a portable camera as the experimental dataset.

[0012] A global overhead image covering the wheat group is taken as test supervision data for subsequent training steps.

[0013] The sampling strategy in step S2 is specifically: first perform stratified random sampling, then perform importance sampling; stratified random sampling is to divide the population into several sub-populations using one or more variables; wherein, the importance sampling is specifically: coarse sampling between the interval [a, b] Points , and obtain its weight , then according to The corresponding weights get new Sampling points , the subsequent rendering process will be for add .

[0014] The step S3 is specifically as follows:

[0015] Step S301: Input the 3D position of the sampling point into the multi-plane hash encoder to obtain the encoded spatial position ; Step S302: The encoded With observation direction Feed it into the Neural Radiance Field Network for rendering loss training;

[0016] Step S303: predict the color and density of the sampling points.

[0017] Wherein, the step S301 is specifically as follows:

[0018] Use three 2D multi-resolution hash encoders to 3D position coordinates of the sampling points The projection coordinates are encoded as follows:

[0019] ;

[0020] in, is the projection coordinate The plane-level geometric features, Is a plane Multi-resolution hash coding of Representative level, Represents the feature size of each entry, and then concatenates the results to get the encoded spatial position :

[0021] ;

[0022] in, They represent the encoding results of the sampling points projected onto the xy, yz and xz planes in three-dimensional space using a 2D multi-resolution hash encoder, respectively. Indicates that the feature is connected to a The concatenation operator on the channel vector of .

[0023] The input of the neural radiation field network is: the encoded spatial position and observation direction ; Among them, the multi-plane hash encoder is used to improve the resolution and expression ability of rendering while improving the rendering speed; the structure of the neural radiation field network includes multiple layers of fully connected layers, and each fully connected layer is connected by a nonlinear activation function.

[0024] The rendering loss training is achieved by minimizing the loss function, and the loss function formula is:

[0025] ;

[0026] in is the total loss function, represents a ray starting from the center of the camera, is the set of all rays, is the network prediction along the ray The rendered color value, is obtained from the real image along the ray The corresponding color value, is a parameter in the network, is a hyperparameter used to balance the weight between RGB color reconstruction loss and regularization loss.

[0027] Among them, in the step S302, after each training cycle step, the rendering formula of the neural radiation field network is used according to the reference image The formulation perspective Predicting new images , and the new image With reference image Perform MSE loss fitting, the loss fitting formula is:

[0028] ;

[0029] Refers to the number of pixels, Refers to the RGB value of the corresponding pixel in the reference image, Refers to the RGB value of the corresponding pixel in the predicted image.

[0030] The beneficial effects of the present invention are as follows: the present invention adopts a reference image guided strategy and improves the multi-plane hash coding of the NeRF network, and can fit complex and dense scene reconstruction very efficiently and with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowcharts of Embodiments 1 and 2 of the present invention.

[0032] Figure 2 It is a network architecture diagram of the NeRF network in Example 1 and Example 2 of the present invention.

[0033] Figure 3 This is a schematic diagram of wheat data shooting in Example 2 of the present invention.

[0034] Figure 4 This is a schematic diagram of obtaining and shooting a reference image in Embodiment 2 of the present invention.

[0035] Figure 5 It is a schematic diagram of the sampling strategy based on the improved NeRF network in Example 2 of the present invention.

[0036] Figure 6 This is a result diagram of a new viewing angle image formed in Example 2 of the present invention. DETAILED DESCRIPTION

[0037] In order to clearly illustrate the technical features of this solution, this solution is described below through a specific implementation method.

[0038] Example 1: See Figure 1 , an embodiment of the present invention provides a method for synthesizing wheat from a new perspective based on an improved neural radiation field group, the method comprising the following steps:

[0039] Step S1: take images of group wheat and construct a group wheat dataset;

[0040] Step S2: construct the neural radiation field of the group wheat scene, generate rays in the neural radiation field, obtain sampling points according to the neural radiation field sampling strategy, and obtain the 3D position and direction of the sampling points;

[0041] Step S3: Input the 3D position x of the sampling point into the multi-plane hash encoder, and then input the encoded data into the neural radiance field network for rendering loss training to predict the sampled color and density;

[0042] Step S4: After the training is completed, a new image with a new perspective can be obtained.

[0043] Wherein, the step S1 is specifically as follows:

[0044] A set of wheat images from different peripheral perspectives were taken using a portable camera as the experimental dataset.

[0045] A global overhead image covering the wheat group is taken as test supervision data for subsequent training steps.

[0046] The sampling strategy in step S2 is specifically: first perform stratified random sampling, then perform importance sampling; stratified random sampling is to divide the population into several sub-populations using one or more variables; wherein, the importance sampling is specifically: coarse sampling between the interval [a, b] Points , and obtain its weight , then according to The corresponding weights get new Sampling points , the subsequent rendering process will be for add .

[0047] The step S3 is specifically as follows:

[0048] Step S301: Input the 3D position of the sampling point into the multi-plane hash encoder to obtain the encoded spatial position ;

[0049] Step S302: The encoded With observation direction Feed it into the Neural Radiance Field Network for rendering loss training;

[0050] Step S303: predict the color and density of the sampling points.

[0051] Wherein, the step S301 is specifically as follows:

[0052] Use three 2D multi-resolution hash encoders to 3D position coordinates of the sampling points The projection coordinates are encoded as follows:

[0053] ;

[0054] in, is the projection coordinate The plane-level geometric features, Is a plane Multi-resolution hash coding of Representative level, Represents the feature size of each entry, and then concatenates the results to get the encoded spatial position :

[0055] ;

[0056] in, They represent the encoding results of the sampling points projected onto the xy, yz and xz planes in three-dimensional space using a 2D multi-resolution hash encoder, respectively. Indicates that the feature is connected to a The concatenation operator on the channel vector of .

[0057] The input of the neural radiation field network is: the encoded spatial position and observation direction ; Among them, the multi-plane hash encoder is used to improve the resolution and expression ability of rendering while improving the rendering speed; the structure of the neural radiation field network includes multiple layers of fully connected layers, and each fully connected layer is connected by a nonlinear activation function.

[0058] The rendering loss training is achieved by minimizing the loss function, and the loss function formula is:

[0059] ;

[0060] in is the total loss function, represents a ray starting from the center of the camera, is the set of all rays, is the network prediction along the ray The rendered color value, is obtained from the real image along the ray The corresponding color value, is a parameter in the network, is a hyperparameter used to balance the weight between RGB color reconstruction loss and regularization loss.

[0061] Among them, in the step S302, after each training cycle step, the rendering formula of the neural radiation field network is used according to the reference image The formulation perspective Predicting new images , and the new image With reference image Perform MSE loss fitting, the loss fitting formula is:

[0062] ;

[0063] Refers to the number of pixels, Refers to the RGB value of the corresponding pixel in the reference image, Refers to the RGB value of the corresponding pixel in the predicted image.

[0064] Example 2: See Figure 1 This embodiment provides a convenient method for synthesizing new perspectives of wheat in a group based on an improved neural radiation field, and the steps include:

[0065] S1: Figure 3 As shown, the group wheat dataset is taken. Adhering to the principle of convenience and practicality, a set of 5 to 10 group wheat images from different perspectives are taken using a digital mobile phone as experimental data;

[0066] S2: The group and density of group wheat data. It was difficult to capture the central wheat plant and thus impossible to reconstruct the central area. Therefore, a top-down image was added. This image captured the central area of ​​the wheat. As a reference image, it can play a key role in NeRF fitting the wheat image in the central area in subsequent operations. Figure 4 As shown, take a reference image and use a digital phone camera to take a global image covering the whole from top to bottom as the test supervision data in the training step;

[0067] S3: Figure 2 This is the improved NeRF network framework proposed in this patent, which will construct the neural radiation field (NeRF) of the group wheat scene and generate rays in the neural radiation field, such as Figure 5 As shown, the sampling points are acquired according to the NeRF sampling strategy, and the 3D position and direction of the sampling points are obtained: Specifically,

[0068] The sampling strategy of NeRF network is to perform stratified sampling first and then importance sampling. Stratified random sampling is to divide the population into several sub-populations using one or more variables. The importance sampling strategy is to calculate the cumulative distribution function (cdf) according to the known probability density function (pdf) after the sampling in the previous step, and then resample according to the inverse cumulative distribution function. The sampling process is as follows: coarse sampling between the interval [a, b] Points , and obtain its weight Then according to The corresponding weights get new Sampling points , the subsequent rendering process will be for add , thereby achieving the purpose of sampling according to weights. In short, stratified sampling can collect key areas as much as possible and reduce calculations.

[0069] S4: Use the multi-plane hash encoder De(x) to encode the location of the sampling points. This step can not only improve the training speed of the model but also better reflect the characteristics of dense scenes:

[0070] Use the multi-plane hash encoder De(x) to encode the position of the sampling point to obtain This step can not only improve the training speed of the model but also better reflect the characteristics of dense scenes, as follows:

[0071] For a given coordinate , use three 2D multi-resolution hash encoders to encode its projection coordinates, the formula is as follows:

[0072] ;

[0073] in is the projection coordinate The plane-level geometric features, Is a plane Multi-resolution hash coding of Representative level, Represents the feature size of each entry, and then concatenates the results to get the encoded spatial position :

[0074] ;

[0075] in, They represent the encoding results of the sampling points projected onto the xy, yz and xz planes in three-dimensional space using a 2D multi-resolution hash encoder, respectively. Indicates that the feature is connected to a The concatenation operator on the channel vector of .

[0076] S5: The encoded With direction Send it to the NeRF network for rendering loss training to predict the color and density of the sampling point:

[0077] The NeRF network will learn how to With direction Mapped to the corresponding color and density. This is achieved by minimizing a loss function that measures the difference between the color and density predicted by the network and the actual image. The loss function formula is as follows:

[0078] ;

[0079] in is the total loss function, represents a ray starting from the center of the camera, is the set of all rays, is the network prediction along the ray The rendered color value, is obtained from the real image along the ray The corresponding color value. is a parameter in the network, is a hyperparameter used to balance the weight between RGB color reconstruction loss and regularization loss.

[0080] S6: After each training cycle step, NeRF’s rendering formula is used to calculate the reference image. The formulation perspective Predicting new images , and the new image With reference image Perform MSE loss fitting:

[0081] Assuming that the total number of training steps is M (50,000), the reference loss is fitted at each step increment of one or two thousand (such as 2000, 4000...8000, etc.), using the mean square error (MSE) loss function, which is defined as:

[0082] ;

[0083] Where n is the number of training data, training data The true output value is The predicted value is The mean square error function measures the quality of the model by calculating the square of the distance between the predicted value and the actual value. The smaller the value, the better. The reference image will be subjected to pixel-by-pixel MSE color loss with the predicted image to fit the network's prediction effect on the central area: , Refers to the number of pixels, Refers to the RGB value of the corresponding pixel in the reference image, Refers to the RGB value of the corresponding pixel in the predicted image.

[0084] S7, training is completed, such as Figure 6As shown, new images from other angles except the existing ones can be obtained with good quality, and the dense wheat area in the center is successfully reconstructed.

[0085] Given a set of relatively sparse wheat group images and a global reference image, a neural radiation field is constructed to generate rays. The position and direction of the sampling point are obtained according to the sampling strategy. The position of the sampling point is put into a multi-plane hash encoder to obtain the multi-dimensional features of the encoded sampling point. The direction of the point is input into the NeRF network for training. After each training cycle step, a new image with the same perspective as the reference image is rendered, and loss fitting is performed on the two images so that the intermediate occluded position of the wheat group can also be inferred. After the training is completed, new images of wheat groups from other perspectives can be obtained.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A new perspective synthesis method for wheat based on improved neural radiation field group, characterized in that: The synthesis method comprises the following steps: Step S1: take images of group wheat and construct a group wheat dataset; Step S2: construct the neural radiation field of the group wheat scene, generate rays in the neural radiation field, obtain sampling points according to the neural radiation field sampling strategy, and obtain the 3D position and direction of the sampling points; Step S3: Input the 3D position x of the sampling point into the multi-plane hash encoder, and then input the encoded data into the neural radiance field network for rendering loss training to predict the sampled color and density; Step S4: After training is completed, a new image with a new perspective can be obtained; Wherein, the step S1 is specifically as follows: A set of wheat images from different peripheral perspectives were taken using a portable camera as the experimental dataset. A global overhead image covering the wheat group is taken as a reference image for the test supervision data in the subsequent training step. P ; Wherein, the step S3 is specifically as follows: Step S301: Input the 3D position of the sampling point into the multi-plane hash encoder to obtain the encoded spatial position ; Step S302: The encoded With observation direction d Feed it into the Neural Radiance Field Network for rendering loss training; Step S303: predicting the color and density of the sampling points; The step S301 is specifically as follows: Use three 2D multi-resolution hash encoders to 3D position coordinates of the sampling points The projection coordinates are encoded as follows: ； in, is the projection coordinate The plane-level geometric features, Is a plane Multi-resolution hash coding of Representative level, Represents the feature size of each entry, and then concatenates the results to get the final encoded spatial position ; ； in, Respectively represent the projection of the sampling point in three-dimensional space to and The encoding result of the plane using the 2D multi-resolution hash encoder, symbol Indicates that the feature is connected to a The concatenation operator in the channel vector of ; Among them, in the step S302, after each training cycle step, the rendering formula of the neural radiation field network is used according to the reference image P The formulation perspective Predicting new images O , and the new image O With reference image P Perform MSE loss fitting, the loss fitting formula is: ； Refers to the number of pixels, Refers to the RGB value of the corresponding pixel in the reference image, Refers to the RGB value of the corresponding pixel in the predicted image.

2. The method for synthesizing wheat from a new perspective based on an improved neural radiation field group according to claim 1 is characterized in that: The sampling strategy in step S2 is specifically: first perform stratified random sampling, then perform importance sampling; stratified random sampling is to divide the population into several sub-populations using one or more variables; wherein, the importance sampling is specifically: in the interval Coarse sampling Points , and obtain its weight , then according to The corresponding weights get new Sampling points , the subsequent rendering process will be for and .

3. The method for synthesizing wheat from a new perspective based on an improved neural radiation field group according to claim 1 is characterized in that: The input of the neural radiation field network is: the encoded spatial position and observation direction d ; Among them, the multi-plane hash encoder is used to improve the resolution and expression ability of rendering while increasing the rendering speed; The structure of the neural radiation field network includes multiple fully connected layers, and each fully connected layer is connected by a nonlinear activation function.

4. The method for synthesizing wheat from a new perspective based on an improved neural radiation field group according to claim 1 is characterized in that: The rendering loss training is achieved by minimizing the loss function, and the loss function formula is: ； in is the total loss function, represents a ray starting from the center of the camera, R is the set of all rays, is the network prediction along the ray The rendered color value, is obtained from the real image along the ray The corresponding color value, is a parameter in the network, is a hyperparameter used to balance the weight between RGB color reconstruction loss and regularization loss.

Images