A neural-radiance-field-based three-dimensional phenotype measurement method for litchi

Abstract

The application discloses a litchi three-dimensional phenotype measurement method based on a neural radiation field, and comprises the following steps: S1, an image or a video of a litchi fruit is shot by using a mobile phone, a two-dimensional image set is made, and the image is labeled; S2, COLMAP technology is used to collect key points of the labeled image, corresponding points between multiple images are found, and the positions and directions of cameras are estimated by using the points; S3, a NeRF network based on a ray marching technology and a tensor radiation field is used to train the labeled image set, and a three-dimensional implicit representation model is obtained; S4, a three-dimensional display expression mode grid model is generated by using volume density, color, light and a three-dimensional implicit scene, and point cloud data of the litchi fruit is obtained by judging and screening abnormal points; and S5, the phenotype parameters of the litchi are measured according to the point cloud data. The application is shot by using a mobile phone, and the ray marching technology and the tensor radiation field are introduced into the NeRF, so that the problems of fruit occlusion and high equipment cost existing in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for measuring the three-dimensional phenotype of litchi based on neural radiation fields. Background Technology

[0002] In lychee production, phenotypic data is crucial, as it relates to lychee growth, ripening, yield, disease resistance, and other quality-related characteristics. Phenotypic data helps researchers and growers understand the growth habits and production potential of lychees. Recording and analyzing the fruit size, shape, color, and other characteristics of different lychee varieties provides valuable reference information for lychee breeding. Breeding varieties with strong adaptability, high yield, and good taste is key to improving the competitiveness of the lychee industry.

[0003] Lychee fruits typically exhibit an oval to nearly heart-shaped appearance, with a slightly pointed or concave apex and a generally broad and rounded base. The fruit's outer surface is rough and scaly. Complete phenotypic measurements of lychees using manual methods or two-dimensional imaging techniques are difficult to achieve; therefore, three-dimensional imaging techniques are needed to obtain precise shape parameters and dimensions. This detailed description helps researchers better understand and classify different lychee varieties and provides valuable data for lychee genetic and physiological research.

[0004] In the fruit phenotypic measurement method, patent CN110660127A constructs the phenotypic of wolfberry fruit based on three-dimensional point cloud scanning technology, and patent CN110458882A measures the phenotypic of plant fruit based on computer vision, fixing the fruit on the yield measurement auxiliary board to take pictures, and performing edge detection and outer contour confirmation.

[0005] Currently, commonly used methods for acquiring 3D data include: The first method involves using a laser scanner to scan an object and obtain its point cloud information. The drawback of this method is that laser scanners are expensive and time-consuming, making it difficult to promote. The second method uses a depth camera to simultaneously acquire RGB images and depth information. However, in bright sunlight in outdoor scenes, infrared light interference can lead to loss or misinterpretation of depth data, and depth cameras are also more expensive than ordinary RGB cameras. The third method is multi-view reconstruction, which uses multiple views or images to reconstruct the 3D structure of an object or scene. It uses a camera or mobile phone to acquire a series of continuous multi-directional views of the object and uses algorithms such as SFM and NeRF to obtain the object's 3D structure. Compared to the first two methods, multi-view reconstruction technology only requires a conventional camera. By observing the scene from multiple angles, it helps to reconstruct hidden or occluded parts of objects. Compared to laser scanning, multi-view reconstruction can obtain a more complete model for occluded lychees hanging on fruit trees.

[0006] Therefore, how to provide a method for measuring the three-dimensional phenotype of litchi based on neural radiation fields is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a method for measuring the three-dimensional phenotypic characteristics of litchi based on neural radiation fields, which solves the problems of fruit occlusion and high equipment costs in phenotypic data measurement.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for measuring the three-dimensional phenotype of litchi based on neural radiation fields includes the following steps:

[0010] S1, Image Acquisition: Acquire images or video data of lychee fruit using a camera device, create a two-dimensional image set of lychee fruit based on the image or video data, and annotate the two-dimensional images;

[0011] S2, Image Matching: Using COLMAP technology, key points of the labeled two-dimensional images are collected to find corresponding points between multiple two-dimensional images, and the position and orientation of the camera device are estimated based on the corresponding points;

[0012] S3, Model Training: Using a NeRF network improved based on ray-travel technology and tensor radiation field, the labeled 2D image set and the corresponding positions and orientations of the 2D images are input into the improved NeRF network for training to obtain a 3D implicit representation model.

[0013] S4, Obtain point cloud data: Based on the three-dimensional implicit representation model, a three-dimensional display expression mesh model is generated using volume density, color, light and three-dimensional implicit scene, and point cloud data of lychee fruit is obtained by judging and filtering out anomalies.

[0014] S5, Fruit phenotypic measurement: The phenotypic parameters of the litchi are measured based on the point cloud data.

[0015] Furthermore, step S1 specifically includes:

[0016] S11, Use the camera device to take 360° surround shots of the lychee fruit and collect images or videos of the lychee fruit;

[0017] S12, the images or videos are filtered, images are extracted from the video at preset interval frames, ensuring that the overlap between each image is more than 40%, and blurry or jittery images are deleted to obtain a two-dimensional image set;

[0018] S13, Use the interactive segmentation and annotation tool to annotate the outline of the lychee fruit in the two-dimensional image and save the Mask file;

[0019] S14, the original two-dimensional image's RGB three channels are expanded into RGBA four channels, and an Alpha channel is added to control the transparency of each pixel in the image. According to the Mask file, the transparency of the lychee fruit is set to 1, and the background transparency is set to 0.

[0020] Furthermore, step S2 specifically includes:

[0021] S21, Feature Detection and Extraction: Use the SIFT algorithm to find sparse feature points from the labeled 2D image and extract key points and SIFT descriptors;

[0022] S22, Feature Matching and Geometric Verification: The SIFT algorithm is used to find the correspondence between key points in different two-dimensional images, and the matching key point pairs are found according to the SIFT descriptor. The geometric relationship between different two-dimensional images is estimated based on the matching key point pairs.

[0023] S23, Camera pose estimation: The camera pose of the new image is estimated using the PnP algorithm. The error between the two-dimensional points observed by the camera and the projection points calculated from the camera parameters and three-dimensional points is minimized using BundleAdjustment. The sum of all projection errors in the entire scene is minimized through iterative optimization to obtain accurate three-dimensional points and internal and external parameters of the camera.

[0024] S24, Depth Image Generation: The 2D image is distorted using camera parameters. The processed 2D images are compared with each other. Stereo Matching is used to find corresponding pixels between the images. The depth of each pixel is calculated. The depth is obtained by minimizing the matching cost. The depth map is filtered by bilateral filtering to remove inconsistent depth values ​​and noise.

[0025] Furthermore, step S3 specifically includes:

[0026] S31 introduces ray travel technology in NeRF, using exponential stepping, blank skipping and sample compression to sample the area near the camera device;

[0027] S32 introduces the tensor radiation field from TensoRF and uses VM tensor decomposition to model the radiation field;

[0028] S33 uses frequency encoding to encode the input coordinates into a higher-dimensional space, uses differentiable volume rendering, and for each pixel, moves along a ray starting from the pixel to calculate the projection between adjacent sampling points, the transmission of light through air or other transparent media between consecutive sampling points along the ray direction, calculates the weight value of the point based on the density of the sampling point and the projection of the previous sampling point, calculates and accumulates the color value of each of all sampling points with the weight value to finally obtain the color value of the pixel;

[0029] S34. For each pixel in the two-dimensional image, repeat the operation of step S33 to obtain a three-dimensional implicit representation model.

[0030] Furthermore, the specific process of modeling the radiation field using VM tensor decomposition in step S32 is as follows:

[0031] Define G σ ∈R I×J×K It is a three-dimensional tensor, G c ∈R I×J×K×P It is a four-dimensional tensor, where I, J, and K represent the feature resolutions of the X, Y, and Z axes, respectively, and P represents the number of channels for the appearance features. Through VM decomposition, G... σ With G c Decomposed into:

[0032]

[0033]

[0034] In the formula, G σ G is a three-dimensional geometric tensor; c For appearance tensor; R σ For G σ The decomposed tensor set; R c For G c The decomposed tensor set; r is a member of the tensor set; v X M is the X-axis vector; Y Z M is the YZ axis plane matrix; X Z M is the XZ-axis plane matrix; X Y is the XY-axis plane matrix; b is the appearance; B is the appearance matrix;

[0035] The model is transformed to the continuous domain using trilinear interpolation. The calculated interpolation value is in For trilinear interpolation, For linear interpolation along the X-axis, Bilinear interpolation in the YZ plane yields:

[0036]

[0037]

[0038] Density and color in a radiation field are represented as follows:

[0039]

[0040] In the formula, σ is the density of the radiation field, c is the color of the radiation field, S is the MLP, and d is the direction of the ray.

[0041] Furthermore, in step S33, the formula for calculating and summing the color values ​​and weight values ​​of all sampling points to obtain the final color value of the pixel is as follows:

[0042]

[0043] In the formula, C is the pixel color value; Q is the set of sampling points; p and q are sampling points; τ q σ is the transmittance at sampling point q; q Δ is the density of q sampling points; q σ is the ray step size for sampling point q; p c is the density of the sampling points p; q The color of the q-sample point; Δ p Let p be the ray step size for sampling point p.

[0044] Furthermore, step S4 specifically includes:

[0045] S41, Establish a query network: Based on the three-dimensional implicit representation model, create a mesh volume covering the entire object, establish a query network on the dense mesh, query any point in the three-dimensional space and a given viewing direction to obtain the color and density values ​​of these points;

[0046] S42, Mesh Extraction: After determining the density values ​​of all cells, the Marching Cube Algorithm is used to extract the mesh, obtaining the vertices and triangles of the mesh;

[0047] S43, Color Assignment: Assign color to each grid vertex, determine whether the grid vertex is occluded, and assign color values ​​to the grid vertices that are not occluded.

[0048] Furthermore, step S5 specifically includes:

[0049] S51, Remove outliers from 3D model: Read the mesh model and obtain vertex data to get point cloud data. Filter the point cloud data by removing outliers by radius. Eliminate points with few neighboring points within the surrounding radius to reduce noise. Estimate the normals from the point cloud and delete points with inward-facing normals to remove internal points. Determine the depth of the points and retain only the point cloud on the object's surface.

[0050] S52, 3D phenotypic measurement: Two points are manually selected from the point cloud data to measure the diameter. These two points need to be the same as the measurement locations of the actual fruit. The Mesh model is then used to calculate the volume and surface area of ​​the fruit.

[0051] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. Compared with devices such as lidar and depth cameras, the device of this invention has low cost, is not affected by terrain factors, has fast data acquisition speed, and can quickly generate three-dimensional models.

[0053] 2. This invention is based on the NeRF model and introduces the ray-tracing technique in Instant-NGP. It uses the voxel grid features in TensoRF to represent the radiation field, quickly collects relevant sampling points, and effectively shortens the training time of the NeRF model.

[0054] 3. The improved network has a higher PSNR value and less noise in the Mesh model and point cloud model.

[0055] 4. This invention provides a phenotypic measurement method that does not damage the fruit and can accurately measure any shape of lychee fruit. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0057] Figure 1 The flowchart for measuring the three-dimensional phenotype of litchi based on neural radiation field provided by this invention.

[0058] Figure 2 The diagram shows the NeRF network structure based on ray travel technology and tensor radiation field improvement provided by this invention.

[0059] Figure 3 A comparison chart of litchi phenotypic measurement results based on various models provided by this invention.

[0060] Figure 4 The image shows the phenotypic measurement results of litchi based on the improved NeRF network model provided by this invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention discloses a method for measuring the three-dimensional phenotypic characteristics of litchi based on neural radiation fields. This method utilizes images or videos captured by a mobile phone to perform three-dimensional reconstruction of litchi fruits, thereby achieving phenotypic measurement and providing valuable data for researchers' physiological studies of litchi fruits. This embodiment is developed based on the Ubuntu 20.04 operating system and trained on the PyTorch deep learning framework platform using an Nvidia GeForce RTX 3090 GPU to train the NeRF algorithm, reconstruct the three-dimensional model of the litchi fruit, and measure the fruit's phenotypic characteristics. Figure 1 As shown, this embodiment includes the following steps:

[0063] S1, Image Acquisition: Image or video data of lychee fruits is acquired using a camera device. A two-dimensional image set of lychee fruits is created based on the image or video data, and the two-dimensional images are labeled. The specific implementation process is as follows:

[0064] The S11 allows users to take 360° panoramic photos of lychee fruits using the phone's camera, capturing 80 to 120 images. During shooting, the aperture and focus need to be manually adjusted to ensure uniform lighting in each image. Alternatively, users can use the phone's camera to take a video of the lychee fruit at a constant speed, setting the image resolution to 2448×2048.

[0065] S12: Filter the acquired images or videos. For videos, extract and save images every 5 frames. Filter the acquired images, removing blurry images of lychee fruits, ensuring that two consecutive images have more than 40% overlap, and removing images with significant lighting differences, to obtain a two-dimensional image set of lychee fruits.

[0066] S13. Use EiSeg software to annotate the 2D image. Import the 2D image into the software, import the segmentation model into EiSeg software, click on the lychee fruit in the 2D image to annotate the outline of the lychee fruit, and save the Mask file in PNG format in sequence.

[0067] S14 uses the OpenCV library to read the original 2D image, expands the RGB three channels into RGBA four channels, and adds an Alpha channel to control the transparency of each pixel in the image. According to the Mask file, the transparency of the lychee fruit is set to 1 (non-transparent) and the background transparency is set to 0 (transparent).

[0068] S2, Image Matching: Key points of the annotated 2D images are acquired using COLMAP technology to find corresponding points between multiple 2D images. The position and orientation of the camera device are estimated based on these corresponding points. The specific implementation process is as follows:

[0069] S21, Feature Detection and Extraction: The SIFT algorithm is used to find sparse feature points from the labeled 2D image and extract key points. For each key point, a SIFT descriptor is calculated. The descriptor is a vector representing the characteristics of the image region around the key point.

[0070] S22, Feature Matching and Geometric Verification: The SIFT algorithm is used to find the correspondence between feature points in different 2D images. Then, based on their descriptors, keypoint pairs matching between different 2D images are found. Using the matched keypoint pairs, the geometric relationship between different 2D images is estimated. Sequential matching reduces the number of 2D image pairs that need to be matched for each 2D image. It no longer needs to match with all 2D images except itself; it only needs to match with its neighboring images, thus greatly speeding up feature matching.

[0071] S23, Camera Pose Estimation: The Perspective-n-Point (PnP) algorithm is used to estimate the camera pose of all input images. Bundle Adjustment is used to minimize the difference between the 2D points observed by the camera and the projected points calculated using camera parameters and 3D points. Iterative optimization is then used to minimize the sum of all projection errors in the entire scene, obtaining accurate 3D points and the camera's intrinsic and extrinsic parameters. This process typically starts with two images and then gradually adds other images to the reconstruction.

[0072] S24, Depth Image Generation: For each image, distortion correction is performed using the corresponding camera parameters. Then, it is compared with other images. Stereo Matching is used to find corresponding pixels between images and estimate the depth of each pixel. The depth map represents the distance of each pixel as seen from the camera's perspective. The depth is obtained by minimizing the matching cost. The depth map is filtered by bilateral filtering to remove inconsistent depth values ​​and noise.

[0073] S3, Model Training: (e.g.) Figure 2As shown, a NeRF network based on ray-tracing technology and tensor radiation field is used to train a labeled 2D image set to obtain a 3D implicit representation model. The specific implementation process is as follows:

[0074] S31, based on NeRF, introduces the ray-tracing technique from Instant-NGP to reduce sampling overhead. It employs exponential stepping, blank skipping, and sample compression to sample more locations near the camera and reduce sampling of more distant areas.

[0075] Among them, exponential stepping refers to the sampling points advancing exponentially in the ray direction to obtain scene information at a higher resolution; blank skipping means that if the density of a certain sampling point is close to zero (i.e., the transparency is 0) is found during the ray's movement, the point can be skipped without sampling to improve sampling efficiency; sample compression refers to compressing the sampling points to reduce storage and computational overhead.

[0076] S32 introduces TensoRF to represent the radiation field as a voxel mesh, using a three-dimensional voxel multichannel mesh to represent the function of the radiation field. This function is then divided into a geometric mesh G through feature channels. σ and the appearance grid G c This is used to output volume density and RGB color values ​​based on the viewing direction, respectively. The geometric mesh directly outputs the volume density value, while the appearance mesh generates RGB values ​​via an MLP. The radiation field is modeled using VM tensor decomposition.

[0077] Define G σ ∈R I×J×K It is a three-dimensional tensor, G c ∈R I×J×K×P It is a four-dimensional tensor, where I, J, and K represent the feature resolutions of the X, Y, and Z axes, respectively, and P represents the number of channels for the appearance features. Through VM decomposition, G... σ With G c Decomposed into:

[0078]

[0079]

[0080] In the formula, G σ G is a three-dimensional geometric tensor; c For appearance tensor; R σ For G σ The decomposed tensor set; R c For G c The decomposed tensor set; r is a member of the tensor set; v X M is the X-axis vector; Y Z M is the YZ axis plane matrix;X Z M is the XZ-axis plane matrix; X Y is the XY-axis plane matrix; b is the appearance; B is the appearance matrix.

[0081] The model is transformed to the continuous domain using trilinear interpolation. The calculated interpolation value is in For trilinear interpolation, For linear interpolation along the X-axis, Bilinear interpolation in the YZ plane yields:

[0082]

[0083]

[0084] Density and color in a radiation field are represented as follows:

[0085]

[0086] In the formula, σ is the density of the radiation field, c is the color of the radiation field, S is the MLP, and d is the direction of the ray.

[0087] S33 uses frequency encoding to encode the input coordinates into a high-dimensional space. Differentiable volume rendering is used. For each pixel, along a ray originating from the pixel, the projection between adjacent sampling points is calculated. The light transmission through air or other transparent media between consecutive sampling points along the ray direction is also considered. The weight value of the sampling point is calculated based on its density and the projection from the previous sampling point. The color value of each pixel is calculated and summed with its weight value to obtain the final color value. The calculation formula is as follows:

[0088]

[0089] In the formula, C is the pixel color value; Q is the set of sampling points; p and q are sampling points; τ q σ is the transmittance at sampling point q; q Δ is the density of q sampling points; q σ is the ray step size for sampling point q; p c is the density of the sampling points p; q The color of the q-sample point; Δ p Let p be the ray step size for sampling point p.

[0090] S34. For each pixel in the 2D image, repeat step S33, adjusting parameters such as contrast and brightness as needed to optimize the rendering results and generate the rendered 2D image, depth map, volume density, color, lighting, and implicit 3D scene.

[0091] S4, Obtaining Point Cloud Data: Based on a 3D implicit representation model, a 3D display representation mesh model is generated using volume density, color, lighting, and the 3D implicit scene. Point cloud data of the lychee fruit is obtained by filtering the vertices of the mesh, as detailed below:

[0092] S41, Establish a query network: Load the weighted model generated after training, create a mesh volume covering the entire object, establish a query network on the dense mesh, query any point in the three-dimensional space and a given viewing direction, and obtain the color and density values ​​of these points.

[0093] S42, Mesh Extraction: After determining the density values ​​of all cells, the Marching Cube Algorithm is used to extract the mesh, obtaining the vertices and triangles of the mesh. For extra generated triangles, connected triangles can be grouped together, and only the group containing the most triangles is retained.

[0094] S43, Color Assignment: A color is assigned to each vertex. For each vertex in the mesh, it is determined whether it is occluded. The color value and density value of that vertex are multiplied. Unoccluded vertices are then assigned a color. The triangular network is processed and analyzed using the Trimesh and Pyrender libraries to save the network.

[0095] S5, Phenotypic Measurement: Phenotypic parameters of litchi are measured based on point cloud data. The Open3D library is used to measure litchi point cloud data, including diameter. The specific implementation process is as follows:

[0096] S51, 3D model outlier removal: Read the mesh model and obtain vertex data to obtain point cloud data. Filter the point cloud data and remove outliers by radius. Eliminate points with few neighboring points within the surrounding radius to reduce noise. Estimate the normals from the point cloud and delete points pointing inwards to remove internal points. Determine the depth of the points and retain only the point cloud on the object surface.

[0097] S52, 3D phenotypic measurement: Two points are manually selected from the point cloud data to measure the diameter. These two points need to be the same as the measurement locations of the actual fruit. The Trimesh library can be used to calculate the volume and surface area of ​​the Mesh model.

[0098] Assuming the image size is m×n, and using PSNR as a comparison metric for 3D rendering results, the mean squared error is first calculated for the original image I and the rendered result K, and then the PSNR is calculated, as shown in the following formula:

[0099]

[0100]

[0101] In the formula, MSE is the mean energy difference between the real image and the noisy image; PSNR is the peak signal-to-noise ratio; MAX I This is the maximum possible pixel value for the image, typically 255.

[0102] Table 1 shows a comparison of the relevant parameters of each model; Table 2 shows a comparison of the application results of each model in multiple scenarios; Table 3 shows a comparison of the measurement results of the litchi phenotype based on each model.

[0103] Table 1 Comparison of Model Results

[0104] Model PSNR Time Model size NeRF 31.54 3.5h 16.8MB Mip-NeRF360 35.69 5.5h 108.1MB Instant-NGP 31.77 5min 25.7MB TensoRF 32.39 17.4min 85.9MB Improved network model 34.2 12.4min 48.8MB

[0105] As can be seen from Table 1, the training speed of the improved network model is more than 15 times faster than NeRF and Mip-NeRF360, the PSNR value is higher than Instant-NGP and TensoRF, and the image noise value is less than TensoRF.

[0106] Table 2 Comparison of Results Across Multiple Scenarios

[0107]

[0108]

[0109] As shown in Table 2, compared with NeRF, Instant-NGP and TensoRF, the improved network model can achieve better reconstruction results in both single fruit and multiple fruit scenarios.

[0110] Table 3 Phenotypic Measurement Results

[0111] Model Longitudinal diameter (mm) transverse diameter (mm) Instant-NGP 32.37 31.91 TensoRF 35.45 36.2 Improved network model 31.93 30.52 GT 30.80 29.70

[0112] As can be seen from Table 3, the reconstruction results obtained by the improved network model have higher accuracy than those of Instant-NGP and TensoRF.

[0113] The results of litchi reconstruction based on each model are as follows: Figure 3 As shown, it can be seen that the results reconstructed by NeRF are rather blurry and lack details; compared with the results reconstructed by Instant-NGP and TensoRF and the improved network model, there are more noise points at the edges.

[0114] The results of measuring litchi phenotypes based on the improved network model are as follows: Figure 4 As shown, the average error of the measurement results is 0.97 mm, the average relative error is 3.21%, and the volume and surface area measured based on the mesh are 15.179 cm². 3 and 30.62cm 2.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for measuring the three-dimensional phenotype of litchi based on neural radiation fields, characterized in that, Includes the following steps: S1, Image Acquisition: Acquire images or video data of lychee fruit using a camera device, create a two-dimensional image set of lychee fruit based on the image or video data, and annotate the two-dimensional images; S2, Image Matching: Using COLMAP technology, key points of the labeled two-dimensional images are collected to find corresponding points between multiple two-dimensional images, and the position and orientation of the camera device are estimated based on the corresponding points; S3, Model Training: Based on the improved NeRF network using ray-travel technology and tensor radiation field, the labeled 2D image set and the corresponding positions and orientations of the 2D images are input into the improved NeRF network for training to obtain a 3D implicit representation model. Step S3 specifically includes: S31 introduces ray travel technology in NeRF, using exponential stepping, blank skipping and sample compression to sample the area near the camera device; S32 introduces the tensor radiation field from TensoRF and uses VM tensor decomposition to model the radiation field; S33 uses frequency encoding to encode the input coordinates into a higher-dimensional space, uses differentiable volume rendering, and for each pixel, moves along a ray starting from the pixel to calculate the projection between adjacent sampling points, the transmission of light through air or other transparent media between consecutive sampling points along the ray direction, calculates the weight value of the point based on the density of the sampling point and the projection of the previous sampling point, calculates and accumulates the color value of each of all sampling points with the weight value to finally obtain the color value of the pixel; S34, For each pixel in the two-dimensional image, repeat the operation of step S33 to obtain a three-dimensional implicit representation model; S4, Obtain point cloud data: Based on the three-dimensional implicit representation model, a three-dimensional explicit representation mesh model is generated using volume density, color, light, and the three-dimensional implicit scene. Point cloud data of lychee fruit is obtained by judging and filtering out anomalies. Step S4 specifically includes: S41, Establish a query network: Based on the three-dimensional implicit representation model, create a mesh volume covering the entire object, establish a query network on the dense mesh, query any point in the three-dimensional space and a given viewing direction to obtain the color and density values ​​of these points; S42, Mesh Extraction: After determining the density values ​​of all cells, the Marching Cube algorithm is used to extract the mesh, obtaining the vertices and triangles of the mesh; S43, Color Assignment: Assign color to each grid vertex, determine whether the grid vertex is occluded, and assign color values ​​to the grid vertices that are not occluded; S5, Fruit phenotypic measurement: The phenotypic parameters of the litchi are measured based on the point cloud data.

2. The method for measuring the three-dimensional phenotype of litchi based on neural radiation fields according to claim 1, characterized in that, Step S1 specifically includes: S11, Use the camera device to take 360° surround shots of the lychee fruit and collect images or videos of the lychee fruit; S12, the images or videos are filtered, images are extracted from the video at preset interval frames, ensuring that the overlap between each image is more than 40%, and blurry or jittery images are deleted to obtain a two-dimensional image set; S13, Use the interactive segmentation and annotation tool to annotate the outline of the lychee fruit in the two-dimensional image and save the Mask file; S14, the original two-dimensional image's RGB three channels are expanded into RGBA four channels, and an Alpha channel is added to control the transparency of each pixel in the image. According to the Mask file, the transparency of the lychee fruit is set to 1, and the background transparency is set to 0.

3. The method for measuring the three-dimensional phenotype of litchi based on neural radiation fields according to claim 1, characterized in that, Step S2 specifically includes: S21, Feature Detection and Extraction: Use the SIFT algorithm to find sparse feature points from the labeled 2D image and extract key points and SIFT descriptors; S22, Feature Matching and Geometric Verification: The SIFT algorithm is used to find the correspondence between key points in different two-dimensional images, and the matching key point pairs are found according to the SIFT descriptor. The geometric relationship between different two-dimensional images is estimated based on the matching key point pairs. S23, Camera pose estimation: The camera pose of the new image is estimated using the PnP algorithm. The error between the two-dimensional points observed by the camera and the projection points calculated by the camera parameters and three-dimensional points is minimized using Bundle Adjustment. The sum of all projection errors in the entire scene is minimized through iterative optimization to obtain accurate three-dimensional points and internal and external parameters of the camera. S24, Depth Image Generation: The 2D image is distorted using camera parameters. The processed 2D images are compared with each other. Stereo Matching is used to find corresponding pixels between the images. The depth of each pixel is calculated. The depth is obtained by minimizing the matching cost. The depth map is filtered by bilateral filtering to remove inconsistent depth values ​​and noise.

4. The method for measuring the three-dimensional phenotype of litchi based on neural radiation fields according to claim 1, characterized in that, The specific process of modeling the radiation field using VM tensor decomposition in step S32 is as follows: definition It is a three-dimensional tensor. It is a four-dimensional tensor, where I, J, and K represent the feature resolutions of the X, Y, and Z axes, respectively, and P represents the number of channels for the appearance features. Through VM decomposition, it is... and Decomposed into: ； In the formula, It is a three-dimensional geometric tensor; For appearance tensor; for The decomposed tensor set; for The decomposed tensor set; r is a member of the tensor set; The X-axis vector; The YZ axis plane matrix; The XZ axis plane matrix; is the XY-axis plane matrix; b is the appearance; B is the appearance matrix; The model is transformed to the continuous domain using trilinear interpolation. The interpolation value is calculated to be... ,in For trilinear interpolation, For linear interpolation along the X-axis, Bilinear interpolation in the YZ plane yields: ； Density and color in a radiation field are represented as follows: ； In the formula, Let be the density of the radiation field, c be the color of the radiation field, S be the MLP, and d be the direction of the ray.

5. The method for measuring the three-dimensional phenotype of litchi based on neural radiation fields according to claim 1, characterized in that, In step S33, the formula for calculating and summing the color value and weight value of each sampling point to obtain the final color value of the pixel is as follows: ； In the formula, C is the pixel color value; Q is the set of sampling points; p and q are sampling points; Let q be the transmittance at sampling point q; Let q be the density of the sampling points; Let q be the ray step size for sampling points; The density of sample points p; The color of the q-sample point; Let p be the ray step size for sampling point p.

6. The method for measuring the three-dimensional phenotype of litchi based on neural radiation fields according to claim 1, characterized in that, Step S5 specifically includes: S51, Remove outliers from 3D model: Read the mesh model and obtain vertex data to get point cloud data. Filter the point cloud data by removing outliers by radius. Eliminate points with few neighboring points within the surrounding radius to reduce noise. Estimate the normals from the point cloud and delete points with inward-facing normals to remove internal points. Determine the depth of the points and retain only the point cloud on the object's surface. S52, 3D phenotypic measurement: Two points are manually selected from the point cloud data to measure the diameter. These two points need to be the same as the measurement locations of the actual fruit. The Mesh model is then used to calculate the volume and surface area of ​​the fruit.

Images