Plant three-dimensional reconstruction and organ segmentation method and device based on multi-camera synchronous acquisition and neural radiation field, medium and product

Abstract

The invention discloses a plant three-dimensional reconstruction and organ segmentation method and device based on multi-camera synchronous acquisition and a nerve radiation field, a medium and a product, and relates to the technical field of image processing, and the method comprises the steps: employing multiple cameras to synchronously acquire a multi-view image of a plant; performing two-dimensional image segmentation on the multi-view image to obtain semantic tags of pixels and construct a Tiff image; training and optimizing a multi-channel plant segmentation nerve radiation field by adopting the Tiff image to obtain an optimized nerve radiation field; extracting a semantic point cloud model of the plant from the optimized neural radiation field by adopting a contour surface extraction algorithm; each pixel point in the semantic point cloud model comprises geometric coordinate information, color attribute information and semantic label information. According to the invention, the efficiency of plant image acquisition and the quality of plant three-dimensional reconstruction and organ segmentation can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, medium and product for plant three-dimensional reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field. Background Art

[0002] Plant 3D point cloud model reconstruction and plant organ segmentation are of great significance in the fields of crop breeding selection and precision management. Through 3D reconstruction and organ segmentation of plants, the morphological structure and growth status of plants can be accurately analyzed, and data support can be provided for subsequent phenotypic analysis and physiological research.

[0003] However, there are still some problems in the existing research on plant 3D point cloud model reconstruction and organ segmentation, mainly manifested in low image acquisition efficiency and poor quality of 3D reconstruction and organ segmentation. Summary of the invention

[0004] The purpose of this application is to provide a method, device, medium and product for plant three-dimensional reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field, which can improve the efficiency of plant image acquisition and the quality of plant three-dimensional reconstruction and organ segmentation.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field, comprising:

[0007] Using multiple cameras to synchronously collect multi-view images of plants; the multi-view images include original images of different viewpoints taken at multiple different points;

[0008] Performing two-dimensional image segmentation on the multi-view image to obtain semantic labels of pixels and constructing a Tiff image; the Tiff image includes four channels, namely, RGB color components of the original image and semantic labels of pixels; the semantic labels are used to represent plant organs;

[0009] The Tiff image is used to train and optimize the multi-channel plant segmentation neural radiation field to obtain an optimized neural radiation field;

[0010] An isosurface extraction algorithm is used to extract a semantic point cloud model of the plant from the optimized neural radiation field; each pixel point in the semantic point cloud model includes geometric coordinate information, color attribute information and semantic label information.

[0011] Optionally, multiple cameras are used to synchronously capture multi-view images of plants, specifically including:

[0012] Multiple cameras in an automated image acquisition system are used to synchronously acquire multi-viewing angle images of plants; the automated image acquisition system comprises: a host computer, a robotic arm, a control box and multiple cameras; the multiple cameras are mounted at the end of the robotic arm; the robotic arm is connected to the control box, the multiple cameras are connected to the host computer, and the control box and the host computer are connected to the same local area network; the host computer stores a camera control program for controlling the multiple cameras to take pictures; the control box stores a robotic arm control program for controlling the robotic arm to move along a preset trajectory.

[0013] Optionally, the multi-channel plant segmentation neural radiation field uses multi-resolution hash coding for position coding, uses spherical harmonic functions for line of sight coding, and predicts volume density, color and semantics through a multi-layer perceptron combination and performs volume rendering.

[0014] Optionally, the Tiff image is used to train and optimize the multi-channel plant segmentation neural radiation field to obtain an optimized neural radiation field, specifically including:

[0015] Input the Tiff image into a multi-channel plant segmentation neural radiation field, predict volume density, color and semantics, and perform volume rendering;

[0016] The light of volume rendering is sampled by a sampling method from coarse to precise, and a loss function is constructed according to the color prediction value of the coarse stage, the color prediction value of the fine stage and the corresponding true value, as well as the semantic prediction value of the coarse stage, the semantic prediction value of the fine stage and the corresponding true value;

[0017] Based on the loss function, the multi-channel plant segmentation neural radiation field is trained and optimized using a back propagation algorithm and a gradient descent optimization strategy to obtain an optimized neural radiation field.

[0018] Optionally, the isosurface extraction algorithm is a Marching Cubes algorithm.

[0019] Optionally, the automated image acquisition system further includes: a cross-shaped bracket; the multiple cameras are fixed on the cross-shaped bracket via the camera bracket; and the cross-shaped bracket is installed at the end of the robotic arm via a mounting hole.

[0020] Optionally, the automated image acquisition system further includes: a hub; and signal lines of the plurality of cameras are connected to the host computer via the hub.

[0021] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field.

[0022] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field.

[0023] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field.

[0024] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0025] The present application provides a method, device, medium and product for plant three-dimensional reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field. By using multiple cameras to synchronously acquire multi-perspective images of plants, rapid acquisition of multi-perspective images of plants is achieved, which improves the image acquisition efficiency compared with manually adjusting the camera position and using a robotic arm to drive a single camera. By using Tiff image training and optimizing the multi-channel plant segmentation neural radiation field, the complex geometric structure and texture details of the plant can be quickly fitted, thereby improving the quality of plant three-dimensional reconstruction and organ segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A flow chart of the method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field provided in this application;

[0028] Figure 2 A structural diagram of the automated image acquisition system provided for this application;

[0029] Figure 3 Operation flow chart of the automated image acquisition system provided for this application;

[0030] Figure 4 A top view of the cross-shaped bracket of the automated image acquisition system provided in this application;

[0031] Figure 5 Three views of the camera bracket of the automated image acquisition system provided in this application;

[0032] Figure 6 Schematic diagram of the motion trajectory of the robotic arm provided for this application;

[0033] Figure 7 The architecture diagram of the multi-layer perceptron combination provided for this application;

[0034] Figure 8 Framework diagram of the plant 3D reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field provided in this application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] First, the relevant technologies of plant 3D reconstruction and organ segmentation are introduced from the two aspects of image acquisition and algorithm.

[0037] In terms of image acquisition, if the plant rotates and the camera is stationary, it depends on a uniform rotating table, and the plant rotation speed cannot be too fast, resulting in an inefficient acquisition mode. If the plant is stationary and the camera is moving, the camera needs to be controlled by a slide rail or a robotic arm. This method can increase the acquisition speed to a certain extent, but due to the large number of viewing angles that need to be acquired and the need to prevent motion blur in the image, the acquisition efficiency of this acquisition method is not high. In addition, the existing methods have strict requirements on the acquisition environment, and often require calibration plates, black background plates, etc. to assist in image segmentation and camera pose inference, which to a certain extent restricts the use scenarios and objects of the methods.

[0038] In terms of algorithms, existing research requires estimating the camera pose, and it takes a long time to infer the precise camera pose. At the same time, after reconstructing the three-dimensional point cloud of plants, existing research needs to pre-process the point cloud in many steps, including denoising and outlier removal, point cloud registration and alignment, filtering and smoothing, geometric repair, and a series of other steps. Even so, the reconstructed point cloud often has holes and damage, resulting in low quality. In addition, the use of point cloud three-dimensional deep learning methods will face problems such as limited number of input point clouds, poor model migration capabilities, and complex point cloud data annotation.

[0039] In view of this, the present application proposes a method, device, medium and product for plant three-dimensional reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field, which can solve the problems of slow image acquisition speed and long camera pose inference time, and realize rapid reconstruction and accurate segmentation of plant three-dimensional point cloud.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0041] In an exemplary embodiment, the present application provides a method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, Figure 1 and Figure 8 As shown, the method includes the following steps 101 to 104.

[0042] Step 101: Use multiple cameras to synchronously capture multi-view images of plants, wherein the multi-view images include original images of different viewpoints taken at multiple different points.

[0043] Specifically, in order to achieve the goal of rapid multi-angle acquisition, an automated image acquisition system consisting of a robotic arm and its control box, a host computer (small host computer) and multiple cameras (six cameras) was designed. The multiple cameras in the automated image acquisition system were used to synchronously capture multi-perspective images of plants.

[0044] like Figure 2 As shown, the automated image acquisition system includes: a host computer 205, a robotic arm 201, a control box 203 and multiple cameras; the multiple cameras are installed at the end of the robotic arm 201 and are located above the measured plant 202; the robotic arm 201 is connected to the control box 203, the multiple cameras are connected to the host computer 205, and the control box 203 and the host computer 205 are connected to the same local area network; the host computer 205 stores a camera control program for controlling the multiple cameras to take pictures; the control box 203 stores a robotic arm control program for controlling the robotic arm 201 to move along a preset trajectory.

[0045] like Figure 3As shown in the figure, the robot control box is programmed with a set of robot posture control and communication programs (i.e., robot control programs) to control the movement of the robot and communicate with the host computer. The small host computer is programmed with a matching camera control program to receive signals sent by the robot and control the camera to shoot. During the acquisition process, the host computer and the robot control box must first be connected to the same local area network. Then, the robot control box is used to start the robot and run the pre-programmed program to control the robot to move along the preset trajectory. When the robot moves to each preset shooting point, it will send a shooting signal to the host computer through the local area network, pause briefly, and wait for the host computer to respond. After receiving the shooting signal, the host computer will immediately control the camera to take pictures and store the acquired image data in the specified folder. This process will be repeated until the robot moves to the last shooting point and completes the shooting. At this time, the robot control box will send an end signal to the host computer, and then automatically return to the shooting start position, waiting for the next acquisition task. Through this automated multi-view image acquisition method, multi-view image data of plants can be acquired efficiently and accurately, providing high-quality input data for subsequent three-dimensional reconstruction and organ segmentation tasks.

[0046] Furthermore, the automatic image acquisition system further comprises: a cross-shaped bracket. The plurality of cameras are fixed on the cross-shaped bracket via the camera bracket; the cross-shaped bracket is installed at the end of the mechanical arm 201 via the mounting hole. The structures of the cross-shaped bracket and the camera bracket are respectively as shown in FIG. Figure 4 and Figure 5 shown.

[0047] In terms of materials and processes, in order to improve the strength and rigidity of the system, metal materials or injection molding processes can be used to manufacture the camera bracket and the cross-shaped bracket. In addition, since metal materials have high processing costs and are heavy, they are not conducive to the movement and adjustment of the system. Although the injection molding process can reduce costs, the mold production cycle is long and it is not suitable for frequent design modifications. In contrast, 3D printing and laser cutting have the advantages of light weight, relative flexibility in design and debugging, and easy version iteration. Therefore, 3D printing and laser cutting can also be used to manufacture camera brackets and cross-shaped brackets.

[0048] Furthermore, the automated image acquisition system further includes: a hub 204; the signal lines of the multiple cameras are connected to the host computer 205 via the hub 204, and the host computer 205 controls the cameras and receives the images captured by the cameras. In addition, the robotic arm 201 is connected to the control box 203 via a control line, and the control box 203 sends a signal to control the movement of the robotic arm 201. The movement trajectory of the robotic arm 201 is shown in FIG. Figure 6 shown.

[0049] Step 102: Perform two-dimensional image segmentation on the multi-view image to obtain pixel semantic labels and construct a Tiff image. The Tiff image includes four channels, namely the RGB color components of the original image and the pixel semantic labels; the semantic labels are used to represent plant organs.

[0050] The purpose of 2D image segmentation of multi-view images is to distinguish different organs of plants, such as stems and leaves, while removing background pixels. Common 2D image segmentation methods include Mask-RCNN and YOLOv8. After 2D image segmentation, each pixel will get a semantic label to represent the plant organs and background, such as stem, leaf, and background. Then, the segmented image is fused with the original RGB image to construct a four-channel Tiff image, where the first three channels are the RGB color components of the image and the fourth channel is the semantic label of the pixel.

[0051] Step 103: using the Tiff image to train and optimize the multi-channel plant segmentation neural radiation field to obtain an optimized neural radiation field.

[0052] Preferably, the multi-channel plant segmentation neural radiation field uses multi-resolution hash coding for position coding, uses spherical harmonics for line of sight coding, and predicts volume density, color and semantics through a multi-layer perceptron combination and performs volume rendering.

[0053] The preprocessed Tiff image is input into the multi-channel plant segmentation neural radiation field (F Θ ) for training and optimization. Θ Multi-resolution hash coding is used to encode the position information, and spherical harmonics are used to represent the direction information. Through the network architecture of multi-layer perceptron (MLP), F Θ The volume density (σ), color (c), and semantics (l) corresponding to each 3D position can be learned.

[0054] Specifically, the Tiff image is input into a multi-channel plant segmentation neural radiation field to predict volume density, color and semantics and perform volume rendering; the light of volume rendering is sampled using a coarse-to-precise sampling method, and a loss function is constructed based on the color prediction value of the coarse stage, the color prediction value of the fine stage and the corresponding true value, as well as the semantic prediction value of the coarse stage, the semantic prediction value of the fine stage and the corresponding true value; based on the loss function, the multi-channel plant segmentation neural radiation field is trained and optimized using a back propagation algorithm and a gradient descent optimization strategy to obtain an optimized neural radiation field.

[0055] The following is a detailed introduction from five aspects: position encoding based on multi-resolution hash coding, line of sight encoding based on spherical harmonics, multi-layer perceptron combination, volume rendering and loss function.

[0056] 1. Position encoding based on multi-resolution hash coding:

[0057] The role of position (i.e. positionx) encoding is to ensure that the network has the ability to learn high- and low-frequency signals at the same time.

[0058] Disadvantages of using MLP for position encoding: the network needs to adaptively select certain dimensions of data in the position encoding to generate volume density and color values, which leads to a large network size and low computational efficiency.

[0059] The steps to use multi-resolution hash encoding are as follows:

[0060] ① Divide the space into multi-resolution grids.

[0061] ② Assign a feature code (position code) to each vertex, which is obtained through training.

[0062] ③ Given an arbitrary point X(x,y,z), find the grid it falls into and obtain its feature code through linear interpolation of the feature codes of the grid vertices.

[0063] ④ Combine feature codes of multiple resolutions and send them to MLP to predict volume density and color values.

[0064] The number of multi-resolution layers used in this application is 16 layers. The position code of each layer is calculated. The position code of each layer is two-dimensional. The 16 layers are spliced ​​together, so the output is a 32-dimensional position code.

[0065] 2. Sight coding based on spherical harmonics:

[0066] The functions of line of sight (viewing angle, direction, i.e. direction d) encoding are as follows:

[0067] ① Represents the changes in the appearance of objects under different viewing angles. In the real world, objects will present different appearances under different viewing angles, such as highlights, reflections, etc. Spherical harmonics can effectively model and represent this appearance that changes with viewing angle. Through line of sight encoding, the model can learn and reconstruct this view-dependent appearance change.

[0068] ② Improve the model's expressiveness and generalization. Through line-of-sight encoding, the model can better process and generate images from different perspectives. Even if certain perspectives have not been seen during training, the model can use the learned perspective-related features to reasonably infer and render the appearance from new perspectives. This improves the model's generalization ability.

[0069] The advantages of using spherical harmonics for line of sight encoding are as follows:

[0070] ① Providing prior knowledge related to the viewing angle. Encoding the viewing direction as spherical harmonic coefficients can provide the neural radiation field model with prior knowledge about the rendering process. Spherical harmonics are a set of orthogonal basis functions on a sphere that can effectively represent the spatial distribution characteristics of illumination and reflection. By using line of sight encoding, the model can better understand and utilize this prior knowledge.

[0071] ② Accelerate the rendering and reasoning process to a certain extent. Mapping the viewing direction into low-dimensional spherical harmonic function coefficients can reduce the input dimension of the model, thereby accelerating the rendering and reasoning process.

[0072] The input of the spherical harmonic function is the polar coordinates θ and φ of the line of sight, and the output is the encoded representation of θ and φ, which is expressed as follows:

[0073]

[0074] Where: θ is the pitch angle (the angle measured from the positive z-axis direction), ranging from (0, π); φ is the yaw angle (the angle measured from the positive x-axis direction), ranging from (0, 2π); is the jth and mth order associated Legendre function, expressed as:

[0075]

[0076]

[0077] Where: is the normalization coefficient, Where j is the order and m is the degree. Both m and j are integers and satisfy -j≤m≤j. For example, when j=1, m can be -1, 0, or 1. The higher the order j, the more complex the illumination model that can be expressed, and the better it can reflect the real physical situation, but the time cost of training will be higher.

[0078] The order j selected in this application is equal to 3. Therefore, the dimension of the code is 16 (1+3+5+7).

[0079] 3. Multi-layer perceptron combination:

[0080] like Figure 7As shown in the figure, the multi-layer perceptron combination includes: Linear, ReLU and Sigmoid structures. Among them, Linear represents a linear fully connected layer; ReLU is the abbreviation of "Rectified Linear Unit", which means rectified linear unit or rectified linear unit, and is a commonly used neural network activation function. The mathematical definition of the ReLU function is: ReLU(x') = max(0,x'). The Sigmoid function is also a common activation function, which is often used in the output layer of a neural network, especially for binary classification problems. The Sigmoid function can map real numbers to the interval (0, 1), so that it can be interpreted as a probability. The mathematical definition of the Sigmoid function is: sigmoid(x') = 1 / (1+exp(-x')).

[0081] 4. Volume Rendering

[0082] Volume rendering refers to the process of projecting all points on a line of sight onto an image to form pixels. The expression is as follows:

[0083]

[0084]

[0085] Where: C(r) is the color information, L(r) is the semantic information, r(t) is a point in three-dimensional space, d is the line of sight direction; t n and t f are the near and far boundaries of the three-dimensional scene respectively; c(r(t), d) is the color value of the three-dimensional point r(t) seen from the direction of d; l(r(t), d) is the semantic value of the three-dimensional point r(t) seen from the direction of d; σ(r()) is the volume density function, which reflects the ability of the physical material to absorb light; T(t) is the value of the ray from t n to f The accumulated transmittance.

[0086] The purpose of volume rendering is to project the color and semantics of points in space onto the image plane. The difference with the true color value and semantic true value of the pixel on the image can be calculated and squared to construct the difference with the true value, which is the loss function used.

[0087] In actual operation, because computing resources are limited and it is impossible to sample all continuous points on the light, a sampling method from coarse to precise is adopted.

[0088] Coarse stage: uniformly sample a certain number of points along each ray. These sampled points are input into the coarse NeRF network, which is responsible for predicting the density, color, and semantics of each point. According to the predicted density value, the pixel value of each ray is accumulated by volume rendering. The purpose of the coarse stage is to quickly obtain a coarse representation of the scene and provide prior information for the next stage.

[0089] Fine stage: Based on the density distribution predicted in the coarse stage, more dense sampling is performed near the denser areas. This sampling method is called importance sampling. These resampled points are merged with the points in the coarse stage and input into the fine NeRF network for more accurate prediction. Through volume rendering, the final pixel value of each ray is accumulated to obtain a higher quality rendering result.

[0090] 5. Loss function:

[0091] The loss function is the sum of the square error between the actual color and the rendered color result and the square error between the actual semantics and the rendered semantic result. The loss function uses color to estimate the volume density and uses a sampling method from coarse to precise. Therefore, there are two loss terms, coarse and fine. The specific expression is:

[0092]

[0093] Where: Represents the color prediction value of the coarse stage, Indicates the color prediction value of the fine stage (fine); represents the semantic prediction value at the rough stage, Represents the semantic prediction value of the fine stage; r represents a sampling ray, R represents all sampling rays; color C is a three-dimensional vector with units of (r, g, b); semantic L is a one-dimensional vector with units of l; represents the sum of squares of a vector term by term, for example, X = (r x ,g x ,b x ) T ,but

[0094] During the training process, volume rendering technology is used to transform F ΘThe predicted color and semantic information are compared with the corresponding values ​​of the original sampling points, and a loss function is constructed to optimize the model parameters. The loss function includes two parts: color loss and semantic segmentation loss, which are used to optimize the accuracy of the color and semantic labels predicted by the model. Through the back propagation algorithm and gradient descent optimization strategy, F is continuously iterated and updated. Θ The parameters of are optimized so that it can fit the input plant image well and implicitly establish a continuous neural radiation field with semantic information, that is, the optimized neural radiation field.

[0095] Step 104: extracting a semantic point cloud model of the plant from the optimized neural radiation field using an isosurface extraction algorithm. Each pixel in the semantic point cloud model includes geometric coordinate information, color attribute information, and semantic label information.

[0096] Isosurface extraction refers to a method of extracting a surface with a specific constant value from a scalar field. The scalar field here can be any scalar function in three-dimensional space, such as occupancy probability, signed distance function, etc. Given an extraction threshold, the isosurface extraction algorithm can generate a surface along the threshold.

[0097] The isosurface extraction algorithm used in this application is the Marching Cubes algorithm, which is an isosurface extraction algorithm based on a regular voxel grid. It divides the space into cubic units, compares the size relationship between the 8 vertex scalar values ​​and the threshold, uses a lookup table to obtain the local patch topology, and then performs linear interpolation to obtain the specific vertex positions of the triangular patches, and finally splices the local patches of each cubic unit to obtain a complete surface.

[0098] Taking multi-view images and camera poses as input, the neural radiation field network model is obtained after training and optimization. For a single input view, the model can generate the predicted image under this view. After sufficient training, the isosurface extraction algorithm is used to extract a high-quality semantic point cloud model from the optimized neural radiation field to obtain a plant point cloud with semantic information. Each extracted point contains complete geometric coordinates (X, Y, Z), color attributes (R, G, B) and semantic label (L) information.

[0099] The above steps can achieve end-to-end reconstruction and segmentation from multi-view images to semantic point cloud models, providing high-quality 3D model data for plant phenotyping and digital agriculture applications. Specific uses include:

[0100] 1. Analyze the phenotypic parameters of the stem, including its growth, branching, thickness, etc.

[0101] 2. Calculate leaf area index, analyze leaf distribution, etc.

[0102] 3. More accurate estimates of crop biomass.

[0103] 4. Analyze the growth of crop yield organs, evaluate the quality of yield organs, and predict the overall crop yield.

[0104] 5. In the field of plant science research, it can be used for high-throughput phenotypic screening to assist crop breeding and gene function research. In modern agricultural production, it can be integrated into plant factories and smart greenhouses to achieve real-time monitoring and precise management of plant growth status. In addition, the multi-perspective data acquisition and three-dimensional reconstruction method used in this application also has important reference significance for other fields (such as industrial detection, cultural relics digitization, etc.).

[0105] 6. Through industry-university-research cooperation and technology transfer, this application is expected to achieve industrial application and promote the transformation and upgrading of the agricultural equipment manufacturing industry. At the same time, the automated image acquisition system developed in this application will help improve the level of intelligence in agricultural production and contribute to the realization of agricultural modernization and sustainable development. In the long run, the promotion and application of this application will drive agricultural scientific and technological innovation, increase farmers' income and comprehensive agricultural benefits, and provide strong support for the high-quality development of agriculture.

[0106] Compared with the prior art, this application has the following advantages:

[0107] 1. Efficient data acquisition: By designing a modular automated image acquisition system and combining it with robotic arm automated control technology, it is possible to quickly acquire multi-view images of plants. Compared with manually adjusting the camera position and using a robotic arm to drive a single camera, this method greatly improves data acquisition efficiency.

[0108] 2. Stronger adaptability: Although the fixed multi-camera array has a simple structure, it has poor flexibility and cannot be adjusted according to the size and shape of the plant. It has a limited scope of application and requires many cameras, which is costly. Although the rotating platform can avoid camera movement errors, the plant will shake when rotating, which limits its modeling accuracy and acquisition speed. The robotic arm control method used in this application can freely adjust the shooting position and angle according to the characteristics of the plant, and is more adaptable.

[0109] 3. High-quality 3D reconstruction and organ segmentation: This application adopts a plant 3D reconstruction method based on neural radiation field. By introducing multi-resolution hash coding and spherical harmonic functions, the complex geometric structure and texture details of plants can be quickly fitted. At the same time, by constructing a loss function that includes color loss and newly added semantic loss, a point cloud model with precise semantic information can be obtained while ensuring reconstruction accuracy.

[0110] 4. Stronger robustness and generalization ability: Traditional multi-view stereo vision methods, such as active laser scanning and structured light methods and passive multi-view stereo matching, have obvious limitations. Laser scanning and structured light instruments are expensive, take a long time to collect data, are sensitive to surface materials and ambient lighting, are easily affected by plant surface reflections and shadows, and cannot directly reconstruct semantic point clouds, and still face the challenge of point cloud segmentation. In addition, when reconstructing plant point clouds, multi-view stereo matching will have obvious noise and many holes, and cannot directly generate high-quality plant semantic point clouds from images. In contrast, the neural radiation field-based method provided in this application can better handle the complexity and diversity of plants, learn implicit geometric and appearance representations, and has stronger robustness and generalization ability.

[0111] In an exemplary embodiment, the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0112] In an exemplary embodiment, the present application further provides a computer-readable storage medium storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0113] In an exemplary embodiment, the present application further provides a computer program product, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant legal provisions.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0116] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0117] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field, characterized in that: include: Using multiple cameras to synchronously collect multi-view images of plants; the multi-view images include original images of different viewpoints taken at multiple different points; Performing two-dimensional image segmentation on the multi-view image to obtain semantic labels of pixels and constructing a Tiff image; the Tiff image includes four channels, namely, RGB color components of the original image and semantic labels of pixels; the semantic labels are used to represent plant organs; The Tiff image is used to train and optimize the multi-channel plant segmentation neural radiation field to obtain an optimized neural radiation field; An isosurface extraction algorithm is used to extract a semantic point cloud model of the plant from the optimized neural radiation field; each pixel point in the semantic point cloud model includes geometric coordinate information, color attribute information and semantic label information.

2. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 1, characterized in that: Multi-view images of plants are collected synchronously using multiple cameras, including: Multiple cameras in an automated image acquisition system are used to synchronously acquire multi-viewing angle images of plants; the automated image acquisition system comprises: a host computer, a robotic arm, a control box and multiple cameras; the multiple cameras are mounted at the end of the robotic arm; the robotic arm is connected to the control box, the multiple cameras are connected to the host computer, and the control box and the host computer are connected to the same local area network; the host computer stores a camera control program for controlling the multiple cameras to take pictures; the control box stores a robotic arm control program for controlling the robotic arm to move along a preset trajectory.

3. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 1, characterized in that: The multi-channel plant segmentation neural radiation field adopts multi-resolution hash coding for position coding, adopts spherical harmonic function for line of sight coding, and predicts volume density, color and semantics through multi-layer perceptron combination and performs volume rendering.

4. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 3 is characterized in that: The Tiff image is used to train and optimize the multi-channel plant segmentation neural radiation field to obtain an optimized neural radiation field, specifically including: Input the Tiff image into a multi-channel plant segmentation neural radiation field, predict volume density, color and semantics, and perform volume rendering; The light of volume rendering is sampled by a sampling method from coarse to precise, and a loss function is constructed according to the color prediction value of the coarse stage, the color prediction value of the fine stage and the corresponding true value, as well as the semantic prediction value of the coarse stage, the semantic prediction value of the fine stage and the corresponding true value; Based on the loss function, the multi-channel plant segmentation neural radiation field is trained and optimized using a back propagation algorithm and a gradient descent optimization strategy to obtain an optimized neural radiation field.

5. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 1, characterized in that: The isosurface extraction algorithm is the Marching Cubes algorithm.

6. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 2, characterized in that: The automated image acquisition system further includes: a cross-shaped bracket; the plurality of cameras are fixed on the cross-shaped bracket via the camera bracket; and the cross-shaped bracket is installed at the end of the robotic arm via a mounting hole.

7. The method for plant 3D reconstruction and organ segmentation based on multi-camera synchronous acquisition and neural radiation field according to claim 2, characterized in that: The automated image acquisition system further includes: a hub; signal lines of the plurality of cameras are connected to the host computer via the hub.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the plant three-dimensional reconstruction and organ segmentation method based on multi-camera synchronous acquisition and neural radiation field described in any one of claims 1 to 7 is implemented.

Images