Multi-view fused plant three-dimensional reconstruction method, device and equipment and storage medium

Abstract

The invention relates to the field of image processing, and provides a multi-view fused plant three-dimensional reconstruction method, device and equipment and a storage medium, and the method comprises the steps: obtaining the three-dimensional priori knowledge of a plant based on a three-dimensional representation network; the three-dimensional representation network is obtained based on unlabeled plant multi-view image training; performing feature extraction on the multi-view time sequence image of the plant to be monitored to obtain multi-view image fusion features; and based on the plant three-dimensional priori knowledge and the multi-view image fusion features, performing three-dimensional reconstruction on the to-be-monitored plant to obtain a three-dimensional reconstruction model of the to-be-monitored plant. Three-dimensional priori knowledge is obtained through a three-dimensional representation network obtained through unmarked plant multi-view image training, multi-view image features are fused with the assistance of an attention mechanism, and finally three-dimensional reconstruction is performed on a to-be-monitored plant based on the three-dimensional priori knowledge and the multi-view image fusion features. And the three-dimensional reconstruction model of the plant to be monitored can be accurately and quickly obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a multi-view fusion three-dimensional reconstruction method, device, equipment and storage medium for plants. Background Art

[0002] Plant 3D reconstruction is mainly used for plant phenotyping. However, in greenhouse crop production, due to the long growth cycle of plants, luxuriant plant branches, and complex and changeable greenhouse environment lighting, existing plant 3D reconstruction methods (such as laser scanning and image matching) are difficult to improve the efficiency and accuracy of plant 3D reconstruction. Summary of the invention

[0003] The present invention provides a multi-view fusion plant three-dimensional reconstruction method, device, equipment and storage medium, which are used to solve the defect that it is difficult to improve the efficiency and accuracy of plant three-dimensional reconstruction in existing plant three-dimensional reconstruction technology, and achieve improvement of plant three-dimensional reconstruction efficiency and accuracy.

[0004] The present invention provides a multi-view fusion plant three-dimensional reconstruction method, comprising the following steps: Obtaining three-dimensional prior knowledge of plants based on a three-dimensional representation network; the three-dimensional representation network is trained based on unlabeled multi-view images of plants; Extract features from multi-view time-series images of the monitored plants to obtain multi-view image fusion features; Based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

[0005] According to a multi-view fusion plant 3D reconstruction method provided by the present invention, the 3D prior knowledge of the plant is obtained based on a 3D representation network, which includes: Based on the unlabeled plant multi-view images, determining the image global reconstruction error and the image contrast loss; Determining a self-supervised learning objective based on the image global reconstruction error and the image contrast loss; A three-dimensional representation network is trained based on the self-supervised learning objective.

[0006] According to a multi-view fusion plant 3D reconstruction method provided by the present invention, the feature extraction of multi-view time-series images of the monitored plant to obtain multi-view image fusion features includes: Perform feature extraction on multi-view time-series images of the monitored plants to obtain image visual features; splicing feature maps of multiple perspectives to obtain a multi-view feature tensor; the feature maps are determined based on the visual features of the image; The multi-view feature tensors are fused to obtain multi-view image fusion features.

[0007] According to a multi-view fusion plant 3D reconstruction method provided by the present invention, the 3D reconstruction of the plant to be monitored is performed based on the 3D prior knowledge of the plant and the multi-view image fusion features to obtain a 3D reconstruction model of the plant to be monitored, which includes: Dividing the continuous three-dimensional space corresponding to the plant to be monitored to obtain a regular grid composed of a plurality of voxels; determining occupied voxels in the regular grid; Extracting coordinates of the occupied voxels to obtain voxelized three-dimensional information of the plant; Based on the three-dimensional prior knowledge of the plant, the multi-view image fusion features and the voxelized three-dimensional information of the plant, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

[0008] According to a multi-view fusion plant 3D reconstruction method provided by the present invention, based on the plant 3D prior knowledge and the multi-view image fusion features, the plant to be monitored is 3D reconstructed to obtain a 3D reconstructed model of the plant to be monitored, and then the method includes: A time series characteristic field is added to the first neural radiation field to obtain a second neural radiation field; the first neural radiation field is used to perform three-dimensional reconstruction of the plant to be monitored; the second neural radiation field is used to perform N-dimensional reconstruction of the plant to be monitored, N>3; the time series characteristic field is determined based on the time series morphology of the plant to be monitored; Determining change information of the time series morphology in the time dimension; Based on the second neural radiation field and the change information, the plant to be monitored is reconstructed in N dimensions to obtain an N-dimensional reconstruction model of the plant to be monitored.

[0009] According to a multi-view fusion plant 3D reconstruction method provided by the present invention, based on the plant 3D priori knowledge and the multi-view image fusion features, the plant to be monitored is 3D reconstructed to obtain a 3D reconstructed model of the plant to be monitored, and then further comprising: Acquiring action information of an image acquisition device; the image acquisition device is used to acquire multi-viewing angle time-series images of the plant to be monitored; Acquiring posture information and a real three-dimensional model of the plant to be monitored; The image acquisition device is optimized based on the action information, the posture information and the real three-dimensional model.

[0010] The present invention also provides a multi-view fusion plant three-dimensional reconstruction device, comprising the following modules: A plant three-dimensional prior knowledge acquisition module is used to acquire plant three-dimensional prior knowledge based on a three-dimensional representation network; the three-dimensional representation network is obtained by training based on unlabeled plant multi-view images; A multi-view image fusion feature acquisition module is used to extract features from multi-view time-series images of the monitored plant to obtain multi-view image fusion features; The plant three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the plant to be monitored based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features to obtain a three-dimensional reconstruction model of the plant to be monitored.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the method for three-dimensional reconstruction of a plant by fusion of multiple views as described above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of a plant by fusion of multiple views as described in any one of the above is implemented.

[0013] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for three-dimensional reconstruction of plants by fusion of multiple views.

[0014] The multi-view fusion plant 3D reconstruction method, device, equipment and storage medium provided by the present invention introduce self-supervised learning into plant 3D reconstruction, obtain three-dimensional prior knowledge through a three-dimensional representation network obtained by training unlabeled multi-view images of plants, and then fuse the multi-view image features with the aid of an attention mechanism. Finally, based on the three-dimensional prior knowledge obtained by training and the fused multi-view image fusion features, three-dimensional reconstruction is performed on the plant to be monitored, and a three-dimensional reconstruction model of the plant to be monitored is obtained accurately and quickly, which improves the efficiency of plant three-dimensional reconstruction while also ensuring the accuracy of plant three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 This is one of the flow charts of the multi-view fusion plant three-dimensional reconstruction method provided by the present invention.

[0017] Figure 2This is the second flow chart of the multi-view fusion plant three-dimensional reconstruction method provided by the present invention.

[0018] Figure 3 It is a schematic structural diagram of a multi-view fusion plant three-dimensional reconstruction device provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Combine the following Figure 1-Figure 4 The invention describes a method, device, equipment and storage medium for three-dimensional reconstruction of a plant by fusion of multiple views.

[0022] Figure 1 FIG. 1 is one of the flow charts of the method for 3D reconstruction of plants by multi-view fusion provided by the present invention. Figure 1 As shown, the method includes the following: Step 100: obtaining three-dimensional prior knowledge of plants based on a three-dimensional representation network; the three-dimensional representation network is obtained by training based on unlabeled multi-view images of plants; The present invention is equipped with multiple cameras on a mobile monitoring trolley, and collects continuous videos of tomato plants at different viewing angles through the movement of the mechanical arm on the mobile monitoring trolley. The three-dimensional representation network is pre-trained on a large number of unlabeled continuous views of tomato plants through self-supervised learning to learn and obtain prior knowledge of the three-dimensional scene.

[0023] Step 200, extracting features from the multi-view time-series images of the plants to be monitored to obtain multi-view image fusion features; The multi-view time-series images of tomato plants are acquired through a mobile monitoring vehicle, and the acquired multi-view video sequences (i.e., the multi-view time-series images of the plants to be monitored) are input into the attention-based multi-view feature aggregation module (AMFAM), and image features at different view angles are adaptively weighted and fused. The present invention designs a multi-view feature aggregation module based on the attention mechanism, which is used to adaptively weight and fuse the image features of tomato plants at different view angles to obtain multi-view image fusion features.

[0024] Step 300: Based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

[0025] This paper designs a NeRF-guided Rapid Modeling Algorithm that uses Neural Radiance Fields (NeRF) as a priori guidance. Under the guidance of NeRF, the three-dimensional structure of the tomato plant is quickly restored through Markov random field optimization, which reduces the computational overhead while ensuring the accuracy of the three-dimensional structure.

[0026] This embodiment introduces self-supervised learning into plant 3D reconstruction. The 3D representation network is obtained through training of unlabeled plant multi-view images to obtain 3D prior knowledge, and then the multi-view image features are fused with the help of the attention mechanism. Finally, based on the 3D prior knowledge obtained through training and the fused multi-view image fusion features, the monitored plant is reconstructed in 3D, and the 3D reconstruction model of the monitored plant is obtained accurately and quickly, which improves the efficiency of plant 3D reconstruction while ensuring the accuracy of plant 3D reconstruction.

[0027] Figure 2 FIG. 2 is a flow chart of the method for 3D reconstruction of a plant by fusion of multiple views provided by the present invention. Figure 2 As shown, the method may also include: Step 10: determining the image global reconstruction error and the image contrast loss based on the unlabeled plant multi-view images; Step 20: determining a self-supervised learning target based on the image global reconstruction error and the image contrast loss; Step 30: Train a three-dimensional representation network based on the self-supervised learning objective.

[0028] Specifically, the video (continuous time image) of the tomato plant at different viewing angles is collected by a mobile monitoring vehicle to determine the first The position coordinates of the cameras are , the viewing direction is ,in, Indicates the horizontal direction angle; represents the vertical angle. By precisely controlling the motion of the mobile monitoring car, the continuous acquisition of multi-view views of the tomato plant is achieved. Assume that the robotic arm of the mobile monitoring car is The position at time is , facing , then in At this moment, the first The position of the camera It can be expressed as shown in formula 1. Among them, For the robot arm The rotation matrix at time.

[0029] ; (1) In order to obtain a 360-degree and high-quality time-series view of the tomato plant, the robot arm’s motion needs to meet the following conditions: Horizontal angle coverage of each viewing angle Panorama, that is ; The angle between adjacent viewing angles is not less than , generally take ,Right now ;in, For the first The pose of each camera; The robotic arm maintains a certain safe distance from the plant , Generally take ,Right now ,in is the approximate center coordinate of the plant; Movement speed of the robot arm satisfy , where the maximum speed Generally take .

[0030] Under the above constraints, the movement of the robot arm can be planned through a gradient-based trajectory optimization algorithm to maximize the viewing angle coverage and interval uniformity of the captured views while avoiding collisions.

[0031] For the unlabeled video data (i.e., unlabeled multi-view images of plants) collected by the mobile monitoring car, the present invention pre-trains a three-dimensional representation network on the unlabeled multi-view images of plants through self-supervised learning. The training set of unlabeled multi-view images of plants is determined as , each video clip Include Frame image, that is One of the goals of self-supervised pre-training is to minimize the global reconstruction error of the image. , as shown in Formula 2, where Represent the network in three dimensions; Reconstruct the network for the image; It means finding the minimum reconstruction error in the time neighborhood; Indicates Video clips at time images; Indicates Video clips at time The 3D shape and appearance of a tomato plant are learned through point cloud representation using the structures of PointNet++ (a deep learning model for point cloud segmentation and classification) and FoldingNet (an autoencoder for point clouds). is the radius of the time neighborhood, usually The concept of temporal neighborhood is introduced to enable the training process to model the spatiotemporal correlations between consecutive video frames.

[0032] ; (2) ; (3) In addition to the reconstruction error, the present invention also adds other self-supervised evaluation indicators (such as contrast loss between images ), as shown in Formula 3, where For images Three-dimensional feature representation of For images Three-dimensional feature representation of Representation and Different moments; is the cosine similarity function; is the temperature hyperparameter. Image contrast loss encourages feature representations from different moments of the same video to be closer in semantic space, while feature representations from different videos are further away.

[0033] ; (4) Combining the reconstruction error and contrast loss, we get the overall self-supervised learning objective , as shown in Formula 4, where and is the weight coefficient. On the dataset of unlabeled multi-view plant images, the 3D representation network After pre-training, the prior knowledge of the three-dimensional structure of the tomato plant can be obtained to guide the subsequent multi-view fusion modeling.

[0034] In this embodiment, multi-view images of tomato plants are continuously collected through motion control of a mobile monitoring vehicle and self-supervised learning to pre-train a three-dimensional representation network and obtain three-dimensional prior knowledge.

[0035] In one embodiment, the multi-view fusion plant 3D reconstruction method provided in the embodiment of the present application may further include: Step 210, extracting features from the multi-view time-series images of the plants to be monitored to obtain image visual features; Step 220: splicing the feature maps of multiple perspectives to obtain a multi-view feature tensor; the feature maps are determined based on the visual features of the image; Step 230: Fuse the multi-view feature tensors to obtain multi-view image fusion features.

[0036] Specifically, determine Video sequences collected from perspectives , where each video sequence Include Frame Image First, use pre-trained convolutional neural networks (such as ResNet and VGG, etc.) to extract visual features from each frame of image. Then, the feature maps of different perspectives at the same time are spliced ​​in the perspective dimension to obtain the multi-view feature tensor .

[0037] Then, the multi-view feature tensor Input to the attention aggregation module for fusion. The attention aggregation module consists of Attention layers are connected in series, and each attention layer contains three parallel fully connected layers (FC), which are used to calculate the three elements of the attention mechanism: query , key and value ,in, is the layer index.

[0038] In the Attention layers, the input features are first compressed by spatial average pooling (AvgPool) to , and then, are input into three fully connected layers in parallel to generate the query matrix, key matrix and value matrix, as shown in Formula 5, where , and They are the latent space dimensions of query, key and value respectively, usually , is the number of attention heads; Indicates The layer is used to generate the fully connected layer of the query matrix; Indicates The layer is used to generate a fully connected layer of the key matrix; Indicates The layer is used to generate a fully connected layer of value matrix; express OK The real matrix space of columns; express OK The real matrix space of columns; express OK A real matrix space of columns.

[0039] ; (5) Then, the attention weight matrix is ​​calculated by scaled dot-product attention (an attention mechanism) , as shown in Formula 6, where Indicates Perspective The attention weight of each view; Represents the key matrix The transpose of express -dimensional real matrix.

[0040] ; (6) Finally, the value matrix is ​​weighted and summed through the attention matrix to obtain the output feature , ,go through After iterative updates of layer attention, the output Then pass a fully connected layer to obtain the compact feature representation after aggregation , for use by the subsequent 3D reconstruction network.

[0041] In order to further improve the accuracy of 3D reconstruction of tomato plants, in addition to the commonly used Chamfer distance (a measure of the difference or similarity between two point sets or shapes), the present invention also introduces normal vector consistency loss and semantic segmentation consistency loss in the design of the loss function. The definition is shown in Formula 7, where For point The predicted normal vector of is the corresponding true normal vector; represents the vector inner product; Indicates time The predicted point cloud set; Indicates the size of the point cloud collection.

[0042] ; (7) ; (8) Semantic Segmentation Consistency Loss The definition is shown in Formula 8, where For point The predicted semantic labels (such as plant leaves, stems and fruits, etc.), is the corresponding true semantic label, For probability prediction.

[0043] ; (9) Finally, the present invention adopts a weighted combination of multi-task loss functions, as shown in Formula 9, where: is a weighted combination of multi-task losses; is the Chamfer distance loss; 、 、 is the weight coefficient, which can be tuned through cross-validation.

[0044] This embodiment fuses image features at different viewing angles through adaptive weighting.

[0045] In one embodiment, the multi-view fusion plant 3D reconstruction method provided in the embodiment of the present application may further include: Step 310, dividing the continuous three-dimensional space corresponding to the plants to be monitored to obtain a regular grid; Step 320, determining occupied voxels in the regular grid; Step 330, extracting coordinates of the occupied voxels to obtain voxelized three-dimensional information of the plant; Step 340: Based on the three-dimensional prior knowledge of the plant, the multi-view image fusion features and the voxelized three-dimensional information of the plant, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

[0046] Specifically, NeRF is an emerging implicit neural representation that learns the geometry and appearance of arbitrarily complex scenes by continuously mapping 5D coordinates of 3D positions and 2D views. Given a set of calibrated multi-view images, NeRF reconstructs the 3D scene by optimizing the following continuum rendering equation (equation 10), where For rays The predicted color on ; is the cumulative transmittance; is the position on the ray; is the viewing direction; represents the far end point of the ray; Indicates the near end point of the ray. and Respectively indicate the location The density and color at are parameterized by two multilayer perceptrons (MLP).

[0047] ; (10) This paper proposes a Markov Random Field (MRF) fast optimization algorithm based on NeRF prior. To reduce the computational complexity, first, the continuous three-dimensional space of the plant to be monitored is divided into Voxels, forming a regular grid Each voxel The occupancy rate is expressed as a binary variable , the problem is transformed into solving the MRF distribution of voxel occupancy. Using the pre-trained NeRF network , which can be calculated for each voxel Density and color , as a priori to guide the MRF energy function The construction of is shown in formula 11, where and are the unary potential energy term and the binary potential energy term respectively; Representing voxels The predicted occupancy rate of Representation and Adjacent voxels; represents the occupancy rate of adjacent voxels; Representing voxels Color; Represents the neighborhood relationship set of a voxel.

[0048] ; (11) ; (12) In order to efficiently solve the problem of minimizing the MRF energy function, the present invention adopts a graph cut algorithm. First, the MRF is mapped into a flow network, where voxels and voxel neighbor relationships correspond to nodes and edges of the network, respectively, and the potential energy function value corresponds to the capacity of the edge. According to the maximum flow minimum cut theorem, finding the minimum cut of the network is equivalent to solving the optimal configuration of the MRF. , as shown in Formula 12.

[0049] The present invention uses the augmented path method Boykov-Kolmogorov algorithm (an algorithm for solving the maximum flow problem) to solve the maximum flow, and its time complexity is ,in, is the maximum flow value of the network, which is related to the scene complexity. After solving, the minimum cut divides the voxels into two sets: occupied and unoccupied, corresponding to the foreground and background areas of the 3D model, respectively.

[0050] Finally, the voxels occupied By extracting the vertex coordinates of the plant, the voxelized 3D structure of the plant can be obtained. Considering the discontinuity of the voxel surface, algorithms such as Marching Cubes can be used to further generate a triangular mesh model with denser vertices and smoother topology.

[0051] This embodiment combines NeRF implicit representation with MRF graph optimization, realizes the dual guidance of prior knowledge and data constraints, and achieves an effective balance between the accuracy and speed of 3D reconstruction.

[0052] In one embodiment, the multi-view fusion plant 3D reconstruction method provided in the embodiment of the present application may further include: Step 400, adding a time series characteristic field to the first neural radiation field to obtain a second neural radiation field; the first neural radiation field is used to perform three-dimensional reconstruction of the plant to be monitored; the second neural radiation field is used to perform N-dimensional reconstruction of the plant to be monitored, N>3; the time series characteristic field is determined based on the time series morphology of the plant to be monitored; Step 500: determining the change information of the time series form in the time dimension; Step 600: Perform N-dimensional reconstruction on the plant to be monitored based on the second neural radiation field and the change information to obtain an N-dimensional reconstruction model of the plant to be monitored.

[0053] Specifically, the present invention proposes a 4D spatiotemporal reconstruction scheme based on temporal feature fields (TFF) for the dynamic characteristics of the three-dimensional morphology changing with time in the growth modeling of tomato plants. The implicit temporal feature field constructed by the present invention can map the plant morphology at different growth stages into a compact low-dimensional manifold space, thereby realizing efficient encoding and decoding of the spatiotemporal evolution law of the plant.

[0054] Based on the original NeRF, this paper introduces an additional implicit time series feature field , which is used to model the continuous change of plant state in the time dimension. The characteristic is expressed as , and compare it with the spatial features splicing to form a temporally consistent NeRF input. The expanded spatiotemporal NeRF can be expressed as shown in formula 13, where is the spatial representation parameter of NeRF; is the parameter of the time series characteristic field; represents the expanded space-time NeRF; Indicates the viewing direction; Indicates time Color output; Indicates time The density output.

[0055] ; (13) This representation expands the original NeRF 5D input to 6D, giving NeRF the ability to depict time-varying scenes. Controls the NeRF space density field and radiation field The shape and distribution of the plant thus affect the rendered plant image at that moment.

[0056] In order to optimize the above-mentioned spatiotemporal NeRF, it is necessary to obtain the Collect multi-view images. For the Collected in the period A collection of images, For the Collected in the period The camera parameter set corresponding to the image.

[0057] The optimization goal of spatiotemporal consistency NeRF is to minimize the reconstruction error , as shown in formula 14, where To use Rendering Period The reconstruction result of the image is shown in formula 15, where Indicates The cumulative transmittance of each period; Indicates time Down Ray In the parameters The density at Indicates time Down Ray In the parameters The viewing direction is The color of the time.

[0058] ; (14) ; (15) To optimize implicit function parameters and implicit time series feature field , the present invention adopts a layered alternating strategy. Its pseudo code is as follows:

[0059] Each episode (a complete data processing cycle or a training / testing experience) consists of two stages: 1. For each period , fixed NeRF parameters , only optimize the feature representation of this period , minimize the reconstruction error within period This stage independently learns the optimal implicit code for each plant in each period.

[0060] 2. Fix the feature representation of all periods , jointly optimize the NeRF parameters of all epochs , minimizing the cumulative reconstruction error In this stage, a temporally and spatially consistent NeRF is optimized through feature guidance.

[0061] Through the above alternating learning, NeRF and implicit feature representations are adapted to each other in spatial and temporal dimensions, and finally a continuous and smooth 4D spatiotemporal implicit scene representation of the plant is obtained, which simultaneously fits multiple time series observation data.

[0062] In the inference phase, given any query time , its timing characteristics By inputting the optimized 4D NeRF, the multi-view image of the plant at that moment can be reconstructed through volume rendering. Furthermore, the NeRF output in the spatial dimension is regularized using the MRF optimization method based on the implicit field to obtain the query time High-fidelity voxel model of plants at all times . By sampling the time series uniformly / adaptively By repeating the single-moment reasoning process above, we can obtain the space-time voxel manifold that describes the growth process of the entire plant. In order to achieve continuous interpolation of this manifold, a 4D convolutional neural network or a graph neural network can be used to learn the spatiotemporal correlation between voxels at different times, and then generate plant growth animations at any time granularity.

[0063] The 4D plant reconstruction paradigm based on time-series feature fields proposed in this embodiment introduces implicit dynamic features to characterize the intrinsic laws of the continuous evolution of plants over time, and designs a hierarchical optimization strategy for spatiotemporal consistency, thereby achieving end-to-end reconstruction from multiple time-series images to high-fidelity 4D models.

[0064] In one embodiment, the multi-view fusion plant 3D reconstruction method provided in the embodiment of the present application may further include: Step 700: obtaining action information of an image acquisition device; the image acquisition device is used to obtain multi-view time-series images of the plant to be monitored; Step 800: obtaining posture information and a real three-dimensional model of the plant to be monitored; Step 900: Optimize the image acquisition device based on the action information, the posture information and the real three-dimensional model.

[0065] Specifically, the present invention adopts Markov decision process (MDP) to model the camera layout optimization task. The camera selection process is abstracted as a four-tuple .

[0066] State Space : Contains the current plant posture parameters (such as height , Projection area etc.), the view index set of the acquired images etc. The above factors together characterize the environmental state in which the decision is made.

[0067] Action Space :Depend on Each action consists of a selection combination of candidate camera perspectives. For one dimensional binary vector, where Indicates selection Perspective, It means not to select.

[0068] State transition probability : Description in status Take action After that, the environment moves to the next state In this problem, the next state depends only on the plant growth dynamics and the update of the collected view set, and has nothing to do with the action selection. Therefore, it degenerates into the state transfer function .

[0069] Reward Function : Metrics in state Take action The reward function designed by the present invention comprehensively considers the balance between reconstruction accuracy and cost, and its specific form is shown in Formula 16, where: For the Real 3D models of plants at all times, To use perspective combination The reconstructed model obtained. It is the intersection-over-union ratio between the two models, reflecting the similarity of the reconstruction. express The norm is the number of selected perspectives. It is a balance factor that controls the relative importance of reconstruction accuracy and cost.

[0070] ; (16) In the MDP framework, the goal of the agent is to find a policy function , so that the series of actions taken under the guidance of this strategy can maximize the expectation (goal) of the cumulative reward , as shown in Formula 17, where is the discount factor; Indicated in strategy expectations; The time step is Discount factor when Indicates in status Next Strategy The action given.

[0071] ; (17) Since both the state space and the action space are high-dimensional discrete spaces, the present invention adopts the policy gradient method in deep reinforcement learning, namely the Proximal Policy Optimization (PPO) algorithm to optimize and solve the above objectives.

[0072] The present invention constructs a parameterized strategy network As an agent (an agent that can perceive the environment, make decisions and take actions to achieve a specific goal, which can be software, hardware or a system), where is the network weight. The network is characterized by state As input, the output is the probability of taking actions from each perspective in this state. Considering the high-dimensional discreteness of the action space, the output layer of the policy network uses the Sigmoid function (activation function) to model the action probability, as shown in Formula 18, where For the Network parameters corresponding to each perspective; is the Sigmoid function; represents Bernoulli distribution; Indicates The selection action of a perspective; Indicates The inner product of the network parameters and state features from each perspective.

[0073] ; (18) To support gradient-based policy optimization, we use the Gumbel-Softmax reparameterization technique (a method to maintain gradient propagation when sampling discrete random variables in deep learning) to continuously relax discrete actions and transform them into differentiable continuous distribution samples.

[0074] The PPO algorithm achieves a good balance between training stability and sampling efficiency by limiting the policy update amplitude. Its core idea is to construct a surrogate objective (a kind of objective function used to optimize the policy in reinforcement learning) so that the current policy The previous strategy The difference is constrained as shown in Equation 19, where is the probability ratio, is the generalized advantage function, representing the action The advantage over the average action value is shown in Equation 20, where is the action value function, is the state value function, which represents the state Take action and the expected cumulative reward of random actions; Indicated in strategy Next pair status and actions expectations. The function compares the probability Crop within the scope to avoid policy updates being too large.

[0075] ; (19) ; (20) The PPO algorithm alternates between two steps of strategy evaluation and strategy improvement: 1. Strategy evaluation: fixed strategy parameters , using the Monte Carlo method to estimate the state value function and the action-value function , as shown in Formula 21 and Formula 22, where Indicates from time arrive The discount is accumulated; Indicates at time Status; Indicates at time Actions taken; represents the discount factor; Indicates the parameters gradient.

[0076] ;(twenty one) ;(twenty two) ;(twenty three) 2. Strategy Improvement: Estimation-Based , the PPO objective function is adjusted by stochastic gradient ascent algorithm conduct Step optimization, update the policy network parameters, as shown in formula 23, where is the learning rate. Constrained optimization reduces the risk of the strategy converging to a local optimum too early.

[0077] By continuously iterating the above two steps, we can finally get the approximately optimal camera layout combination strategy. In practical applications, given a growth process of a plant to be reconstructed, the agent observes the state at each moment. , using the learned policy network to infer the best view selection in real time , guiding the complete process of multi-view data acquisition and 4D reconstruction.

[0078] The camera layout optimization algorithm based on reinforcement learning proposed in this embodiment can autonomously learn the perspective selection strategy from a large amount of historical plant data, and adaptively adjust the strategy according to the current plant state, thereby maximizing the 4D reconstruction quality while saving hardware costs.

[0079] The following is a description of the plant 3D reconstruction device based on multi-view fusion provided by the present invention. The plant 3D reconstruction device based on multi-view fusion described below and the plant 3D reconstruction method based on multi-view fusion described above can be referenced to each other.

[0080] Please refer to Figure 3 The present invention also provides a multi-view fusion plant three-dimensional reconstruction device, comprising: The plant three-dimensional prior knowledge acquisition module 301 is used to acquire the plant three-dimensional prior knowledge based on a three-dimensional representation network; the three-dimensional representation network is obtained by training based on unlabeled plant multi-view images; The multi-view image fusion feature acquisition module 302 is used to extract features from the multi-view time-series images of the plants to be monitored to obtain multi-view image fusion features; The plant 3D reconstruction module 303 is used to perform 3D reconstruction on the plant to be monitored based on the 3D prior knowledge of the plant and the multi-view image fusion features to obtain a 3D reconstruction model of the plant to be monitored.

[0081] Optionally, the multi-view fusion plant 3D reconstruction device further includes: A first determination module is used to determine the image global reconstruction error and the image contrast loss based on the unlabeled plant multi-view image; A self-supervised learning target determination module, used to determine a self-supervised learning target based on the image global reconstruction error and the image contrast loss; The three-dimensional representation network acquisition module is used to obtain the three-dimensional representation network based on the self-supervised learning objective training.

[0082] Optionally, the multi-view image fusion feature acquisition module includes: A feature extraction unit, used for extracting features from multi-view time-series images of the monitored plant to obtain image visual features; A feature map splicing unit, used for splicing feature maps of multiple perspectives to obtain a multi-view feature tensor; the feature map is determined based on the visual features of the image; The feature fusion unit is used to fuse the multi-view feature tensors to obtain multi-view image fusion features.

[0083] Optionally, the plant 3D reconstruction module includes: A regular grid determination unit, used for dividing the continuous three-dimensional space corresponding to the plants to be monitored to obtain a regular grid; an occupied voxel determination unit, configured to determine occupied voxels in the regular grid; A coordinate extraction unit, used for extracting the coordinates of the occupied voxels to obtain voxelized three-dimensional information of the plant; The three-dimensional reconstruction model determination unit is used to perform three-dimensional reconstruction on the plant to be monitored based on the three-dimensional prior knowledge of the plant, the multi-view image fusion features and the voxelized three-dimensional information of the plant to obtain a three-dimensional reconstruction model of the plant to be monitored.

[0084] Optionally, the multi-view fusion plant 3D reconstruction device further includes: A three-dimensional reconstruction module, used for adding a time series characteristic field to the first neural radiation field to obtain a second neural radiation field; the first neural radiation field is used for three-dimensional reconstruction of the plant to be monitored; the second neural radiation field is used for N-dimensional reconstruction of the plant to be monitored, N>3; the time series characteristic field is determined based on the time series morphology of the plant to be monitored; A change information determination module, used to determine the change information of the time series form in the time dimension; An N-dimensional reconstruction module is used to perform N-dimensional reconstruction on the plant to be monitored based on the second neural radiation field and the change information to obtain an N-dimensional reconstruction model of the plant to be monitored.

[0085] Optionally, the multi-view fusion plant 3D reconstruction device further includes: A multi-view time-series image acquisition module, used to acquire the action information of an image acquisition device; the image acquisition device is used to acquire multi-view time-series images of the plant to be monitored; A first acquisition module, used to acquire the posture information and the real three-dimensional model of the plant to be monitored; An image acquisition device optimization module is used to optimize the image acquisition device based on the action information, the posture information and the real three-dimensional model.

[0086] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a multi-view fusion plant three-dimensional reconstruction method, the method comprising: obtaining three-dimensional prior knowledge of the plant based on a three-dimensional representation network; the three-dimensional representation network is obtained based on unlabeled multi-view images of the plant training; feature extraction is performed on the multi-view time series images of the plant to be monitored to obtain multi-view image fusion features; based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstruction model of the plant to be monitored.

[0087] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-view fusion plant three-dimensional reconstruction method provided by the above-mentioned methods, and the method includes: obtaining three-dimensional prior knowledge of the plant based on a three-dimensional representation network; the three-dimensional representation network is obtained based on unlabeled multi-view images of the plant training; feature extraction is performed on the multi-view time series images of the monitored plant to obtain multi-view image fusion features; based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstruction model of the plant to be monitored.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-view fusion plant three-dimensional reconstruction method provided by the above-mentioned methods, the method comprising: obtaining three-dimensional prior knowledge of the plant based on a three-dimensional representation network; the three-dimensional representation network is obtained based on training of unlabeled multi-view images of the plant; performing feature extraction on the multi-view time-series images of the monitored plant to obtain multi-view image fusion features; performing three-dimensional reconstruction of the plant to be monitored based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features to obtain a three-dimensional reconstruction model of the plant to be monitored.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for 3D reconstruction of plants by multi-view fusion, characterized in that: include: Obtaining 3D prior knowledge of plants based on 3D representation network; The three-dimensional representation network is trained based on unlabeled multi-view images of plants; Extract features from multi-view time-series images of the monitored plants to obtain multi-view image fusion features; Based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

2. The method for 3D reconstruction of plants by multi-view fusion according to claim 1, characterized in that: The method of obtaining three-dimensional prior knowledge of plants based on a three-dimensional representation network includes: Based on the unlabeled plant multi-view images, determining the image global reconstruction error and the image contrast loss; Determining a self-supervised learning objective based on the image global reconstruction error and the image contrast loss; A three-dimensional representation network is trained based on the self-supervised learning objective.

3. The method for 3D reconstruction of plants by multi-view fusion according to claim 1, characterized in that: The feature extraction of the multi-view time-series images of the monitored plant to obtain the multi-view image fusion features includes: Perform feature extraction on multi-view time-series images of the monitored plants to obtain image visual features; splicing feature maps of multiple perspectives to obtain a multi-view feature tensor; the feature maps are determined based on the visual features of the image; The multi-view feature tensors are fused to obtain multi-view image fusion features.

4. The method for 3D reconstruction of plants by multi-view fusion according to claim 1, characterized in that: The three-dimensional reconstruction of the plant to be monitored based on the three-dimensional prior knowledge of the plant and the multi-view image fusion feature to obtain the three-dimensional reconstruction model of the plant to be monitored includes: Dividing the continuous three-dimensional space corresponding to the plants to be monitored to obtain a regular grid; determining occupied voxels in the regular grid; Extracting coordinates of the occupied voxels to obtain voxelized three-dimensional information of the plant; Based on the three-dimensional prior knowledge of the plant, the multi-view image fusion features and the voxelized three-dimensional information of the plant, the plant to be monitored is three-dimensionally reconstructed to obtain a three-dimensional reconstructed model of the plant to be monitored.

5. The method for 3D reconstruction of plants by multi-view fusion according to claim 1, characterized in that: The three-dimensional reconstruction of the plant to be monitored is performed based on the three-dimensional prior knowledge of the plant and the multi-view image fusion feature to obtain a three-dimensional reconstruction model of the plant to be monitored, and then includes: A time series characteristic field is added to the first neural radiation field to obtain a second neural radiation field; the first neural radiation field is used to perform three-dimensional reconstruction of the plant to be monitored; the second neural radiation field is used to perform N-dimensional reconstruction of the plant to be monitored, N>3; the time series characteristic field is determined based on the time series morphology of the plant to be monitored; Determining change information of the time series morphology in the time dimension; Based on the second neural radiation field and the change information, the plant to be monitored is reconstructed in N dimensions to obtain an N-dimensional reconstruction model of the plant to be monitored.

6. The method for 3D reconstruction of plants by multi-view fusion according to claim 1, characterized in that: The three-dimensional reconstruction of the plant to be monitored is performed based on the three-dimensional prior knowledge of the plant and the multi-view image fusion feature to obtain a three-dimensional reconstruction model of the plant to be monitored, and then further includes: Acquiring action information of an image acquisition device; the image acquisition device is used to acquire multi-viewing angle time-series images of the plant to be monitored; Acquiring posture information and a real three-dimensional model of the plant to be monitored; The image acquisition device is optimized based on the action information, the posture information and the real three-dimensional model.

7. A plant 3D reconstruction device based on multi-view fusion, characterized in that: include: A plant three-dimensional prior knowledge acquisition module, used to acquire plant three-dimensional prior knowledge based on a three-dimensional representation network; The three-dimensional representation network is trained based on unlabeled multi-view images of plants; A multi-view image fusion feature acquisition module is used to extract features from multi-view time-series images of the monitored plant to obtain multi-view image fusion features; The plant three-dimensional reconstruction module is used to perform three-dimensional reconstruction on the plant to be monitored based on the three-dimensional prior knowledge of the plant and the multi-view image fusion features to obtain a three-dimensional reconstruction model of the plant to be monitored.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for three-dimensional reconstruction of a plant by fusion of multiple views as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for three-dimensional reconstruction of a plant by multi-view fusion as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for three-dimensional reconstruction of a plant by multi-view fusion as claimed in any one of claims 1 to 6 is implemented.

Images