Nerve implicit surface-based three-dimensional reconstruction method and related device

Abstract

The embodiment of the invention provides a three-dimensional reconstruction method based on a neural implicit surface and a related device. The method comprises the following steps: acquiring banana bunch images at different visual angles through a camera; performing feature extraction and feature matching on the banana bunch image to generate an MVS-Mask; a sparse point cloud model is adopted to generate an SDF coordinate system; inputting the banana bunch image and the camera pose into a reconstruction network, and learning and constructing a geometric SDF field and a color field of the banana bunch by the reconstruction network through a body rendering mode; a Marching Cubes algorithm is adopted to extract the implicit surface as a three-dimensional mesh model; and calculating the vertex color of the grid and mapping the vertex color to the three-dimensional mesh model to obtain a three-dimensional grid model with appearance information. Based on this, the embodiment of the invention distinguishes main body and background information in a complex orchard environment, overcomes detection background interference and repeated detection, and reduces the influence caused by environmental noise, thereby more accurately reconstructing the shape and details of the banana bunches.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method based on neural implicit surface and related devices. Background Art

[0002] Existing orchard vision systems are mostly targeted at small-volume fruits and vegetables, and there is less research on large-volume fruits. Some researchers have used 2D vision methods to understand the phenotyping of bananas under precision agriculture. Some researchers have used the YOLO network to detect banana bunches and stem parts in natural environments. There are also improved deep learning algorithms for multi-target recognition of banana fruits, inflorescence axes, and flower buds. In addition, there is a method that combines deep images with neural networks to calculate the center point of the banana stem, and then estimates the weight of the banana bunch by bunch size and effective pixel rate. However, these studies only start from the perspective of 2D vision, and the information obtained is limited, and it is impossible to provide richer and more detailed data for the phenotypic analysis of banana bunches.

[0003] A more three-dimensional phenotypic analysis of banana fruits can promote the construction of banana orchard agriculture with precise analysis, and a key step is to reconstruct the three-dimensional model of the green banana bunches that have not been picked in the banana orchard. Since the emergence of research on visual inspection of banana orchards, visual inspection of banana spikes, pseudostems, fruit axes and inflorescence axes has been gradually carried out, but there are relatively few studies on three-dimensional reconstruction of banana orchard fruits. Therefore, how to overcome background interference and repeated inspections and accurately reconstruct the shape and details of banana bunches has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a three-dimensional reconstruction method and related devices based on neural implicit surfaces, which can overcome detection background interference and repeated detection, reduce the impact of environmental noise, and thus more accurately reconstruct the shape and details of a banana bunch.

[0005] In a first aspect, an embodiment of the present invention provides a three-dimensional reconstruction method based on a neural implicit surface, comprising:

[0006] The banana bunch images at different viewing angles are acquired by using a camera;

[0007] Performing feature extraction and feature matching on the banana bunch image to generate an MVS-Mask, wherein the MVS-Mask is used to supervise the reconstruction network;

[0008] A sparse point cloud model is used to generate an SDF coordinate system, wherein the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate;

[0009] Inputting the banana bunch image and camera pose into the reconstruction network, the reconstruction network learns to construct the geometric SDF field and color field of the banana bunch by volume rendering;

[0010] The implicit surface is extracted into a three-dimensional mesh model using the Marching Cubes algorithm;

[0011] The mesh vertex colors are calculated and mapped to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

[0012] In some embodiments, the extracting and matching features of the banana bunch image to generate an MVS-Mask includes:

[0013] Performing feature extraction and feature matching on the banana bunch image to obtain geometric structure features and appearance color features;

[0014] Sparse reconstruction is performed based on geometric structure features and appearance color features to obtain a sparse point cloud model;

[0015] Generate dense point cloud by densely matching and deeply fusing sparse point cloud model files;

[0016] The dense point cloud is projected onto a two-dimensional plane using the decomposed camera internal and external parameter information to generate a depth mask for each projected image.

[0017] In some embodiments, constructing the geometric SDF field and color field of the banana bunch includes:

[0018] The geometric SDF field of banana bunch modeling based on SDF Network;

[0019] The color field of the banana bunch is modeled based on Color Network.

[0020] In some embodiments, the geometric SDF field of the banana bunch modeling based on the SDF Network includes:

[0021] The sampled spatial points are input as training points into the SDF Network using the ray marching method to calculate the SDF value and gradient;

[0022] Preprocessing the banana bunch image data to obtain camera parameters and SFM sparse feature points;

[0023] According to the SFM sparse feature point supervised SDF field modeling, the loss is calculated and the network parameters of the SDF Network are optimized to obtain the geometric SDF field, wherein the network parameters include SDF values ​​and gradients.

[0024] In some embodiments, projecting the dense point cloud onto a two-dimensional plane using the decomposed camera internal and external parameter information to generate a depth mask for each projected image includes:

[0025] Obtaining camera parameters of the camera;

[0026] Decompose the camera's projection matrix from the camera parameters;

[0027] Decomposing the projection matrix into a camera intrinsic parameter matrix and a camera extrinsic parameter matrix;

[0028] The dense point cloud is projected onto a two-dimensional plane according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and a depth mask of each projected image is generated.

[0029] In some embodiments, the method further comprises:

[0030] Obtain image projection pixel blocks at different viewing angles;

[0031] Establishing a relationship between the image projection pixel blocks at different viewing angles using a homography induced by a spatial plane;

[0032] When rendering the image projection pixel block, the pixel through which the light travels is taken as the center point, and the pixel block with a size of 10×10 is used as the rendering block for window matching;

[0033] Calculate the NCC coefficient of the same viewpoint of the projected image and the rendered image of the viewpoint under different viewing angles;

[0034] Normalizing the NCC coefficient, and using the NCC coefficient to perform deviation consistency loss;

[0035] The banana bunch images at different viewing angles are weighted averaged to obtain a weighted average result, wherein the weight of the weighted average is determined according to the confidence level and exposure level of the viewing angle.

[0036] In some embodiments, the method further comprises:

[0037] Generate a geometric SDF field by obtaining salient points from feature points in each view through the SDF network;

[0038] When a single image is rendered using volume integral, the ID mapping relationship of the single image is used to find the corresponding image feature points to supervise the geometric SDF field.

[0039] In a second aspect, an embodiment of the present invention further provides a three-dimensional reconstruction device based on a neural implicit surface, the device comprising:

[0040] An acquisition module is used to acquire banana bunch images at different viewing angles through a camera;

[0041] A first generation supervision module is used to perform feature extraction and feature matching on the banana bunch image to generate an MVS-Mask, and the MVS-Mask is used to supervise the reconstruction network;

[0042] A second generation supervision module is used to generate an SDF coordinate system using a sparse point cloud model, wherein the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate;

[0043] A reconstruction module, used for inputting the banana bunch image and camera pose into the reconstruction network, and the reconstruction network learns to construct a geometric SDF field and a color field of the banana bunch by volume rendering;

[0044] Extraction module, used to extract implicit surface into 3D mesh model using Marching Cubes algorithm;

[0045] The mapping module is used to calculate the colors of the mesh vertices and map them to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

[0046] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional reconstruction method based on neural implicit surface as described in the first aspect is implemented.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the three-dimensional reconstruction method based on neural implicit surface as described in the first aspect.

[0048] According to the three-dimensional reconstruction method based on neural implicit surface and related devices provided by the embodiment of the present invention, the three-dimensional reconstruction method based on neural implicit surface includes: obtaining banana bunch images under different viewing angles through a camera; performing feature extraction and feature matching on the banana bunch images to generate MVS-Mask, which is used to supervise the reconstruction network; using a sparse point cloud model to generate an SDF coordinate system, which is used to supervise the geometric SDF field, and the sparse point cloud model includes the mapping relationship between the image ID and the 3D point ID and the mapping relationship between the 3D point ID and the corresponding 3D coordinates; inputting the banana bunch image and the camera pose into the reconstruction network, and the reconstruction network learns to construct the geometric SDF field and color field of the banana bunch by volume rendering; using the Marching Cubes algorithm to extract the implicit surface into a three-dimensional mesh model; calculating the mesh vertex color and mapping it to the three-dimensional mesh model, and obtaining a three-dimensional mesh model with appearance information. The complex banana orchard environment brings more environmental noise, and the twisted and irregular geometric structure of the banana bunch brings greater depth changes. In order to enable the reconstruction network to better learn the sudden depth change of the banana bunch and reduce the deviation of the original volume rendering, the present invention introduces a supervision mechanism for SDF field modeling. By constructing the SDF field to describe the geometric structure of the banana bunch and constructing the color field to describe the appearance color of the banana bunch, more accurate geometric information and color information of the banana bunch can be provided in a complex orchard environment. Using MVS-Mask to establish high-quality mask information as a two-dimensional constraint, this can provide clear geometric information and color information of the banana bunch, and can pay more attention to the reconstruction of the banana bunch, which can not only reduce the interference of environmental noise, but also improve the problem of self-occlusion. The volume rendering integral weighted method is used to render the banana bunch images under different viewing angles, and the Marching Cubes algorithm is used to extract the implicit surface as a three-dimensional mesh model, calculate the mesh vertex color and map it to the three-dimensional mesh model, and finally obtain a high-quality banana bunch three-dimensional mesh model with texture color information. Based on this, the embodiment of the present invention can distinguish the main body and background information in a complex orchard environment, and can provide clear geometric information of the banana bunch. Let the reconstruction network pay more attention to the banana bunch, reduce the influence of environmental noise, and thus more accurately reconstruct the shape and details of the banana bunch. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1A is a flow chart of a three-dimensional reconstruction method based on neural implicit surface provided by an embodiment of the present invention;

[0050] Figure 1B This is a diagram of a reconstructed network framework provided by an embodiment of the present invention;

[0051] Figure 2is a structural diagram of an SDF Network and a Color Network provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of ray marching and sampling rendering provided by an embodiment of the present invention;

[0053] Figure 4 This is a diagram of view inconsistency caused by light projection and occlusion provided by an embodiment of the present invention;

[0054] Figure 5 is a viewpoint correspondence diagram under different viewing angles provided by an embodiment of the present invention;

[0055] Figure 6 is a comparison diagram between MVS-Mask and MiVOS provided by an embodiment of the present invention;

[0056] Figure 7 is a comparison diagram of the rendering results provided by an embodiment of the present invention and the original NeuS network;

[0057] Figure 8 is a comparison diagram of the rendering process provided by an embodiment of the present invention and the original NeuS;

[0058] Fig. 9 is a comparison diagram of a normal map of an original NeuS network provided by an embodiment of the present invention;

[0059] Fig.10 is a schematic diagram of a three-dimensional reconstruction device based on neural implicit surface provided by an embodiment of the present invention;

[0060] Fig.11 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the following drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0063] In the embodiments of the present invention, the words "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of the words "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.

[0064] First, several terms involved in the present invention are analyzed:

[0065] Structure from Motion (SfM): Structure from Motion restores the camera's posture and the three-dimensional position of sparse points in the scene through feature point matching and geometric constraints. The corresponding relationship of the same pixel points in the image is obtained through the matching algorithm, and then the three-dimensional coordinate information of the spatial point is obtained by using the matching constraint relationship and the triangulation principle, and then the three-dimensional model of the object is reconstructed.

[0066] Multi-view Stereo (MVS): Multi-view stereo vision calculates the depth information of the scene from the disparity information between pixels in image pairs obtained from different perspectives, infers the three-dimensional coordinates of each pixel by matching pixels between images from multiple perspectives, and then reconstructs a dense three-dimensional scene from images from multiple perspectives.

[0067] Neural Radiance Fields (NeRF): NeRF is a new view synthesis method that inputs coordinate point information, obtains the density attributes corresponding to the coordinate points, and uses MLP to perform implicit neural scene expression to learn the geometric density field of the 3D scene.

[0068] Volume rendering: When light passes through a bunch of particles in a straight line, if the radiation intensity of each light ray from the initial emission to the final hitting the imaging plane can be calculated, a projected image can be rendered, and volume rendering is used to model this process.

[0069] Neural implicit surfaces (NeuS): A method of using neural networks to represent 3D geometric shapes, which defines the position of the surface through a function. The neural network takes a point in 3D space as input and then outputs its attributes, which usually represent the distance from the point to the surface or a "field" value.

[0070] Signed Distance Function (SDF): SDF represents the minimum distance from a point to a surface. If the point is inside the surface, the distance is negative, if the point is outside the surface, the distance is positive, and if the point is on the surface, the distance is 0. The signed distance field establishes a spatial field in which each pixel or voxel records the distance between itself and the surrounding pixels or voxels.

[0071] Marching Cubes algorithm: is a voxel-level reconstruction method used to extract isosurfaces from 3D data fields. The core idea of ​​the algorithm is to approximate isosurfaces by linear interpolation. By traversing the entire model, using small cubes (called voxels) to fit the model, and constructing a triangulated mesh with a topological relationship based on the intersection of the voxels and the surface, the surface is approximated.

[0072] In order to more conveniently describe the working principle of the embodiment of the present invention later, an introduction to relevant technical scenarios is first given below.

[0073] Existing orchard vision systems are mostly targeted at small-volume fruits and vegetables, and there is less research on large-volume fruits. Some researchers have used 2D vision methods to understand the phenotyping of bananas under precision agriculture. Some researchers have used the YOLO network to detect banana bunches and stem parts in natural environments. There are also improved deep learning algorithms for multi-target recognition of banana fruits, inflorescence axes, and flower buds. In addition, there is a method that combines deep images with neural networks to calculate the center point of the banana stem, and then estimates the weight of the banana bunch by bunch size and effective pixel rate. However, these studies only start from the perspective of 2D vision, and the information obtained is limited, and it is impossible to provide richer and more detailed data for the phenotypic analysis of banana bunches.

[0074] A more three-dimensional phenotypic analysis of banana fruits can promote the construction of banana orchard agriculture with precise analysis, and a key step is to reconstruct the three-dimensional model of the green banana bunches that have not been picked in the banana orchard. Since the emergence of research on visual inspection of banana orchards, visual inspection of banana spikes, pseudostems, fruit axes and inflorescence axes has been gradually carried out, but there are relatively few studies on three-dimensional reconstruction of banana orchard fruits. Therefore, how to overcome background interference and repeated inspections and accurately reconstruct the shape and details of banana bunches has become a technical problem that needs to be solved urgently.

[0075] Based on this, the present invention provides a three-dimensional reconstruction method based on neural implicit surface and related devices. Among them, the three-dimensional reconstruction method based on neural implicit surface includes: obtaining banana bunch images under different viewing angles through a camera; performing feature extraction and feature matching on the banana bunch images to generate MVS-Mask, and MVS-Mask is used to supervise the reconstruction network; using a sparse point cloud model to generate an SDF coordinate system, and the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model contains the mapping relationship between the image ID and the 3D point ID and the mapping relationship between the 3D point ID and the corresponding 3D coordinates; inputting the banana bunch image and the camera pose into the reconstruction network, and the reconstruction network learns to construct the geometric SDF field and color field of the banana bunch by volume rendering; using the Marching Cubes algorithm to extract the implicit surface into a three-dimensional mesh model; calculating the mesh vertex color and mapping it to the three-dimensional mesh model, and obtaining a three-dimensional mesh model with appearance information. The complex banana orchard environment brings more environmental noise, and the twisted and irregular geometric structure of the banana bunch brings greater depth changes. In order to enable the reconstruction network to better learn the sudden depth change of the banana bunch and reduce the deviation of the original volume rendering, the present invention introduces a supervision mechanism for SDF field modeling. By constructing the SDF field to describe the geometric structure of the banana bunch and constructing the color field to describe the appearance color of the banana bunch, more accurate geometric information and color information of the banana bunch can be provided in a complex orchard environment. Using MVS-Mask to establish high-quality mask information as a two-dimensional constraint, this can provide clear geometric information and color information of the banana bunch, and can pay more attention to the reconstruction of the banana bunch, which can not only reduce the interference of environmental noise, but also improve the problem of self-occlusion. The volume rendering integral weighted method is used to render the banana bunch images under different viewing angles, and the Marching Cubes algorithm is used to extract the implicit surface as a three-dimensional mesh model, calculate the mesh vertex color and map it to the three-dimensional mesh model, and finally obtain a high-quality banana bunch three-dimensional mesh model with texture color information. Based on this, the embodiment of the present invention can distinguish the main body and background information in a complex orchard environment, and can provide clear geometric information of the banana bunch. Let the reconstruction network pay more attention to the banana bunch, reduce the influence of environmental noise, and thus more accurately reconstruct the shape and details of the banana bunch.

[0076] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0077] like Figure 1A As shown, Figure 1A It is a flowchart of a three-dimensional reconstruction method based on neural implicit surface provided by an embodiment of the present invention. The three-dimensional reconstruction method based on neural implicit surface may include but is not limited to steps S101 to S106.

[0078] Step S101, acquiring images of banana bunches at different viewing angles through a camera;

[0079] Step S102, performing feature extraction and feature matching on the banana bunch image to generate an MVS-Mask, which is used to supervise the reconstruction network;

[0080] Step S103, using the sparse point cloud model to generate an SDF coordinate system, the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate;

[0081] Step S104, inputting the banana bunch image and camera pose into a reconstruction network, and the reconstruction network learns to construct a geometric SDF field and a color field of the banana bunch by volume rendering;

[0082] Step S105, using the Marching Cubes algorithm to extract the implicit surface into a three-dimensional mesh model;

[0083] Step S106, calculating the mesh vertex colors and mapping them to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

[0084] The complex banana orchard environment brings more environmental noise, and the twisted and irregular geometric structure of the banana bunch brings greater depth changes. In order to enable the reconstruction network to better learn the sudden depth changes of the banana bunch and reduce the deviation of the original volume rendering, the present invention introduces a supervision mechanism for SDF field modeling. By constructing the SDF field to describe the geometric structure of the banana bunch and constructing the color field to describe the appearance color of the banana bunch, more accurate banana bunch geometry and color information can be provided in a complex orchard environment. Using MVS-Mask to establish high-quality mask information as a two-dimensional constraint, this can provide clear banana bunch geometry and color information, and can pay more attention to the reconstruction of the banana bunch, which can not only reduce the interference of environmental noise, but also improve the problem of self-occlusion. The volume rendering integral weighted method is used to render the banana bunch images under different viewing angles, and the Marching Cubes algorithm is used to extract the implicit surface as a three-dimensional mesh model, and the mesh vertex color is calculated and mapped to the three-dimensional mesh model, and finally a high-quality banana bunch three-dimensional mesh model with texture color information can be obtained. Based on this, the embodiment of the present invention can distinguish the subject and background information in a complex orchard environment, and can provide clear banana bunch geometry information. Let the reconstruction network pay more attention to the banana bunch and reduce the influence of environmental noise, so as to more accurately reconstruct the shape and details of the banana bunch.

[0085] It can be understood that two MLPs can be used to model the radiation field, where the SDF Network is used to model the geometric SDF field of the banana bunch, which is used to encode the geometric structure of the scene object, and the Color Network models the color field, which encodes the appearance color of the banana bunch. The SDF field describes the geometric structure characteristics of the banana bunch, and the color field describes the appearance information such as the texture color of the banana bunch. By inputting a two-dimensional image of a banana bunch with a posture in an orchard environment, a high-quality three-dimensional mesh model of a banana bunch with texture and color information can be finally obtained. The reconstruction network architecture of the present invention is as follows Figure 1B As shown, Figure 2 The figure is the structure diagram of the radiation field network. After preprocessing the banana bunch image data, the camera parameters and SFM sparse points can be obtained. These two parameters are sent to the neural network as input data, and the neural network is used to train the SDF field and color field. Adding SFM sparse points can supervise the training of the SDF field of the banana bunch, reduce the depth error, and allow the network to accurately learn the complex geometric structure of the banana bunch.

[0086] It can be understood that in order to learn the accurate SDF field of banana bunches in the orchard, when modeling the SDF field, the ray marching method is used to input the sampled spatial points as training points into the SDF Network to calculate the SDF value and gradient, such as Figure 3 . Each camera ray passes through an image pixel, samples the spatial particles on the camera's traveling ray, and calculates the spatial coordinates of each spatial particle to obtain the SDF value of each sampling point. Then use the sparse feature points of SFM to supervise the SDF field modeling, and finally calculate the loss and optimize the network parameters. In this way, the deviation of the SDF value is reduced and the depth misjudgment is alleviated. This allows the network to more accurately predict the SDF value of any spatial point, and finally extract the point where Sig(x)=0 as the surface point of the banana bunch. While optimizing the photometric deviation and rendering deviation of multiple viewpoints, a novel two-dimensional mask constraint is used to allow the network to focus more on the geometric structure of the object itself, reduce the network disturbance caused by background information, and enhance the ability of the radiation field to express the surface of the banana bunch.

[0087] It is understandable that the three-dimensional reconstruction of banana bunches in agricultural orchards has more uncertainty compared with the stable indoor experimental environment. The environmental information of the banana orchard is relatively complex, and the difference between the banana bunch and the background is small. The reconstruction network without effective supervision may not be able to effectively distinguish the main information in this environment, which will lead to confusion in the reconstruction result and artifacts. The present invention proposes to use MVS-Mask to establish high-quality mask information as a two-dimensional constraint, which can provide clear geometric information of the banana bunch. The network can pay more attention to the reconstruction of the banana bunch and reduce the impact of environmental noise, so as to more accurately reconstruct the shape and details of the banana bunch. For the two-dimensional constraint MVS-Mask, feature extraction and feature matching are first performed on the banana bunch image, and then a sparse reconstruction operation is performed to obtain a sparse point cloud model of the banana bunch. The camera projection matrix is ​​decomposed in the camera parameters, and then the projection matrix is ​​decomposed into a camera intrinsic parameter matrix and a camera extrinsic parameter matrix, as shown in formula (1).

[0088] P=K[R|-RC](1)

[0089] Where K is a 3×3 intrinsic parameter matrix, C is a column vector, which represents the camera position in the world coordinate system. [R|-RC] is a 3×4 extrinsic parameter matrix, which describes the camera posture. t = -RC, and then the relationship between the projection matrix P and the extrinsic parameter matrix is ​​obtained (2).

[0090] P=K[R|t](2)

[0091] Multiply the internal parameter matrix K and the external parameter matrix [R|t] to obtain a 3×4 projection matrix P, as shown in formula (3). x , f y is the focal length parameter, o x , o y is the principal point offset.

[0092]

[0093] The external parameter matrix [R|t] includes a 3×3 rotation matrix R and a 3×1 translation vector t. Performing QR decomposition on the submatrix consisting of the first 3x3 elements of the projection matrix P can uniquely decouple the product of K and R, and finally obtain an upper triangular matrix and a standard orthogonal matrix, namely the internal parameter matrix K and the external parameter matrix R, as shown in formula (4).

[0094]

[0095] Performing matrix operation on the last column of the projection matrix P yields the product of the intrinsic parameter matrix K and the translation vector t, formula (5).

[0096]

[0097] Then the inverse matrix of the K matrix is ​​calculated, and finally the translation vector t is obtained, formula (6).

[0098]

[0099] After obtaining the sparse point cloud, dense matching and deep fusion can be performed to generate a dense point cloud, and then the dense point cloud is projected onto a two-dimensional plane through the decomposed camera internal and external parameter information to generate a depth mask for each projected image. The effective depth information contained in the pixel point is identified through the depth mask, thereby identifying the banana bunch and environmental information. The interactive object segmentation method requires the user to manually calibrate the coverage points to process the mask, and this method is easily affected by the quality level of personal annotation. It is suitable for scenes with simple background information and obvious color differences, so it has great limitations and limited application scenarios. In complex banana orchard environments, misidentification is prone to occur, and the applicability is low. The method for generating a mask of the present invention can better cope with complex agricultural environments such as banana orchards. The method of the present invention can efficiently distinguish between the reconstructed subject and environmental information, not only generating high-quality object masks, but also improving the efficiency of network learning.

[0100] It is understandable that the complex banana orchard environment brings more environmental noise, and the twisted and irregular geometric structure of the banana bunch brings greater depth changes. In order to enable the reconstruction network to better learn the sudden depth changes of the banana bunch and reduce the deviation of the original volume rendering, a supervision mechanism for SDF field modeling is introduced. For the homogeneous coordinates of the three-dimensional space point X in the world coordinate system X = [x, y, z, 1] T , if the point is observed from two perspectives at the same time, their projection points are p1 and p2 respectively, and their image coordinates are x1 = [x1, y1, 1] T ,x2=[x2,y2,1] T , the camera projection matrices corresponding to the two images are P1 and P2, and there is formula (7).

[0101] x1=P1X,x2=P2X(7)

[0102] By cross-multiplying x1 on both sides of the first equation, we get x1×(P1X)=0, and p1 is expressed as [P 11 X,P 21 X,P 31 X], where P 11 ,P 21 ,P 31 They are the 1st to 3rd rows of the projection matrix P1, respectively, and we can get formula (8).

[0103] x1(P 13 X)-P 11 X=0

[0104] y1(P 13 X)-P 12 X=0(8)

[0105] x1(P 12 X)-y1(P 11 X)=0

[0106] The third equation of formula (8) can be obtained by linear transformation of the first two equations. Each perspective can provide two constraints, and then combined with the constraints of the second perspective, AX=0 can be obtained, where A is formula (9).

[0107]

[0108] Using the projective geometric relationship, the feature points between the matched image pairs are triangulated to obtain three-dimensional points, and finally a sparse point cloud of the object is generated. The sparse file contains the mapping relationship between the image ID and the 3D point ID and the mapping relationship between the 3D point ID and the corresponding 3D coordinates. Through these two mapping relationships, it is relatively easy to find the corresponding 3D coordinates using the image ID. Finally, these 3D point coordinates are standardized to the SDF coordinate system as the prior knowledge of the network learning the SDF field. With the zero-value prior of SDF as supervision, the reconstruction accuracy of the banana bunch surface is greatly improved.

[0109] The present invention uses a volume rendering integral weighted method to render multiple views of a banana bunch. The SDF network is used to obtain the points where SDF: Sig(x) = 0 from the feature points in each view to generate an SDF field. When a single image is rendered using volume integral, the ID mapping relationship of the image is used to find the corresponding image feature points to supervise the SDF field. The present invention uses the sparse point surface of the banana bunch as the zero-set level of the SDF. These sparse points may contain background noise, which needs to be manually removed to ensure the accuracy of the SDF zero value point.

[0110] It is understandable that the image data collected in the banana orchard environment may encounter the problem of sunlight projection, which will cause the banana bunches captured by the camera at different viewing angles to have different surface luminosity. The image data collected in the banana orchard environment may encounter the problem of sunlight projection, which will cause the banana bunches captured by the camera at different viewing angles to have different surface luminosity. Specifically, Figure 4(a)(b), due to the direct projection of light, light spots are formed on the surface and stems of the banana bunch. As the viewing angle of the banana bunch image changes, the color of the image viewpoint at that viewing angle deviates from the color of the banana bunch itself. When the sunlight on the surface of the banana bunch is blocked by obstacles such as leaves, the brightness of the surface of the banana bunch becomes lower, forming a shadow. That is, the light reflection intensity of the same viewpoint on the banana bunch may be different at different shooting angles.

[0111] When performing volume rendering integration, the same surface point presents different pixel values ​​in images at different viewing angles, which will directly affect the rendering results. However, the geometric appearance of the banana bunch should be rendered the same in different views. For the area block s on the surface of the banana bunch, such as Figure 5 The camera projection image I of the region block s at the viewpoint i ′ The projected pixel block on is pi ′ , the present invention needs to let the image projection pixel block pi of the area block s at different viewing angles ′ It is consistent with the projected pixel block pi" of the rendered image I".

[0112] The present invention uses the homography transformation (10) induced by the spatial plane to establish the relationship. Where R1 and R2 are the rotation matrices of the cameras under the two viewing angles, K1 and K2 are their intrinsic parameter matrices, C1 and C2 are the world coordinates of the optical centers of the two cameras, X represents a point on the induced plane, and n represents the plane normal vector. If the pixel block pi ′ The image point of is X, and the image point of the pixel block pi″ is Y, then the spatial plane induced homography formula is (11).

[0113]

[0114] X=HY(11)

[0115] The present invention uses NCC (Normalized Cross-Correlation) as the measure of image point similarity. The value range of NCC coefficient is -1 to 1. Its formula is expressed as (12), where I ′ is the projected image, I″ is the projected rendered image, Cov(I ′ (x),I″(y)) is I ′ and I″, Var(I ′ (x)) and Var(I″(y)) represent the variances of the two images themselves.

[0116]

[0117] In the projected image I ′When performing volume rendering, the pixel that the light passes through is taken as the center point, and a pixel block of 10×10 is used as the rendering block for window matching. The correlation between the two windows is measured by NCC, that is, the NCC coefficient of the same viewpoint of the projected image and the rendered image of the viewpoint under different viewing angles. The matching result is limited to the range of [-1,1] through normalization, and the NCC coefficient is used for deviation consistency loss. Then, the banana bunch images under different viewing angles are weighted averaged, and the weight is determined according to the confidence and exposure level of the viewing angle.

[0118] MVS-Mask improves the accuracy of 2D supervision, such as Figure 6 Compared with the interactive object segmentation method, MVS-Mask significantly reduces the environmental noise, and the stalk of the banana bunch is also accurately identified. More accurate mask information during training can more accurately locate the boundary of the object, thereby distinguishing the subject and background information, improving the self-occlusion problem, and allowing the network to focus more on learning the geometric structure of the object itself.

[0119] By comparing the rendering of image data taken in a banana orchard environment with the original neural implicit surface network, the rendering results of the original network are shown in Figure 2. Figure 7 As shown in part (b), the volume rendering process is disturbed by non-subject information, i.e., other banana bunches and branches, e.g. Figure 7 As shown in part (a), it incorrectly learns the background information outside the main body of the banana bunch and does not actually locate the surface of the banana bunch. This also shows that inaccurate mask information will cause incorrect volume rendering, resulting in the rendering result also containing more noise. The method of the present invention adds accurate mask information, which can clearly supervise the main body of the banana bunch, reduce the volume rendering deviation, and the rendering result has no obvious noise. Figure 7 As shown in part (c), it accurately locates the surface of the object and also improves the robustness of the neural network learning the SDF field.

[0120] The original network samples multiple spatial points on each camera ray by means of volume rendering integration, then performs weighted sum of the colors of the sampled points on the ray, and finally represents the output pixel color as the integral of the SDF field. It does not directly optimize the learning of the implicit surface points of the banana bunch, so the volume rendering of the original network is biased, and the learning effect on the surface of the banana bunch is poor in the orchard environment. The present invention significantly reduces the rendering error by using the sparse point zero-value prior as a loss during the volume rendering integration process. The original network without the SDF supervision mechanism contains a lot of noisy noise before the model training iteration number reaches 30,000, such as Figure 8. The method of the present invention adds SDF zero-value prior, which reduces the deviation between volume rendering and rendered surface points and accelerates the convergence of the network. When iter=10000, the method has learned the rough outline of the banana bunch, and when iter=30000, it has learned a clearer geometric structure. The pixel inference value obtained by volume rendering integration is less different from the reference image, where iter is the number of iterations of network training.

[0121] The normal map is an implementation of the bump map, which simulates the lighting effect at bumps and adds details to the model without adding polygons. In the process of iterative training of the comparison network model, the details of the surface of the banana bunch learned by the network were analyzed by exporting the normal map. The fruit surface of the banana bunch is relatively smooth, the geometric structure is relatively distorted, and the depth variation varies greatly. The original network lacks obvious geometric constraints, and it is difficult to represent the accurate details on the surface of the banana bunch in agricultural scenarios. When using the original network to model the banana bunch, the details of the surface of the banana bunch model are distorted, and the normal map has more depth error information, such as Fig. 9 The original network also adds wrong details to the surface of the banana bunch and lacks the details that the surface should have. The normal map of the banana bunch of the present invention is extracted during the training process, as shown in part (b). Fig. 9 As shown in part (c), the background noise of the banana bunch is significantly reduced and the overall outline is clearer, the depth prediction deviation is smaller, and the surface of a single banana fruit has better details.

[0122] Based on this, the present invention proposes an implicit neural surface reconstruction method BBNE-NeuS suitable for generating a high-fidelity 3D model from a 2D image of a banana bunch in an agricultural production environment. The present invention represents the surface of the reconstructed object as a zero set of SDF, and describes the geometry of the banana bunch by constructing an SDF field. In order to provide more accurate geometric information of the banana bunch in a complex orchard environment, the present invention also proposes MVS-Mask, which can not only reduce the interference of environmental noise, but also improve the problem of self-occlusion. At the same time, the SDF supervision mechanism and deviation consistency are introduced to optimize the mesh surface of the banana bunch. The network can eventually restore a high-fidelity three-dimensional mesh model of the banana bunch with texture and color information.

[0123] It should be pointed out that in the complex outdoor banana orchard environment, the stems and leaves may block the banana bunches, and the color of the unpicked banana bunches is very similar to the environmental color of the banana orchard. The present invention proposes MVS-Mask, which enables the reconstruction network to accurately identify the reconstruction subject in the complex banana orchard environment and complete high-quality reconstruction, reducing environmental noise interference and self-occlusion problems.

[0124] It should be pointed out that the shape of the banana bunch is relatively irregular, and it is composed of multiple single fruits arranged closely in a spiral shape. The present invention introduces SDF supervision in the reconstruction network to reduce the error of SDF zero value points and reduce depth misjudgment, so that the network can capture more geometric details of the banana bunch.

[0125] It should be pointed out that, in order to solve the problem of inconsistent viewpoints of banana surfaces under multiple viewing angles caused by unstable ambient light in the orchard, the present invention uses the deviation consistency of multiple views to reduce the difference in rendering results.

[0126] The present invention adopts a method based on neural implicit representation. Compared with explicit representation, neural implicit representation can smoothly approximate continuous functions, can more naturally process the surface of banana bunches and can represent its complex shape details. When processing incomplete or noisy data, neural implicit representation can infer and complete the missing parts through the learning ability of neural networks, thereby generating a complete model. The present invention also provides accurate two-dimensional constraint information, which enables the reconstruction network to learn to reconstruct the subject in a complex environment.

[0127] This paper takes bananas in banana orchards under natural environment as the research object, proposes a three-dimensional reconstruction method based on neural implicit surface, and deeply studies the 3D reconstruction information perception of banana fruit by adopting theoretical methods such as deep learning, mask constraint, and field modeling. Phenotypic analysis of the three-dimensional mesh model of banana bunches can estimate the size and weight parameters of the fruit, so that transportation vehicles can be reasonably arranged in advance, transportation costs can be accurately controlled, and accurate prediction of yield can be achieved. This also provides an innovative theory for the fine management of banana orchards in my country, and builds a bridge for realizing a new smart agricultural ecology with precise analysis.

[0128] In addition, if Fig.10 As shown, one embodiment of the present invention further discloses a three-dimensional reconstruction device based on neural implicit surface, the device comprising:

[0129] An acquisition module 110 is used to acquire images of banana bunches at different viewing angles through a camera;

[0130] A first generation supervision module 120 is used to extract and match features of the banana bunch image to generate an MVS-Mask, and the MVS-Mask is used to supervise the reconstruction network;

[0131] A second generation supervision module 130 is used to generate an SDF coordinate system using a sparse point cloud model, the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate;

[0132] A reconstruction module 140 is used to input the banana bunch image and camera pose into a reconstruction network, and the reconstruction network learns to construct a geometric SDF field and a color field of the banana bunch by volume rendering;

[0133] An extraction module 150, for extracting the implicit surface into a three-dimensional mesh model using a Marching Cubes algorithm;

[0134] The mapping module 160 is used to calculate the colors of mesh vertices and map them to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

[0135] The lithium-ion battery capacity prediction device of the embodiment of the present invention is used to execute the three-dimensional reconstruction method based on neural implicit surface in the above embodiment. Its specific processing process is the same as the three-dimensional reconstruction method based on neural implicit surface in the above embodiment, and will not be repeated here.

[0136] In addition, if Fig.11 As shown, an embodiment of the present invention also discloses an electronic device, comprising: at least one processor 210; at least one memory 220, for storing at least one program; when the at least one program is executed by the at least one processor 210, a three-dimensional reconstruction method based on neural implicit surface as in any of the previous embodiments is implemented.

[0137] In addition, an embodiment of the present invention further discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the three-dimensional reconstruction method based on neural implicit surface as in any of the previous embodiments.

[0138] The system architecture and application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

[0139] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0140] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0141] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process or an execution thread, and a component may be located on a computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems through signals).

Claims

1. A 3D reconstruction method based on neural implicit surface, It is characterized in that include: The banana bunch images at different viewing angles are acquired by using a camera; Performing feature extraction and feature matching on the banana bunch image to generate an MVS-Mask, wherein the MVS-Mask is used to supervise the reconstruction network; A sparse point cloud model is used to generate an SDF coordinate system, wherein the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate; Inputting the banana bunch image and camera pose into the reconstruction network, the reconstruction network learns to construct the geometric SDF field and color field of the banana bunch by volume rendering; The Marching Cubes algorithm is used to extract the implicit surface into a three-dimensional mesh model; The mesh vertex colors are calculated and mapped to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

2. The three-dimensional reconstruction method based on neural implicit surface according to claim 1, It is characterized in that The step of extracting and matching features of the banana bunch image to generate an MVS-Mask includes: Performing feature extraction and feature matching on the banana bunch image to obtain geometric structure features and appearance color features; Sparse reconstruction is performed based on geometric structure features and appearance color features to obtain a sparse point cloud model; Generate dense point cloud by densely matching and deeply fusing sparse point cloud model files; The dense point cloud is projected onto a two-dimensional plane using the decomposed camera internal and external parameter information to generate a depth mask for each projected image.

3. The three-dimensional reconstruction method based on neural implicit surface according to claim 1, It is characterized in that The geometric SDF field and color field of the banana bunch are constructed, including: The geometric SDF field of banana bunch modeling based on SDF Network; The color field of the banana bunch is modeled based on the Color Network.

4. The three-dimensional reconstruction method based on neural implicit surface according to claim 3, It is characterized in that The geometric SDF field of the banana bunch modeling based on SDF Network includes: The sampled spatial points are input as training points into the SDF Network using the ray marching method to calculate the SDF value and gradient; Preprocessing the banana bunch image data to obtain camera parameters and SFM sparse feature points; According to the SFM sparse feature point supervised SDF field modeling, the loss is calculated and the network parameters of the SDF Network are optimized to obtain the geometric SDF field, wherein the network parameters include SDF values ​​and gradients.

5. The three-dimensional reconstruction method based on neural implicit surface according to claim 2, It is characterized in that The dense point cloud is projected onto a two-dimensional plane through the decomposed camera internal and external parameter information to generate a depth mask for each projected image, including: Obtaining camera parameters of the camera; Decompose the camera's projection matrix from the camera parameters; Decomposing the projection matrix into a camera intrinsic parameter matrix and a camera extrinsic parameter matrix; The dense point cloud is projected onto a two-dimensional plane according to the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and a depth mask of each projected image is generated.

6. The three-dimensional reconstruction method based on neural implicit surface according to claim 1, It is characterized in that The method further comprises: Obtain image projection pixel blocks at different viewing angles; Establishing a relationship between the image projection pixel blocks at different viewing angles using a homography induced by a spatial plane; When rendering the image projection pixel block, the pixel through which the light travels is taken as the center point, and the pixel block with a size of 10×10 is used as the rendering block for window matching; Calculate the NCC coefficient of the same viewpoint of the projected image and the rendered image of the viewpoint under different viewing angles; Normalizing the NCC coefficient, and using the NCC coefficient to perform deviation consistency loss; The banana bunch images at different viewing angles are weighted averaged to obtain a weighted average result, wherein the weight of the weighted average is determined according to the confidence level and exposure level of the viewing angle.

7. The three-dimensional reconstruction method based on neural implicit surface according to claim 1, It is characterized in that The method further comprises: Generate a geometric SDF field by obtaining salient points from feature points in each view through the SDF network; When a single image is rendered using volume integral, the ID mapping relationship of the single image is used to find the corresponding image feature points to supervise the geometric SDF field.

8. A three-dimensional reconstruction device based on neural implicit surface, It is characterized in that The device comprises: An acquisition module is used to acquire banana bunch images at different viewing angles through a camera; A first generation supervision module is used to perform feature extraction and feature matching on the banana bunch image to generate an MVS-Mask, and the MVS-Mask is used to supervise the reconstruction network; A second generation supervision module is used to generate an SDF coordinate system using a sparse point cloud model, wherein the SDF coordinate system is used to supervise the geometric SDF field, and the sparse point cloud model includes a mapping relationship between an image ID and a 3D point ID and a mapping relationship between a 3D point ID and a corresponding 3D coordinate; A reconstruction module, used for inputting the banana bunch image and camera pose into the reconstruction network, and the reconstruction network learns to construct a geometric SDF field and a color field of the banana bunch by volume rendering; Extraction module, used to extract implicit surface into 3D mesh model using Marching Cubes algorithm; The mapping module is used to calculate the colors of the mesh vertices and map them to the three-dimensional mesh model to obtain a three-dimensional mesh model with appearance information.

9. An electronic device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional reconstruction method based on neural implicit surface as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the three-dimensional reconstruction method based on neural implicit surface as described in any one of claims 1 to 7.

Images