3D Reconstruction Methods, Devices, Equipment, and Media Based on Multi-View Normal Aggregation

Abstract

This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for 3D reconstruction based on multi-view normal aggregation. The method includes: acquiring multi-view images of a target object under different lighting conditions; obtaining normal images corresponding to the multi-view images, and obtaining a relative depth map of the normal images using a normal integration method; inputting the normal images and the relative depth map into a trained preset network for 3D reconstruction to obtain a 3D reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network; evaluating the 3D reconstruction result, and optimizing the preset network based on the evaluation result. This application solves the problem of 3D reconstruction of highly reflective and sparsely textured objects, enabling reconstruction without prior knowledge of the camera pose, and improving reconstruction accuracy through multi-view normal aggregation.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device and medium for three-dimensional reconstruction based on multi-view normal aggregation. Background Technology

[0002] Detailed 3D reconstruction from multi-view image observations is a fundamental task in computer vision, driving applications such as virtual reality and electronic cultural heritage. A typical workflow involves first calibrating the camera pose for each image, then using the pose-calibrated images to reconstruct the shape. Many methods have achieved good results in scenes with diffuse and specular surfaces. However, surface reconstruction without pose calibration (which is crucial in practical, informal capture scenarios) remains challenging for reflective and textureless surfaces.

[0003] Existing methods for reconstructing reflective surfaces, such as NeRO, require precise camera pose calibration. To facilitate this calibration, a calibration plate is typically placed under the object, limiting the applicability of the method in more casual settings. For reflective and textureless surfaces, extracting reliable feature correspondences is particularly challenging due to the view-dependent reflectivity of the reflections, which alters pixel values. Therefore, the lack of texture further complicates reliable shape estimation using learning-based monocular depth estimators. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a three-dimensional reconstruction method, apparatus, device, and medium based on multi-view normal aggregation. This method requires no pose input, facilitates the extraction of reliable feature correspondences, and has few limitations in applicability. Based on multi-view normal aggregation, it can achieve accurate three-dimensional reconstruction results.

[0005] The first aspect of this application provides a three-dimensional reconstruction method based on multi-view normal aggregation, the method comprising:

[0006] Acquire multi-view images of the target object under different lighting conditions;

[0007] Obtain the normal image corresponding to the multi-view image, and use the normal integration method to obtain the relative depth map of the normal image;

[0008] The normal image and the relative depth map are input into a trained preset network for 3D reconstruction to obtain the 3D reconstruction result. The preset network includes a pose parameter network and an SDF shape network.

[0009] The three-dimensional reconstruction results are evaluated, and the preset network is optimized based on the evaluation results.

[0010] In some embodiments of this application, obtaining the normal image corresponding to the multi-view image includes:

[0011] For the multi-view images, they are grouped according to images with the same viewpoint but different lighting conditions;

[0012] The photometric stereo method is used to process each group of images to obtain the normal image of the corresponding viewpoint.

[0013] In some embodiments of this application, obtaining the relative depth map of the normal image using the normal integration method includes:

[0014] Surface normal information is extracted from the normal image to obtain the gradient field;

[0015] The gradient field is optimized by minimizing the functional;

[0016] The relative depth map for each viewpoint is calculated based on the optimization results.

[0017] In some embodiments of this application, the step of inputting the normal image and the relative depth map into a trained preset network for 3D reconstruction to obtain the 3D reconstruction result includes:

[0018] The normal image and the relative depth map are input into the pose parameter network to obtain the camera pose parameters for each viewpoint;

[0019] The camera pose parameters are used to transform the normals of each viewpoint from the camera coordinate system to the world coordinate system, and sampling is performed in three-dimensional space based on the transformation results.

[0020] The sampled spatial points are input into the SDF shape network to obtain the symbolic distance value of each sampled point;

[0021] The three-dimensional reconstruction result is obtained based on the symbolic distance value.

[0022] In some embodiments of this application, obtaining the 3D reconstruction result based on the symbolic distance value includes:

[0023] Volume rendering samples are performed on the spatial points along the light direction, and the opacity and cumulative opacity of each sample point are calculated.

[0024] Based on the opacity and the cumulative opacity, a depth map, a surface normal map, and an opacity map are rendered for each viewpoint.

[0025] The 3D reconstruction results are obtained based on the depth map, surface normal map, and opacity map from each viewpoint.

[0026] In some embodiments of this application, evaluating the 3D reconstruction results and optimizing the preset network based on the evaluation results includes:

[0027] Calculate the chamfer distance and shape score between the point cloud of the 3D reconstruction result and the point cloud of the real shape corresponding to the target object;

[0028] The parameters of the preset network are optimized based on the chamfer distance and shape score.

[0029] In some embodiments of this application, the training method of the preset network includes:

[0030] Construct a training dataset that includes both synthetic and real data;

[0031] Design a joint optimization loss function, including normal loss, integral loss, and multi-view loss. Figure 1 Consistency loss, masking loss, and isometric loss;

[0032] The pose parameter network and the SDF shape network are jointly trained end-to-end using the gradient descent method;

[0033] Training stops when the loss function converges and the preset number of training iterations is reached.

[0034] A second aspect of this application provides a three-dimensional reconstruction apparatus based on multi-view normal aggregation, the apparatus comprising:

[0035] The acquisition module is used to acquire multi-view images of the target object under different lighting conditions;

[0036] The acquisition module is used to acquire the normal image corresponding to the multi-view image, and to obtain the relative depth map of the normal image using the normal integration method;

[0037] The reconstruction module is used to input the normal image and the relative depth map into a trained preset network for 3D reconstruction to obtain a 3D reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network.

[0038] An evaluation module is used to evaluate the 3D reconstruction results and optimize the preset network based on the evaluation results.

[0039] A third aspect provides an electronic device including a memory and a processor, wherein the memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform a three-dimensional reconstruction method based on multi-view normal aggregation as described in various embodiments of this application.

[0040] The fourth aspect provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional reconstruction method based on multi-view normal aggregation as described in the embodiments of this application.

[0041] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0042] The 3D reconstruction method based on multi-view normal aggregation described in various embodiments of this application acquires multi-view images of the target object under different lighting conditions, obtains the normal images corresponding to the multi-view images, and uses the normal integration method to obtain the relative depth map of the normal image. The normal image and the relative depth map are then input into a trained preset network for 3D reconstruction to obtain the 3D reconstruction result. The preset network includes a pose parameter network and an SDF shape network. The 3D reconstruction result is evaluated, and the preset network is optimized based on the evaluation result. Thus, this application solves the problem of 3D reconstruction of highly reflective and sparsely textured objects, enabling reconstruction without prior knowledge of the camera pose, and improving reconstruction accuracy through multi-view normal aggregation. Furthermore, the training process of the preset network, due to the design of an optimized loss function, improves the model's generalization ability, ensuring the geometric consistency and accuracy of the 3D reconstruction. In addition, by evaluating the 3D reconstruction result and optimizing the preset network based on the evaluation result, an objective quality evaluation standard is provided, which can guide the network optimization direction, thereby ensuring the reliability of the reconstruction result.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0045] Figure 1 This is a schematic diagram illustrating the steps of a three-dimensional reconstruction method based on multi-view normal aggregation in an exemplary embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a normal map acquisition process in an exemplary embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the structure of a preset network in an exemplary embodiment of this application;

[0048] Figure 4This is a schematic diagram of the structure of a three-dimensional reconstruction method based on multi-view normal aggregation in an exemplary embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0051] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments depicted herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0052] High-precision 3D reconstruction from multi-view images is a core task in computer vision, widely applied in virtual reality, digital cultural heritage preservation, and many other fields. Traditional methods typically require calibrating the camera pose for each image first, and then using the calibrated pose information to recover the 3D shape of objects from the image data. These methods have achieved significant results when dealing with scenes with diffuse and specular reflective surfaces. However, for highly reflective and textureless surfaces, especially for surface reconstruction without pose calibration (such as in informal shooting scenarios), significant technical challenges remain.

[0053] Existing methods for reconstructing reflective surfaces (such as NeRO) rely on precise camera pose calibration. Achieving this calibration typically requires placing a calibration plate beneath the object, limiting its applicability in casual shooting scenarios. Furthermore, for highly reflective and textureless surfaces, extracting reliable feature correspondences becomes particularly difficult because reflective properties change pixel values ​​with varying viewing angles. The lack of texture further increases the uncertainty in shape estimation for learning-based monocular depth estimation methods.

[0054] To address this, this application proposes a 3D reconstruction method, apparatus, device, and medium based on multi-view normal aggregation. This method does not rely on camera pose input, can easily extract reliable feature correspondences, and has wide applicability. Through multi-view normal aggregation processing, high-precision 3D reconstruction results can be achieved, effectively solving the technical bottlenecks of reconstruction on highly reflective and textureless surfaces.

[0055] Figure 1 The steps of a 3D reconstruction method based on multi-view normal aggregation are shown, such as... Figure 1 As shown, the method includes steps S1-S4.

[0056] S1. Acquire multi-view images of the target object under different lighting conditions.

[0057] In one possible implementation, multiple images of the target object are first taken from different perspectives, and the lighting conditions at the time of image capture should vary to some extent. For specific implementation, refer to... Figure 2 Using a standard RGB camera in an indoor environment, multiple images of an object are captured from each viewpoint, each illuminated by different angles of light. The lighting is achieved by hand-holding and controlling the flash angle. When capturing images from the same viewpoint, the camera position is fixed using a tripod. The flash is then held hand-held, orbiting the object and its position controlled to ensure varying angles of illumination. Images from the same viewpoint are captured under these different lighting conditions, completing one viewpoint's capture. For other viewpoints, the camera pose is varied, and the same steps are followed. The resulting images are RGB images from multiple viewpoints and under various lighting conditions. When pre-training a preset model or a model for estimating normals, these images need to be processed to obtain normal images, which are then used as labels to train the model. Figure 2 The shooting setup shown involves placing the target object between the camera and a light source. The intensity of the light source can be continuously adjusted, allowing the camera to capture images from different angles, thus obtaining multiple results. Figure 2 The images shown are multi-view and multi-light images. It can be understood that multi-view images can provide rich depth information, and changes in lighting can help obtain the normal information of the object's surface, which is helpful for the subsequent generation of normal images and depth maps.

[0058] S2. Obtain the normal image corresponding to the multi-view image, and use the normal integration method to obtain the relative depth map of the normal image.

[0059] In one possible implementation, the multi-view images are grouped according to images with the same viewpoint but different illuminations; each group of images is processed using a photometric stereo method to obtain the normal image of the corresponding viewpoint (e.g., ...). Figure 2 The normal map is obtained by processing the photometric stereo method in the image. The method of obtaining the relative depth map of the normal image using the normal integral method includes: extracting surface normal information from the normal image to obtain a gradient field; optimizing the gradient field by minimizing the functional; and calculating the relative depth map for each viewpoint based on the optimization result.

[0060] It's important to note that training the preset model requires not only real data but also a large amount of synthetic data. For synthetic data, we can directly obtain normal maps from Blender rendering software, which is relatively simple. However, processing real data is more complex, requiring the use of photometric stereo techniques to process the image and obtain the normal map. Specifically, photometric stereo processing techniques such as the SDM_UNIPS algorithm are employed. These algorithms can estimate the surface normals of an object by analyzing the brightness changes of the object under different lighting conditions. This process involves complex image processing techniques, such as image segmentation, feature extraction, and photometric analysis, to ensure accurate extraction of normal information from images captured under multiple viewpoints and lighting conditions.

[0061] S3. Input the normal image and the relative depth map into the trained preset network for 3D reconstruction to obtain the 3D reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network.

[0062] refer to Figure 3 The preset network comprises two parts: a pose parameter network and an SDF shape network, which work together to complete the 3D reconstruction task. The pose parameter network first optimizes the estimation of camera pose, including rotation and translation parameters, based on the input data. This process ensures that we can accurately know the camera's position and orientation at the time of each image capture, thereby enabling us to correctly interpret the data captured from each viewpoint.

[0063] Then, the SDF shape network is responsible for learning the 3D shape of the object from the optimized pose and the processed data. The SDF (Signed Distance Function) network represents the shape of an object implicitly; that is, for any input spatial point, the network outputs the nearest distance from that point to the object's surface and its sign, indicating whether the point is inside or outside the object. Training the SDF network involves a significant amount of mathematical and computational processes, including network structure design, loss function construction, and optimization algorithm selection.

[0064] The pose parameter network is used to optimize the camera's pose parameters (rotation and translation matrices). The pose parameter network and the SDF shape network are jointly trained to ensure that pose optimization and 3D reconstruction tasks complement each other. This application performs joint optimization of pose and shape, that is, by jointly optimizing the pose parameter network and the SDF shape neural network, it ensures the geometric consistency between the 3D reconstruction and the camera pose. Ultimately, this improves the accuracy of pose optimization and achieves high-precision 3D reconstruction of the object surface.

[0065] In one possible implementation, the normal image and the relative depth map are input into a pre-trained preset network for 3D reconstruction. To obtain the 3D reconstruction result, firstly, the normal image and the relative depth map are input into the pose parameter network to obtain camera pose parameters for each viewpoint; then, the normals of each viewpoint are transformed from the camera coordinate system to the world coordinate system using the camera pose parameters, and sampling is performed in 3D space based on the transformed result; then, the sampled spatial points are input into the SDF shape network to obtain the signed distance value of each sampling point; then, volume rendering sampling is performed on the spatial points along the ray direction, and the opacity and cumulative opacity of each sampling point are calculated; based on the opacity and cumulative opacity, a depth map, surface normal map, and opacity map are rendered for each viewpoint; finally, the 3D reconstruction result is obtained based on the depth map, surface normal map, and opacity map for each viewpoint.

[0066] Specifically, monocular normal integration aims to reconstruct a relative height map from a given normal map. The surface normal at a point in camera space is defined by formula (1):

[0067]

[0068] Where n c The 'c' represents the surface normal of a point in camera space (c is an abbreviation for camera, representing the camera coordinate system), and x, y, z represent the three axes of the camera coordinate system.

[0069] Gradient field under orthogonal projection It can be calculated from the surface normal using formula (2):

[0070]

[0071] Furthermore, the problem of normal integrals can be solved by minimizing the following functional using formula (3):

[0072]

[0073] in, and They represent Ω respectively n →R is the partial derivative of the normal image along the u and v axes on the image plane, and z represents the relative depth map of the normal image. Based on this optimization method, single-view depth can be recovered under perspective projection.

[0074] like Figure 3As shown, after inputting a 3D spatial point x, the output is the signed distance f(x) from that point to the object's surface, where f(x) > 0 indicates the point is outside the object, and f(x) < 0 indicates the point is inside the object. The input is a 3D spatial point. Hash encoding is used to efficiently represent the spatial features of the input point, improving the model's ability to represent complex geometry. The main network h(x; φ) is a multilayer perceptron (MLP), which learns the SDF value through nonlinear mapping. Then, a connection operation is performed, inputting the original input point x along with the hash-encoded features into the main network, preserving the original geometric information to improve reconstruction details. Furthermore, a nonlinear activation function is used to increase the model's expressive power and adapt to complex geometric shapes. The final output f(x) is the signed distance from the input point to the object's surface. f(x) = 0 represents the point being on the object's surface, and f(x) < 0 represents the object being inside the object. The surface can be considered as the set of all points x that satisfy f(x) = 0, as shown in formula (4):

[0075]

[0076] Based on NeuS, SDF representation is connected to volume rendering. Specifically, given K ordered 3D points on a ray... and its SDF value The volume opacity of a spatial point x1 is calculated as shown in formula (5):

[0077]

[0078] in It is a sigmoid function with a learnable parameter s, representing a point x on a ray. i Cumulative transparency T i This can be expressed as formula (6):

[0079]

[0080] The opacity α was calculated above. i and transparency T i Based on this, we can render the depth, surface normal, and opacity of pixel p, as shown in formulas (7)-(9):

[0081]

[0082] in: d represents the estimated depth value at pixel p. i This represents the depth value at the i-th point. This represents the normal to the predicted world coordinate system. This represents the gradient of the SDF value at the i-th point. This represents the estimated opacity. By supervising the information obtained from the above calculations, the SDF neural network can be constrained, thereby achieving 3D reconstruction.

[0083] In one possible implementation, the training method for the preset network includes: constructing a training dataset comprising synthetic data and real data; designing a jointly optimized loss function, including normal loss, integral loss, and multi-view loss. Figure 1 Consistency loss, masking loss, and isometric loss are used; the pose parameter network and the SDF shape network are jointly trained end-to-end using gradient descent; training stops when the loss function converges and the number of training iterations reaches a preset number. The loss function for joint optimization is Equation (10):

[0084]

[0085] Where, λ i It is a coefficient that balances different loss terms. This represents the surface normal loss from the world coordinate system to the camera coordinate system. Given the world surface normal projected onto pixel position p by an SDF shape network. And the camera space surface normals recorded in the input surface normal map. Defined as formula (11):

[0086]

[0087] Where N represents the total number of input viewpoints, R i This represents the transformation from the camera coordinate system to the world coordinate system. Representing the loss due to normal integration, given a surface normal map, the integration depth map z is obtained using the normal integration method described above. ni The depth map z obtained by integrating the normal is... ni There exists an inherent scale ratio α between the actual depth z and the depth z, i.e., z = αz. ni To address this problem, given a depth map z rendered by a neural network... r The depth map z obtained by integrating with the normal ni The scaling factor α is calculated using the least squares method, as shown in formula (12):

[0088]

[0089] Then, scale ambiguity is calculated for each view, using these integral depth maps, and obtained through a regularized SDF neural network. As in formula (13):

[0090]

[0091] Where i represents the i-th viewpoint, and there are a total of N viewpoints.

[0092] The multi-view normal consistency loss is represented by reprojecting from scene points sampled by the SDF to a 2D image plane defined by the camera pose, finding dynamic 2D correspondences in each iteration. Based on these 2D correspondences, we measure the consistency of the surface normal map in camera space.

[0093] Specifically, first, a ray is projected from the reference viewpoint, and the point of contact x between this ray and the object's surface is found. Then, the projection matrix Π={π i =[R i ,t i Projecting x onto all other camera views from the group ] |i=0,…,N-1}, we obtain the corresponding surface normal n in camera space. c (π i (x)). Theoretically, these surface normals can be rotated to the same world normal by the corresponding camera pose. Under this constraint, the loss at pixel p is defined as Equation (14):

[0094]

[0095] in, And represents the rotation matrix in the reference viewpoint, R represents the camera coordinate system normal in the reference viewpoint. i The rotation matrix representing the i-th viewpoint. π represents the normal to the camera coordinate system in the i-th viewpoint. i This is the projection matrix for the i-th viewpoint. Since point x is not visible in all viewpoints, this application introduces a visibility mask function γ based on ray tracing. i (x), that is:

[0096]

[0097] Based on this visibility check, we reformulate the multi-view normal consistency loss as Equation (15):

[0098]

[0099] This loss is calculated and accumulated under different pixels and reference viewpoints.

[0100] This represents the Mask loss, which is used to construct the labeled contour of the constraint / target shape, as shown in Equation (16):

[0101]

[0102] Among them, oi (p) and These correspond to the ground truth mask and the rendered mask at pixel position p, respectively. BCE(·) represents the binary cross-entropy function. The mask loss can constrain the contour of the target object.

[0103] at last, Let Eikonal loss be denoted as . To make the SDF gradient norm close to 1 in almost all regions, thus making the neural SDF approximately effective, we introduce Eikonal loss, as shown in the following formula (17):

[0104]

[0105] Where x represents the sampling point. The Eikonal loss further optimizes the surface smoothness, ultimately achieving accurate and coherent 3D surface reconstruction.

[0106] It is understandable that by introducing parameters specific to surface normals, depth integrals, and model views... Figure 1 The loss terms include consistency, contour mask, and Eikonal, and the total loss function (10) guides the network to learn model parameters that can simultaneously optimize camera pose, surface geometry, and visual appearance. This ensures that the network can learn accurate 3D shape, pose parameters, and surface details from the data, greatly improving the accuracy and consistency of 3D reconstruction.

[0107] Through steps S1 to S3, the problem of 3D reconstruction of highly reflective and sparsely textured objects is solved. Reconstruction can be completed without prior knowledge of the camera pose, and the reconstruction accuracy is improved by multi-view normal aggregation.

[0108] S4. Evaluate the three-dimensional reconstruction results and optimize the preset network based on the evaluation results.

[0109] In one specific implementation, the three-dimensional reconstruction result is evaluated, and the preset network is optimized based on the evaluation result, including: calculating the chamfer distance and shape score between the point cloud of the three-dimensional reconstruction result and the real shape point cloud corresponding to the target object; and optimizing the parameters of the preset network based on the chamfer distance and shape score.

[0110] Chamfer distance (CD) is a metric that measures the similarity between two sets of point clouds. It is calculated based on the average of the nearest distances between individual points in the point clouds. Here, it is used to measure the quality of 3D model reconstruction, specifically the difference between the shape point cloud recovered by the algorithm and the true shape point cloud.

[0111] For example, given two point clouds χ1 and χ2, we first define the distance from point x1 in point cloud χ1 to point cloud χ2. For formula (18):

[0112]

[0113] Similarly, the distance from point x2 in point cloud x2 to point cloud x1 is... Defined as formula (19):

[0114]

[0115] The magnitude of the CD index, d(χ1,χ2), is defined by formula (20):

[0116]

[0117] Where |χ1| represents the number of points in point cloud χ1, and |χ2| represents the number of points in point cloud χ2.

[0118] Understandably, this metric can intuitively reflect the difference in geometric distance between the reconstructed model and the real model, and is an important basis for optimizing the parameters of the reconstruction network.

[0119] In addition, this application also uses a shape score (F-Score) for evaluation, which is a metric for evaluating the quality of 3D reconstruction based on the harmonic mean of precision and recall. This score comprehensively considers the completeness and accuracy of the reconstructed point cloud. Precision measures the proportion of points in the reconstructed point cloud that are correct (i.e., match the true shape point cloud within a certain distance threshold), while recall measures the proportion of points in the true shape point cloud that are correctly covered by the reconstructed point cloud.

[0120] Specifically, precision and recall are defined based on the distance from a point to a set of points as shown in Equations (21) and (22):

[0121]

[0122]

[0123] Where [·] is the Iverson symbol, with a value of 1 when the condition within the brackets is true, and 0 otherwise; τ is the distance threshold, used to determine whether a point is close enough to another set of points. |χ1| represents the number of points in point cloud χ1, and |χ2| represents the number of points in point cloud χ2. The F-score is calculated as follows: P and R represent precision and recall, respectively. τ is usually set to a small constant. In the evaluation of this application, τ is preferably set to 0.5 mm to ensure the rigor and practicality of the evaluation.

[0124] By calculating the chamfer distance and shape score between the point cloud of the 3D reconstruction result and the point cloud of the true shape of the target object, the accuracy and completeness of the reconstruction model can be accurately and quantitatively analyzed. These evaluation results not only visually demonstrate the performance of the reconstruction model but also serve as optimization targets during the training process. In particular, by incorporating these evaluation metrics as part of the loss function, the pre-defined network can be guided to better learn the geometric features and details of the real object during training, thereby achieving continuous optimization of model performance.

[0125] In addition, this application also uses the following two metrics to evaluate the accuracy of pose recovery: RPEr (Relative Pose Error in Rotation) and RPEt (Relative Pose Error in Translation).

[0126] Relative rotation error measures the error between the estimated rotation and the actual rotation between two frames, as shown in Equation (23):

[0127]

[0128] Among them, R est R is the estimated rotation matrix. gt It is a real rotation matrix, and Tr(·) represents the trace of the matrix (the sum of the diagonal elements).

[0129] The relative translation error measures the error between the estimated and actual translation of the camera between two frames, as shown in Equation (24):

[0130] RPEt=|t est -t gt |2 Formula (24)

[0131] Among them, t est It is the estimated translation vector, t gt It is the actual translation vector, and |·|2 represents the Euclidean norm (i.e. the magnitude of the absolute value of the translation error).

[0132] It is understandable that step S4 is not only a crucial step in evaluating reconstruction quality, but also a key step in continuously improving and optimizing the 3D reconstruction neural network model based on detailed and specific quantitative analysis results. Through this iterative process of evaluation-optimization-re-evaluation, the overall performance of 3D reconstruction can be significantly improved, achieving the goal of accurately reconstructing highly reflective and sparsely textured objects.

[0133] In some embodiments of this application, a three-dimensional reconstruction device based on multi-view normal aggregation is provided, such as... Figure 4 As shown, the device includes:

[0134] Acquisition module 401 is used to acquire multi-view images of the target object under different lighting conditions;

[0135] The acquisition module 402 is used to acquire the normal image corresponding to the multi-view image, and to obtain the relative depth map of the normal image using the normal integration method;

[0136] Reconstruction module 403 is used to input the normal image and the relative depth map into a trained preset network for three-dimensional reconstruction to obtain a three-dimensional reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network;

[0137] Evaluation module 404 is used to evaluate the three-dimensional reconstruction results and optimize the preset network based on the evaluation results.

[0138] The 3D reconstruction device based on multi-view normal aggregation solves the problem of 3D reconstruction of highly reflective and sparsely textured objects. It can complete the reconstruction without knowing the camera pose in advance and improves the reconstruction accuracy through multi-view normal aggregation.

[0139] Please refer to the following. Figure 5 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 5 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected via the bus 202. The memory 201 stores a computer program that can run on the processor 200. When the processor 200 runs the computer program, it executes the three-dimensional reconstruction method based on multi-view normal aggregation provided in any of the foregoing embodiments of this application. The method includes: acquiring multi-view images of a target object under different lighting conditions; obtaining normal images corresponding to the multi-view images and using a normal integration method to obtain a relative depth map of the normal images; inputting the normal images and the relative depth map into a trained preset network for three-dimensional reconstruction to obtain a three-dimensional reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network; evaluating the three-dimensional reconstruction result and optimizing the preset network based on the evaluation result.

[0140] The memory 201 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 203 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0141] Bus 202 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs. After receiving an execution instruction, the processor 200 executes the program. The control method disclosed in any of the foregoing embodiments of this application can be applied to the processor 200, or implemented by the processor 200.

[0142] The processor 200 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 200 or by instructions in software form. The processor 200 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 201. The processor 200 reads the information in memory 201 and, in conjunction with its hardware, completes the steps of the three-dimensional reconstruction method based on multi-view normal aggregation.

[0143] This application also provides a computer-readable storage medium corresponding to the three-dimensional reconstruction method based on multi-view normal aggregation provided in the foregoing embodiments, wherein a computer program is stored thereon, and the computer program, when run by a processor, executes the three-dimensional reconstruction method based on multi-view normal aggregation provided in any of the foregoing embodiments.

[0144] In addition, examples of the computer-readable storage medium may include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be described in detail here.

[0145] In addition, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional reconstruction method based on multi-view normal aggregation provided in any of the foregoing embodiments. The method includes: acquiring multi-view images of a target object under different lighting conditions; obtaining normal images corresponding to the multi-view images and obtaining a relative depth map of the normal images using a normal integration method; inputting the normal images and the relative depth map into a trained preset network for three-dimensional reconstruction to obtain a three-dimensional reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network; evaluating the three-dimensional reconstruction result and optimizing the preset network based on the evaluation result.

[0146] Those skilled in the art will understand that the various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art should understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application.

[0147] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method based on multi-view normal aggregation, characterized in that, The method includes: Acquire multi-view images of the target object under different lighting conditions; Obtain the normal image corresponding to the multi-view image, and use the normal integration method to obtain the relative depth map of the normal image; The normal image and the relative depth map are input into a trained preset network for 3D reconstruction to obtain the 3D reconstruction result. The preset network includes a pose parameter network and an SDF shape network. The 3D reconstruction results are evaluated, and the preset network is optimized based on the evaluation results; The training method for the preset network includes: Construct a training dataset that includes both synthetic and real data; Design joint optimization loss functions, including normal loss, integral loss, multi-view consistency loss, mask loss, and isometric loss; The pose parameter network and the SDF shape network are jointly trained end-to-end using the gradient descent method; Training is stopped when the loss function converges and the preset number of training iterations is reached. The loss function for the joint optimization is: in, These are coefficients that balance the different loss terms. Indicates normal loss. Indicates the integral loss. This represents the loss of consistency across multiple views. Indicates equidistant loss. This indicates the loss of the mask.

2. The three-dimensional reconstruction method based on multi-view normal aggregation according to claim 1, characterized in that, The step of obtaining the normal image corresponding to the multi-view image includes: For the multi-view images, they are grouped according to images with the same viewpoint but different lighting conditions; The photometric stereo method is used to process each group of images to obtain the normal image of the corresponding viewpoint.

3. The three-dimensional reconstruction method based on multi-view normal aggregation according to claim 1, characterized in that, The process of obtaining the relative depth map of the normal image using the normal integral method includes: Surface normal information is extracted from the normal image to obtain the gradient field; The gradient field is optimized by minimizing the functional; The relative depth map for each viewpoint is calculated based on the optimization results.

4. The three-dimensional reconstruction method based on multi-view normal aggregation according to claim 3, characterized in that, The step of inputting the normal image and the relative depth map into a trained preset network for 3D reconstruction to obtain the 3D reconstruction result includes: The normal image and the relative depth map are input into the pose parameter network to obtain the camera pose parameters for each viewpoint; The camera pose parameters are used to transform the normals of each viewpoint from the camera coordinate system to the world coordinate system, and sampling is performed in three-dimensional space based on the transformation results. The sampled spatial points are input into the SDF shape network to obtain the symbolic distance value of each sampled point; The three-dimensional reconstruction result is obtained based on the symbolic distance value.

5. The three-dimensional reconstruction method based on multi-view normal aggregation according to claim 4, characterized in that, The 3D reconstruction result obtained based on the symbolic distance value includes: Volume rendering samples are performed on the spatial points along the light direction, and the opacity and cumulative opacity of each sample point are calculated. Based on the opacity and the cumulative opacity, a depth map, a surface normal map, and an opacity map are rendered for each viewpoint. The 3D reconstruction results are obtained based on the depth map, surface normal map, and opacity map from each viewpoint.

6. The three-dimensional reconstruction method based on multi-view normal aggregation according to claim 1, characterized in that, The step of evaluating the 3D reconstruction results and optimizing the preset network based on the evaluation results includes: Calculate the chamfer distance and shape score between the point cloud of the 3D reconstruction result and the point cloud of the real shape corresponding to the target object; The parameters of the preset network are optimized based on the chamfer distance and shape score.

7. A three-dimensional reconstruction device based on multi-view normal convergence, characterized in that, The device includes: The acquisition module is used to acquire multi-view images of the target object under different lighting conditions; The acquisition module is used to acquire the normal image corresponding to the multi-view image, and to obtain the relative depth map of the normal image using the normal integration method; The reconstruction module is used to input the normal image and the relative depth map into a trained preset network for 3D reconstruction to obtain a 3D reconstruction result, wherein the preset network includes a pose parameter network and an SDF shape network. An evaluation module is used to evaluate the 3D reconstruction results and optimize the preset network based on the evaluation results; The training method for the preset network includes: Construct a training dataset that includes both synthetic and real data; Design joint optimization loss functions, including normal loss, integral loss, multi-view consistency loss, mask loss, and isometric loss; The pose parameter network and the SDF shape network are jointly trained end-to-end using the gradient descent method; Training is stopped when the loss function converges and the preset number of training iterations is reached. The loss function for the joint optimization is: in, These are coefficients that balance the different loss terms. Indicates normal loss. Indicates the integral loss. This represents the loss of consistency across multiple views. Indicates equidistant loss. This indicates the loss of the mask.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the three-dimensional reconstruction method based on multi-view normal aggregation as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the three-dimensional reconstruction method based on multi-view normal aggregation as described in any one of claims 1-6.

Images