Three-dimensional reconstruction method, device, storage medium and program product

Abstract

The present disclosure relates to a three-dimensional reconstruction method, device, storage medium and program product. The method comprises: obtaining a set of reference image pairs of a target scene; obtaining a point graph corresponding to each reference image pair in the set of reference image pairs through a preset three-dimensional reconstruction model; constructing a depth scale consistency optimization function corresponding to the set of reference image pairs according to the point graph corresponding to each reference image pair and a depth scale adjustment parameter; performing eigenvalue decomposition on the depth scale consistency optimization function to obtain a reduced depth scale consistency optimization function corresponding to the set of reference image pairs; obtaining a depth scale adjustment parameter value corresponding to each reference image pair by minimizing the reduced depth scale consistency optimization function; and generating a three-dimensional model corresponding to the target scene based on the point graph corresponding to each reference image pair and the depth scale adjustment parameter value. The present disclosure can reduce the memory and computing power overhead.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a three-dimensional reconstruction method, a three-dimensional reconstruction apparatus, a non-volatile computer-readable storage medium, and a computer program product. Background Technology

[0002] With the rapid development of computer vision and 3D reconstruction technologies, 3D scene reconstruction has been widely used in fields such as virtual reality, augmented reality, autonomous driving, and robot navigation. Although traditional methods can generate high-precision 3D models, they often face problems such as high computational complexity and large memory requirements when dealing with large-scale scenes, which limits their widespread application in practice.

[0003] In recent years, deep learning-based 3D reconstruction methods have gradually become a research hotspot. These methods estimate depth information directly from images using neural networks, thereby generating 3D models. Among them, the DUSt3R (Dense and Unconstrained Stereo 3D Reconstruction) model is a deep learning-based 3D reconstruction model. It takes two frames of RGB (Red, Green, Blue) images as input and outputs the 3D coordinates of each pixel in each frame, along with the corresponding confidence score. The advantage of DUSt3R lies in its ability to generate relatively accurate depth maps through global optimization methods, making it suitable for the reconstruction of small-scale scenes.

[0004] However, existing DUSt3R methods have significant limitations when handling large-scale scenes. Specifically, DUSt3R's global optimization method requires processing a large number of image frames simultaneously, leading to a sharp increase in the demand for video memory and computing power. For example, when processing 1000 frames of images, the video memory requirement may reach several hundred GB, which is impractical in real-world applications. Summary of the Invention

[0005] In view of this, this disclosure provides a three-dimensional reconstruction technical solution.

[0006] According to one aspect of this disclosure, a three-dimensional reconstruction method is provided, comprising:

[0007] Obtain a set of reference image pairs for the target scene, wherein any reference image pair includes two reference images;

[0008] The point map corresponding to each reference image pair in the reference image pair set is obtained by using a preset three-dimensional reconstruction model;

[0009] Based on the corresponding point map and depth scale adjustment parameters of each reference image pair, construct the depth scale consistency optimization function corresponding to the set of reference image pairs;

[0010] The depth scale consistency optimization function is decomposed into eigenvalues ​​to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set.

[0011] By minimizing the depth scale consistency optimization function of the dimensionality reduction, the depth scale adjustment parameter values ​​corresponding to each reference image pair are obtained;

[0012] Based on the reference images, the corresponding point map and depth scale parameter values ​​are adjusted to generate a 3D model corresponding to the target scene.

[0013] In one possible implementation, the three-dimensional reconstruction model is a dense, unconstrained stereoscopic three-dimensional reconstruction model.

[0014] In one possible implementation, the reference image pair satisfies at least two of the following conditions:

[0015] The number of keypoint matches between two reference images is greater than or equal to a preset number;

[0016] The difference in rotation angle between the two reference images is within a preset angle range;

[0017] The ratio of the z-axis component of the translation vector between two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

[0018] In one possible implementation, performing eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimensionality-reduced depth scale consistency optimization function corresponding to the reference image pair set includes:

[0019] Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain the equivalent point coordinates corresponding to the point map;

[0020] Based on the equivalent point coordinates corresponding to the point map, the depth scale consistency optimization function for the dimensionality reduction of the reference image set is obtained.

[0021] In one possible implementation,

[0022] The method further includes: obtaining confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model;

[0023] The step of constructing the depth scale consistency optimization function corresponding to the reference image pair set based on the corresponding point map and depth scale adjustment parameters of each reference image pair includes: constructing the depth scale consistency optimization function corresponding to the reference image pair set based on the corresponding point map, confidence map and depth scale adjustment parameters of each reference image pair.

[0024] In one possible implementation, the reference image pair includes a first reference image and a second reference image;

[0025] The point map and confidence map corresponding to each reference image pair in the reference image pair set are obtained through a preset 3D reconstruction model, including:

[0026] The reference image pair is input into a preset 3D reconstruction model in the order of the first reference image to the second reference image. The 3D reconstruction model outputs a first point map and a first confidence map corresponding to the reference image pair. The first point map represents the 3D point coordinates of each pixel in the first reference image in the camera coordinate system of the first reference image.

[0027] The reference image pair is input into the 3D reconstruction model in the order from the second reference image to the first reference image. The 3D reconstruction model outputs a second point map and a second confidence map corresponding to the reference image pair. The second point map represents the 3D point coordinates of each pixel in the second reference image in the camera coordinate system of the second reference image.

[0028] In one possible implementation,

[0029] Before inputting the reference image pair into a preset 3D reconstruction model in the order of the first reference image to the second reference image, the method further includes: performing distortion correction and rescaling on the first reference image and the second reference image to obtain a preprocessed reference image pair;

[0030] The step of inputting the reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image includes: inputting the preprocessed reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image;

[0031] The step of inputting the reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image includes: inputting the preprocessed reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image.

[0032] In one possible implementation, the step of generating a 3D model of the target scene by adjusting the parameter values ​​of the corresponding point map and depth scale based on each reference image includes:

[0033] Based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair, the optimized depth map corresponding to each reference image pair is determined.

[0034] Based on the optimized depth maps corresponding to each reference image pair, a 3D model corresponding to the target scene is generated.

[0035] In one possible implementation, determining the optimized depth map corresponding to each reference image pair based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair includes:

[0036] By using an assisted 3D reconstruction model, a set of 3D key points corresponding to each reference image pair is generated;

[0037] Based on the point map, confidence map, depth scale adjustment parameter value, 3D key point set and correction parameter of each reference image pair, construct the joint optimization function corresponding to the reference image pair set;

[0038] By minimizing the joint optimization function, the corresponding correction parameter values ​​for each reference image pair are obtained;

[0039] Based on the corresponding point map, depth scale adjustment parameter value, and correction parameter value of each reference image pair, the corresponding optimized depth map is determined.

[0040] In one possible implementation, generating a 3D model of the target scene based on the optimized depth maps corresponding to each reference image pair includes:

[0041] By fusing the optimized depth maps corresponding to each reference image pair using a truncated symbolic distance function, a 3D model corresponding to the target scene is obtained.

[0042] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, comprising:

[0043] The first acquisition module is used to acquire a set of reference image pairs for the target scene, wherein any reference image pair includes two reference images;

[0044] The second acquisition module is used to obtain the point map corresponding to each reference image pair in the reference image pair set through a preset three-dimensional reconstruction model;

[0045] The construction module is used to construct the depth scale consistency optimization function corresponding to the set of reference image pairs by adjusting the parameters of the corresponding point map and depth scale according to each reference image pair.

[0046] The eigenvalue decomposition module is used to perform eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set.

[0047] The minimization module is used to obtain the corresponding depth scale adjustment parameter values ​​for each reference image pair by minimizing the depth scale consistency optimization function of the dimensionality reduction.

[0048] The generation module is used to adjust the parameter values ​​of the corresponding point map and depth scale based on each reference image to generate a three-dimensional model corresponding to the target scene.

[0049] In one possible implementation, the three-dimensional reconstruction model is a dense, unconstrained stereoscopic three-dimensional reconstruction model.

[0050] In one possible implementation, the reference image pair satisfies at least two of the following conditions:

[0051] The number of keypoint matches between two reference images is greater than or equal to a preset number;

[0052] The difference in rotation angle between the two reference images is within a preset angle range;

[0053] The ratio of the z-axis component of the translation vector between two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

[0054] In one possible implementation, the eigenvalue decomposition module is used for:

[0055] Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain the equivalent point coordinates corresponding to the point map;

[0056] Based on the equivalent point coordinates corresponding to the point map, the depth scale consistency optimization function for the dimensionality reduction of the reference image set is obtained.

[0057] In one possible implementation,

[0058] The second obtaining module is further configured to: obtain the confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model;

[0059] The construction module is used to: construct a depth scale consistency optimization function corresponding to the set of reference image pairs based on the corresponding point map, confidence map and depth scale adjustment parameters of each reference image pair.

[0060] In one possible implementation, the reference image pair includes a first reference image and a second reference image;

[0061] The second obtaining module is used for:

[0062] The reference image pair is input into a preset 3D reconstruction model in the order of the first reference image to the second reference image. The 3D reconstruction model outputs a first point map and a first confidence map corresponding to the reference image pair. The first point map represents the 3D point coordinates of each pixel in the first reference image in the camera coordinate system of the first reference image.

[0063] The reference image pair is input into the 3D reconstruction model in the order from the second reference image to the first reference image. The 3D reconstruction model outputs a second point map and a second confidence map corresponding to the reference image pair. The second point map represents the 3D point coordinates of each pixel in the second reference image in the camera coordinate system of the second reference image.

[0064] In one possible implementation,

[0065] The device further includes: a preprocessing module, used to perform distortion correction and rescaling on the first reference image and the second reference image to obtain a preprocessed reference image pair;

[0066] The second obtaining module is used to: input the preprocessed reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image; and input the preprocessed reference image pair into the three-dimensional reconstruction model in the order of the second reference image to the first reference image.

[0067] In one possible implementation, the generation module is used to:

[0068] Based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair, the optimized depth map corresponding to each reference image pair is determined.

[0069] Based on the optimized depth maps corresponding to each reference image pair, a 3D model corresponding to the target scene is generated.

[0070] In one possible implementation, the generation module is used to:

[0071] By using an assisted 3D reconstruction model, a set of 3D key points corresponding to each reference image pair is generated;

[0072] Based on the point map, confidence map, depth scale adjustment parameter value, 3D key point set and correction parameter of each reference image pair, construct the joint optimization function corresponding to the reference image pair set;

[0073] By minimizing the joint optimization function, the corresponding correction parameter values ​​for each reference image pair are obtained;

[0074] Based on the corresponding point map, depth scale adjustment parameter value, and correction parameter value of each reference image pair, the corresponding optimized depth map is determined.

[0075] In one possible implementation, the generation module is used to:

[0076] By fusing the optimized depth maps corresponding to each reference image pair using a truncated symbolic distance function, a 3D model corresponding to the target scene is obtained.

[0077] According to another aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0078] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0079] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0080] In this embodiment, a set of reference image pairs for the target scene is obtained, wherein each reference image pair includes two reference images. A preset 3D reconstruction model is used to obtain the point map corresponding to each reference image pair in the set. Based on the point map and depth scale adjustment parameters of each reference image pair, a depth scale consistency optimization function is constructed for the set of reference image pairs. Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain a dimensionality-reduced depth scale consistency optimization function for the set of reference image pairs. By minimizing the dimensionality-reduced depth scale consistency optimization function, the depth scale adjustment parameter values ​​for each reference image pair are obtained. Based on the point map and depth scale adjustment parameter values ​​of each reference image pair, a 3D model corresponding to the target scene is generated. Thus, by introducing a depth scale consistency optimization function and performing eigenvalue decomposition and dimensionality reduction, the overhead of video memory and computing power is significantly reduced while ensuring reconstruction accuracy. Related technologies, when processing large-scale scenes, require processing a large number of image frames simultaneously, leading to a sharp increase in video memory and computing power demands, making them difficult to implement in practical applications. This disclosure reduces computational complexity and memory usage during optimization by constructing a depth scale consistency optimization function and performing dimensionality reduction on it, making 3D reconstruction of large-scale scenes possible. Furthermore, by minimizing the dimensionality-reduced optimization function, the depth scale parameters of each reference image pair can be effectively adjusted, ensuring the consistency of depth information across multiple frames. This allows for the generation of high-precision 3D models while reducing computational resource consumption. This disclosure is not only applicable to small-scale scenes but can also be extended to the reconstruction of large-scale scenes, demonstrating broad application prospects.

[0081] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0082] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0083] Figure 1 A flowchart illustrating the three-dimensional reconstruction method provided in an embodiment of this disclosure is shown.

[0084] Figure 2 A block diagram of a three-dimensional reconstruction apparatus provided in an embodiment of this disclosure is shown.

[0085] Figure 3 This is a block diagram illustrating a three-dimensional reconstruction apparatus 1900 according to an exemplary embodiment. Detailed Implementation

[0086] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0087] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0088] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0089] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0090] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0091] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0092] As mentioned above, while the DUSt3R (Dense and Unconstrained Stereo 3D Reconstruction) method in related technologies performs well in small-scale scenes, its application scope is limited by the high demand for video memory and computing power when dealing with large-scale scenes.

[0093] To address the technical problems described above, this disclosure provides a 3D reconstruction method. This method obtains a set of reference image pairs for a target scene, where each reference image pair includes two reference images. A preset 3D reconstruction model is used to obtain the point maps corresponding to each reference image pair in the set. Based on the point maps and depth scale adjustment parameters of each reference image pair, a depth scale consistency optimization function is constructed for the set of reference image pairs. Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain a dimensionality-reduced depth scale consistency optimization function for the set of reference image pairs. By minimizing the dimensionality-reduced depth scale consistency optimization function, the depth scale adjustment parameter values ​​for each reference image pair are obtained. Based on the point maps and depth scale adjustment parameter values ​​of each reference image pair, a 3D model corresponding to the target scene is generated. By introducing a depth scale consistency optimization function and performing eigenvalue decomposition and dimensionality reduction, the overhead of GPU memory and computing power is significantly reduced while maintaining reconstruction accuracy. Related technologies, when processing large-scale scenes, require processing a large number of image frames simultaneously, leading to a sharp increase in GPU memory and computing power demands, making them difficult to implement in practical applications. This disclosure reduces computational complexity and memory usage during optimization by constructing a depth scale consistency optimization function and performing dimensionality reduction on it, making 3D reconstruction of large-scale scenes possible. Furthermore, by minimizing the dimensionality-reduced optimization function, the depth scale parameters of each reference image pair can be effectively adjusted, ensuring the consistency of depth information across multiple frames. This allows for the generation of high-precision 3D models while reducing computational resource consumption. This disclosure is not only applicable to small-scale scenes but can also be extended to the reconstruction of large-scale scenes, demonstrating broad application prospects.

[0094] The three-dimensional reconstruction method provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0095] Figure 1 A flowchart illustrating a three-dimensional reconstruction method provided in an embodiment of this disclosure is shown. In one possible implementation, the executor of the three-dimensional reconstruction method can be a three-dimensional reconstruction device, for example, the three-dimensional reconstruction method can be executed by a terminal device, a server, or other electronic equipment. The terminal device can be user equipment (UE), a user terminal, a terminal, or a computing device, etc. In some possible implementations, the three-dimensional reconstruction method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the three-dimensional reconstruction method includes steps S11 to S16.

[0096] In step S11, a set of reference image pairs for the target scene is obtained, wherein any reference image pair includes two reference images.

[0097] In step S12, the point map corresponding to each reference image pair in the reference image pair set is obtained through a preset three-dimensional reconstruction model.

[0098] In step S13, a depth scale consistency optimization function is constructed based on the corresponding point map and depth scale adjustment parameters of each reference image pair.

[0099] In step S14, the depth scale consistency optimization function is decomposed into eigenvalues ​​to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set.

[0100] In step S15, the depth scale adjustment parameter values ​​corresponding to each reference image pair are obtained by minimizing the depth scale consistency optimization function of the dimensionality reduction.

[0101] In step S16, the parameter values ​​of the corresponding point map and depth scale are adjusted based on the reference images to generate a three-dimensional model corresponding to the target scene.

[0102] In this embodiment of the disclosure, the target scene can represent a physical space environment that requires 3D reconstruction or the generation of a new perspective. The target scene can consist of a series of 3D objects, surfaces, and backgrounds, and can be captured and reconstructed using multi-view image data. The target scene can be an indoor environment (such as a room, office, studio, etc.) or an outdoor environment (such as a street, building exterior, natural landscape, etc.).

[0103] A reference image of a target scene can represent an image taken from a specific perspective that contains information about the target scene. The reference image can be used to generate a 3D model corresponding to the target scene. In some applications, the reference image may also be called a training image, ground truth image, etc., without further elaboration here.

[0104] A reference image pair refers to two images taken from two different perspectives that contain information about the same target scene. The two images in a reference image pair can be called the first reference image and the second reference image. In one example, i and j can be used to represent the frame numbers of the first and second reference images.

[0105] In the embodiments of this disclosure, selecting appropriate reference images is crucial for generating a 3D model corresponding to the target scene, as they need to provide sufficient viewpoint differences and sufficient overlapping areas to obtain the 3D structure and depth information of the target scene.

[0106] In one possible implementation, the reference image pair satisfies at least two of the following conditions: the number of key point matches between the two reference images is greater than or equal to a preset number; the difference in rotation angle between the two reference images is within a preset angle range; and the ratio of the z-axis component of the translation vector between the two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

[0107] As an example of this implementation, the reference image pair satisfies the following condition: the number of keypoint matches between the two reference images is greater than or equal to a preset number. In one example, the preset number can be 30. In this example, keypoint matching can refer to finding corresponding feature points (such as corner points, edge points, etc.) in the two reference images. These matched points provide the geometric relationship between the two reference images, helping the 3D reconstruction model extract the 3D structure of the target scene. If the number of matched keypoints is large enough, it indicates that there is sufficient overlap between the two reference images, and the 3D reconstruction model can more accurately reconstruct the geometric information of the target scene. Therefore, by adopting this example, it is possible to ensure that there is sufficient overlap between the two reference images so that the 3D reconstruction model can extract the 3D structure of the target scene.

[0108] As an example of this implementation, the reference image pair satisfies the following condition: the difference in rotation angle between the two reference images is within a preset angle range. The difference in rotation angle can refer to the difference in the shooting perspective of the two reference images in the rotation direction. In one example, the preset angle range can be [16°, 60°]. In this example, if the difference in rotation angle between the two reference images is between 16° and 60°, it indicates that the perspectives of the two reference images have a certain difference, but not too large. This moderate difference in perspective helps the 3D reconstruction model extract depth information of the target scene. This example ensures that the difference in perspective between the two reference images is moderate, neither lacking sufficient depth information due to an excessively small difference nor lacking sufficient overlap between the reference images due to an excessively large difference.

[0109] As an example of this implementation, the reference image pair satisfies the following condition: the ratio of the z-axis component of the translation vector between the two reference images to the magnitude of the translation vector is less than or equal to a preset ratio. Here, the translation vector describes the relative positional change of the two reference images in space. The z-axis component can be related to the depth direction, while the magnitude of the translation vector represents the total displacement between the two reference images. The ratio in this example can be used to limit the relative motion between the two reference images, avoiding situations where the two reference images in the same reference image pair only move back and forth. For example, the preset ratio can be 0.95. If the ratio of the z-axis component of the translation vector between the two reference images to the magnitude of the translation vector is less than or equal to 0.95, it indicates that the motion between the two reference images is not a pure back-and-forth movement. By adopting this example, it is possible to avoid selecting reference image pairs with too small a viewpoint difference, ensuring sufficient viewpoint difference between the reference image pairs, thereby improving the 3D reconstruction effect.

[0110] By adopting this implementation method, suitable reference image pairs can be selected so that the 3D reconstruction model can more effectively extract the 3D structure and depth information of the target scene, thereby improving the 3D reconstruction effect.

[0111] In this embodiment of the disclosure, pointmaps corresponding to each reference image pair in the set of reference image pairs can be obtained through a preset 3D reconstruction model. That is, the input reference image pairs can be processed using a preset 3D reconstruction model, and pointmaps corresponding to each reference image pair can be generated. Here, a pointmap is the coordinate representation of each pixel in the reference image in 3D space, i.e., the 3D point coordinates corresponding to each pixel. A pointmap can be a 3D matrix of the same size as the input reference image, where each pixel location stores the coordinates (x, y, z) of the pixel in the reference image in 3D space. Through pointmaps, pixels in a 2D image can be mapped to 3D space, thereby providing basic data for subsequent 3D reconstruction.

[0112] In one possible implementation, the three-dimensional reconstruction model is a dense, unconstrained stereoscopic three-dimensional reconstruction model.

[0113] DUSt3R is an advanced 3D reconstruction technology that can handle the problem of insufficient geometric restoration in weakly textured areas (such as white walls and uniformly lit surfaces). DUSt3R can accept two RGB (Red, Green, Blue) images as input. 1 ,I 2 Output two frames of dot plot X 1,1 ,X 2,1 And the corresponding confidence plot C 1,1 C 2,1 Among them, point graph X 1,1 Image I1 Each pixel in I 1 The corresponding 3D point coordinates in the camera coordinate system, point image X 2,1 Image I 2 The 3D point coordinates corresponding to the pixels in the image are converted to I. 1 The coordinates in the camera coordinate system. Confidence plot C 1,1 C 2,1 Used to measure the reliability of depth estimation. DUSt3R's advantages lie in its efficient point map regression strategy and the fact that it does not require camera calibration, making 3D reconstruction more flexible and efficient.

[0114] Besides DUSt3R, 3D reconstruction models can also employ learning-based methods (such as MVSNet and MonoDepth2), traditional MVS (Multi-View Stereo) methods (such as COLMAP and OpenMVS), hybrid methods (such as HybridMVSNet), sensor-based methods (such as Kinect and LiDAR), or optimization-based methods (such as BundleFusion). The choice of method depends on the specific application scenario, data type, and performance requirements.

[0115] In one possible implementation, the method further includes: obtaining confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model; and constructing a depth scale consistency optimization function corresponding to the set of reference image pairs based on the point maps and depth scale adjustment parameters corresponding to each reference image pair, which includes: constructing a depth scale consistency optimization function corresponding to the set of reference image pairs based on the point maps, confidence maps, and depth scale adjustment parameters corresponding to each reference image pair.

[0116] In this implementation, the confidence map can be a two-dimensional matrix with the same size as the input reference image, where the value at each pixel location represents the confidence level of the depth estimate for that pixel.

[0117] This implementation, by introducing a confidence map, can more accurately adjust the depth scale across multiple frames, ensuring global consistency of the 3D reconstruction results. The confidence map plays a weighted role in the optimization process, making more reliable depth estimates have a greater impact on the results, thereby improving the accuracy and robustness of 3D reconstruction.

[0118] In one possible implementation, the reference image pair includes a first reference image and a second reference image; obtaining the point map and confidence map corresponding to each reference image pair in the reference image pair set through a preset 3D reconstruction model includes: inputting the reference image pair into the preset 3D reconstruction model in the order from the first reference image to the second reference image, and outputting a first point map and a first confidence map corresponding to the reference image pair through the 3D reconstruction model, wherein the first point map represents the 3D point coordinates corresponding to each pixel in the first reference image in the camera coordinate system of the first reference image; inputting the reference image pair into the 3D reconstruction model in the order from the second reference image to the first reference image, and outputting a second point map and a second confidence map corresponding to the reference image pair through the 3D reconstruction model, wherein the second point map represents the 3D point coordinates corresponding to each pixel in the second reference image in the camera coordinate system of the second reference image.

[0119] For example, the frame number of the first reference image in a reference image pair is i, and the frame number of the second reference image is j. The reference image pair can be input into a preset 3D reconstruction model in the order (i, j), and the 3D reconstruction model outputs the first point map corresponding to the reference image pair. And the first confidence plot Among them, the first confidence graph It can represent the first point diagram The confidence level of each pixel value in the model. Reference image pairs can be input into a preset 3D reconstruction model in (j,i) order, and the 3D reconstruction model outputs the second point map corresponding to the reference image pair. Second confidence plot Second confidence plot This can represent the second point diagram. The confidence level of the pixel value for each pixel in the dataset.

[0120] For the reference image pair (i,j), the output of the DUSt3R model can be expressed as: Furthermore, it can be adopted Represents a set of reference images The DUSt3R output results for all reference image pairs.

[0121] The first point image can be extracted. The first depth map in And the second point diagram The second depth map Among them, the first depth map The second depth map can represent the predicted depth value of each pixel in the first reference image. This can represent the predicted depth value for each pixel in the second reference image. It can also be used to represent the first depth map. and the second depth map Depth optimization is performed separately to obtain the first optimized depth map and the second optimized depth map.

[0122] In one possible implementation, before inputting the reference image pair into a preset 3D reconstruction model in the order of the first reference image to the second reference image, the method further includes: performing distortion correction and rescaling on the first reference image and the second reference image to obtain a preprocessed reference image pair; inputting the reference image pair into the preset 3D reconstruction model in the order of the first reference image to the second reference image includes: inputting the preprocessed reference image pair into the preset 3D reconstruction model in the order of the first reference image to the second reference image; inputting the reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image includes: inputting the preprocessed reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image.

[0123] In this implementation, to ensure the input image meets the requirements of the 3D reconstruction model (such as DUSt3R), the reference image pair needs to be preprocessed. Preprocessing can include two main steps: distortion correction and rescaling. Distortion correction corrects camera lens distortion and improves the geometric quality of the reference image; rescaling adjusts the reference image to a suitable size, ensuring its dimensions meet the requirements (e.g., DUSt3R requires the long side of the input image to be 512 pixels) to satisfy the input requirements of the 3D reconstruction model. The preprocessing function can employ... This indicates that the preprocessed reference image can be used The symbol represents the reference image.

[0124] The preprocessed reference image pairs can be input into the 3D reconstruction model in the order of first reference image to second reference image (R(I1), R(I2)) and second reference image to first reference image (R(I2), R(I1)). The 3D reconstruction model can output the first point map and the first confidence map, as well as the second point map and the second confidence map, respectively.

[0125] This preprocessing method significantly improves the accuracy and efficiency of 3D reconstruction. Distortion correction corrects lens distortion and improves image geometric quality, while rescaling ensures the input image meets the size requirements of the 3D reconstruction model, thus improving its processing efficiency. This approach not only enhances the accuracy of 3D reconstruction but also strengthens the robustness of the 3D reconstruction model in complex scenes.

[0126] In this embodiment of the disclosure, a depth scale consistency optimization function corresponding to the set of reference image pairs can be constructed based on the corresponding point map and depth scale adjustment parameters of each reference image pair. In one possible implementation, the depth scale consistency optimization function corresponding to the set of reference image pairs can be constructed based on the corresponding point map, confidence map, and depth scale adjustment parameters of each reference image pair.

[0127] Depth maps output by 3D reconstruction models (such as DUST3R) often exhibit scale inconsistencies across different views. This is because the depth map for each view is estimated independently, lacking global consistency. In this embodiment, to ensure global consistency of depth maps across different views, a depth scale consistency optimization function is constructed based on the corresponding point map and depth scale adjustment parameters of each reference image pair, to perform scale adjustment on the depth map.

[0128] Take the DUSt3R model as an example. The DUSt3R model can only process two reference images at a time, and the depth scale of the calculation results is inconsistent each time, that is, it lacks global consistency. Let the depth scale adjustment parameter corresponding to the reference image pair (i,j) be s. i,j In the embodiments of this disclosure, the reference image pair set can be optimized. All reference images are adjusted for their corresponding depth scale parameters. To achieve global consistency. Assume there are N reference images in the reference image set, and the set of camera poses for these N reference images estimated by the auxiliary 3D reconstruction model (e.g., COLMAP) is: The set of world coordinate point maps for N frames of reference images is as follows: Among them, X i,W Let represent the world coordinate point map corresponding to the i-th frame reference image. Then, the depth scale consistency optimization function can be constructed as follows:

[0129]

[0130] in, ⊙ represents element-wise multiplication. In the above depth-scale consistency optimization function, X can be considered to be expressed as homogeneous coordinates, and the fourth dimension of the homogeneous coordinates can be ignored during minimization. Elimination is achieved through substitution. An equivalent depth-scale consistency optimization function can be obtained:

[0131]

[0132] in,

[0133] The above formula needs to be maintained in video memory. This is still too expensive.

[0134] In one possible implementation, the step of performing eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimensionality-reduced depth scale consistency optimization function corresponding to the reference image pair set includes: performing eigenvalue decomposition on the depth scale consistency optimization function to obtain the equivalent point coordinates corresponding to the point map; and obtaining the dimensionality-reduced depth scale consistency optimization function corresponding to the reference image pair set based on the equivalent point coordinates corresponding to the point map.

[0135] In this implementation, the depth scale consistency optimization function can be decomposed using eigenvalue decomposition to obtain the dimensionality-reduced depth scale consistency optimization function corresponding to the reference image pair set. That is, this implementation proposes an equivalent point dimensionality reduction strategy based on eigenvalue decomposition, which equivalences the point matching problem of the reference image pair (i,j) to 7 points. and These represent the equivalent point coordinates of the viewpoint corresponding to the first reference image (i.e., the equivalent point coordinates corresponding to the first scatter plot) and the equivalent point coordinates of the viewpoint corresponding to the second reference image (i.e., the equivalent point coordinates corresponding to the second scatter plot), respectively. This represents the corresponding weight vector. The depth-scale consistency optimization function can then be equivalently transformed into:

[0136]

[0137] in, w represents the set of depth scale adjustment parameters. i,j For the weights calculated based on the confidence plot, T i and T j For reference image I i and reference image I j Camera pose, x i,j and y i,j These are the equivalent point coordinates obtained through eigenvalue decomposition.

[0138] The equivalence proof is as follows: For the general weighted point set matching problem, the formula can be expressed as follows: in and It can represent any three-dimensional affine transformation. Indicate the source point and the target point. Let b = b1 - b2, then the formula can be equivalently expressed as:

[0139]

[0140] in, The result of eigenvalue decomposition, This can be represented as a point matching form: ||w⊙([A1,-A2,b][x,y,1]) T )||, where w=ξV [:7] x = V [:1-3] / V [:7] y = V [:4-6] / V [:7] V [:.] This represents the column corresponding to V.

[0141] Optimize function based on depth scale consistency The depth scale adjustment parameter S can be optimized to obtain the result. in, because Let z represent the z-coordinate of the first point graph, and Located in the camera coordinate system, therefore This can be equivalent to a depth map. Similarly, we can obtain d. j,i and w j,i The first optimized depth map corresponding to the first reference image can be represented by d. i,j This indicates that the second optimized depth map corresponding to the second reference image can be expressed as d. j,i This indicates that the first confidence plot can be achieved using w. i,j This indicates that the second confidence plot can be achieved using w. j,i express.

[0142] In this embodiment, the depth scale adjustment parameter values ​​for each reference image pair can be obtained by minimizing the dimensionality-reduced depth scale consistency optimization function. During 3D reconstruction, the depth scale consistency optimization function is used to adjust the depth scale differences between different reference image pairs to ensure the consistency of global depth information. By minimizing the dimensionality-reduced depth scale consistency optimization function, the optimal depth scale adjustment parameter values ​​can be found. This is because the minimization process is essentially finding a set of parameters that minimizes the value of the depth scale consistency optimization function, i.e., under the constraint of depth scale consistency, makes the depth information of each reference image pair most coordinated and unified globally. By minimizing the depth scale consistency function, the depth scale parameters of each reference image pair can be effectively adjusted, thereby ensuring that the depth maps of different views are globally consistent, ultimately providing accurate depth information for generating a high-precision 3D model.

[0143] In one possible implementation, generating a 3D model of the target scene based on the corresponding point map and depth scale adjustment parameter values ​​of each reference image pair includes: determining an optimized depth map corresponding to each reference image pair based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair; and generating a 3D model of the target scene based on the optimized depth map corresponding to each reference image pair.

[0144] In this implementation, parameter values ​​can be adjusted according to the corresponding point map, confidence map, and depth scale of each reference image pair to optimize the depth map in the corresponding point map of each reference image pair, thereby obtaining the optimized depth map of each reference image pair. Based on the optimized depth map of each reference image pair, a 3D model corresponding to the target scene can be generated.

[0145] In one possible implementation, determining the optimized depth map corresponding to each reference image pair based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​includes: generating a set of 3D keypoints corresponding to each reference image pair using an auxiliary 3D reconstruction model; constructing a joint optimization function corresponding to the reference image pair set based on the corresponding point map, confidence map, depth scale adjustment parameter values, 3D keypoint set, and correction parameters; obtaining the correction parameter values ​​corresponding to each reference image pair by minimizing the joint optimization function; and determining the optimized depth map corresponding to each reference image pair based on the corresponding point map, depth scale adjustment parameter values, and correction parameter values.

[0146] In this implementation, the assisted 3D reconstruction model and the 3D reconstruction model are two different models. The assisted 3D reconstruction model can use COLMAP or other multi-view stereo methods; no limitation is made here. The assisted 3D reconstruction model can provide sparse 3D keypoints, that is, the set of 3D keypoints corresponding to the reference image pair.

[0147] In this implementation, an auxiliary 3D reconstruction model can be used to generate a first 3D keypoint set corresponding to the first reference image and a second 3D keypoint set corresponding to the second reference image in each reference image pair set. The first 3D keypoint set represents the position of keypoints in the first reference image in 3D space, and the second 3D keypoint set represents the position of keypoints in the second reference image in 3D space. After obtaining the 3D keypoint sets, they can be projected onto the camera coordinate system of the corresponding reference image. For example, for the i-th frame, the coordinates and depth of the 3D points projected onto the image plane of the i-th frame can be represented as follows: in, This represents the image coordinates of the k-th keypoint. This represents the depth value of the k-th keypoint in the camera coordinate system of the i-th frame.

[0148] The generated first and second 3D keypoint sets can be used to optimize the corresponding first and second depth maps of each reference image. The optimization objective can include adjusting the depth values ​​in the depth maps to more closely approximate the geometry of the real scene, based on the geometric constraints provided by the 3D keypoint sets.

[0149] As an example of this implementation, a learnable depth correction function can be introduced:

[0150] φ(d;r,l)=d r +l·d

[0151] Where d can represent the depth map in the point map output by DUSt3R, and r and l can represent correction parameters used to adjust the accuracy of the depth map.

[0152] In this implementation, the first and second 3D keypoint sets can provide geometric constraints for the depth map, helping the optimization process to better match depth information across different views. The first and second confidence maps can represent the reliability of the depth estimate for each pixel. During optimization, the confidence map can be used as a weight, making the optimization process focus more on regions with high confidence. The first and second depth scale adjustment parameters can be used to adjust the scale of the depth map, ensuring that the depth maps across different views are globally consistent. Correction parameters can be used to further adjust the accuracy and consistency of the depth map, such as geometric transformation parameters like translation and rotation.

[0153] Based on the input information above, a joint optimization function can be constructed. The objective of the joint optimization function can include minimizing the difference between the depth map and the keypoint set, while considering the constraints of the confidence map and depth scale adjustment parameters. The joint optimization function can be a weighted loss function.

[0154] In one example, the accuracy of the depth map in the point map can be adjusted by optimizing the following joint optimization function:

[0155]

[0156] in,

[0157] By minimizing the joint optimization function, the optimal values ​​of the corresponding correction parameters for each reference image pair can be obtained. Based on the optimized correction parameters, the first depth map can be adjusted to obtain a first optimized depth map. Similarly, based on the optimized correction parameters, the second depth map can be adjusted to obtain a second optimized depth map. In one example, the first optimized depth map can be... This indicates that the second optimized depth map can be adopted. express.

[0158] This implementation combines a 3D keypoint set, a confidence map, depth scale adjustment parameters, and correction parameters to construct a joint optimization function, thereby optimizing the depth map within the point map. The optimized depth map more accurately reflects the geometry of the target scene, improving its accuracy and robustness, and providing high-quality input for subsequent 3D reconstruction.

[0159] In one possible implementation, the first depth map and the second depth map corresponding to each reference image pair can be optimized based on the corresponding correction parameter values ​​and the first confidence map and the second confidence map corresponding to each reference image pair, to obtain the first optimized depth map and the second optimized depth map corresponding to each reference image pair.

[0160] In one example, for a reference image pair (i,j), the optimized depth map can be represented as: Similarly, we can obtain For reference image pairs set After optimizing all reference image pairs, we can obtain:

[0161] In one possible implementation, generating a 3D model of the target scene based on the optimized depth maps corresponding to each of the reference image pairs includes: fusing the optimized depth maps corresponding to each of the reference image pairs using a truncated signed distance function (TSDF) to obtain the 3D model of the target scene.

[0162] The Signed Distance Function (SDF) represents the distance from each point in space to the nearest object surface. A positive distance indicates that the point is outside the object, and a negative distance indicates that the point is inside the object. The truncated signed distance function is a variant of the SDF that only considers points within a certain range from the object surface (i.e., points within the truncation distance); distance values ​​outside this range are truncated. This truncation operation can reduce the influence of noise and improve the robustness of reconstruction.

[0163] By fusing optimized depth maps from various reference image pairs using a truncated signed distance function, depth information from multiple perspectives can be integrated into a unified voxel grid, generating a globally consistent 3D model. The truncation operation of the truncated signed distance function and the weighted average update strategy can effectively suppress noise and improve the robustness and accuracy of the reconstruction.

[0164] In one possible implementation, keypoints outside the boundary of the truncated symbolic distance function can be included in the initialization points to improve the reconstruction quality of distant backgrounds.

[0165] The three-dimensional reconstruction method provided in this disclosure can be applied to the fields of AI (Artificial Intelligence)-CV (Computer Vision)-3D (3 Dimensions) reconstruction, dense stereo model, DUSt3R, etc., and is not limited thereto.

[0166] The following describes the three-dimensional reconstruction method provided in this disclosure through a specific application scenario.

[0167] In this application scenario, a set of reference image pairs for the target scene can be obtained, wherein any reference image pair includes two reference images. Each reference image pair satisfies the following three conditions: the number of keypoint matches between the two reference images is greater than or equal to a preset number; the difference in rotation angle between the two reference images is within a preset angle range; and the ratio of the z-axis component of the translation vector between the two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

[0168] Distortion correction and rescaling can be performed on the first and second reference images to obtain a preprocessed reference image pair. The preprocessed reference image pair can be input into a DUBSt3R in the order of the first reference image to the second reference image, and the DUBSt3R will output the first dot map corresponding to the reference image pair. and the first confidence graph The reference image pair can be input into DUSt3R in the order of the second reference image to the first reference image, and DUSt3R can output the second point map corresponding to the reference image pair. and the second confidence plot The first point image can be extracted. The first depth map in And the second point diagram Second depth map

[0169] A depth scale consistency optimization function for the set of reference image pairs can be constructed based on the corresponding point map, confidence map, depth scale adjustment parameters, camera pose, and world coordinate point map for each reference image pair. in, The depth scale consistency optimization function can be decomposed into eigenvalues ​​to obtain the dimensionality-reduced depth scale consistency optimization function corresponding to the reference image pair set. The first depth scale adjustment parameter value s for each reference image pair can be obtained by minimizing the depth scale consistency optimization function of the dimensionality reduction. i,j Second depth scale adjustment parameter value s j,i .

[0170] The COLMAP assisted 3D reconstruction model can be used to generate 3D keypoint sets corresponding to each reference image pair. Based on the point map, confidence map, depth scale adjustment parameter values, 3D keypoint sets, and correction parameters corresponding to each reference image pair, a joint optimization function corresponding to the reference image pair set can be constructed. in, The joint optimization function yields the corresponding correction parameter values ​​r for each reference image pair. i,j ,r j,i ,l i,j ,l j,i Based on the corresponding point map, depth scale adjustment parameter values, and correction parameter values ​​for each reference image pair, an optimized depth map is determined for each reference image pair. For example, for reference image pair (i,j), the optimized depth map can be represented as:

[0171] The optimized depth maps corresponding to the various reference image pairs can be fused using the truncated symbolic distance function to obtain the 3D model corresponding to the target scene.

[0172] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0173] In addition, this disclosure also provides a three-dimensional reconstruction apparatus, a non-volatile computer-readable storage medium, and a computer program product, all of which can be used to implement any of the three-dimensional reconstruction methods provided in this disclosure. The corresponding technical solutions and effects can be found in the relevant descriptions in the method section, and will not be repeated here.

[0174] Figure 2 A block diagram of a three-dimensional reconstruction apparatus provided in an embodiment of this disclosure is shown. Figure 2 As shown, the three-dimensional reconstruction device includes:

[0175] The first obtaining module 21 is used to obtain a set of reference image pairs of the target scene, wherein any reference image pair includes two reference images;

[0176] The second acquisition module 22 is used to obtain the point map corresponding to each reference image pair in the reference image pair set through a preset three-dimensional reconstruction model;

[0177] Construction module 23 is used to construct the depth scale consistency optimization function corresponding to the set of reference images by adjusting the parameters of the corresponding point map and depth scale according to each reference image pair;

[0178] The eigenvalue decomposition module 24 is used to perform eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set.

[0179] Minimization module 25 is used to obtain the corresponding depth scale adjustment parameter values ​​for each reference image pair by minimizing the depth scale consistency optimization function of the dimensionality reduction.

[0180] The generation module 26 is used to adjust the parameter values ​​of the corresponding point map and depth scale based on each reference image to generate a three-dimensional model corresponding to the target scene.

[0181] In one possible implementation, the three-dimensional reconstruction model is a dense, unconstrained stereoscopic three-dimensional reconstruction model.

[0182] In one possible implementation, the reference image pair satisfies at least two of the following conditions:

[0183] The number of keypoint matches between two reference images is greater than or equal to a preset number;

[0184] The difference in rotation angle between the two reference images is within a preset angle range;

[0185] The ratio of the z-axis component of the translation vector between two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

[0186] In one possible implementation, the eigenvalue decomposition module 24 is used for:

[0187] Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain the equivalent point coordinates corresponding to the point map;

[0188] Based on the equivalent point coordinates corresponding to the point map, the depth scale consistency optimization function for the dimensionality reduction of the reference image set is obtained.

[0189] In one possible implementation,

[0190] The second obtaining module 22 is further configured to: obtain the confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model;

[0191] The construction module 23 is used to: construct a depth scale consistency optimization function corresponding to the set of reference image pairs based on the corresponding point map, confidence map and depth scale adjustment parameters of each reference image pair.

[0192] In one possible implementation, the reference image pair includes a first reference image and a second reference image;

[0193] The second obtaining module 22 is used for:

[0194] The reference image pair is input into a preset 3D reconstruction model in the order of the first reference image to the second reference image. The 3D reconstruction model outputs a first point map and a first confidence map corresponding to the reference image pair. The first point map represents the 3D point coordinates of each pixel in the first reference image in the camera coordinate system of the first reference image.

[0195] The reference image pair is input into the 3D reconstruction model in the order from the second reference image to the first reference image. The 3D reconstruction model outputs a second point map and a second confidence map corresponding to the reference image pair. The second point map represents the 3D point coordinates of each pixel in the second reference image in the camera coordinate system of the second reference image.

[0196] In one possible implementation,

[0197] The device further includes: a preprocessing module, used to perform distortion correction and rescaling on the first reference image and the second reference image to obtain a preprocessed reference image pair;

[0198] The second obtaining module 22 is used to: input the preprocessed reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image; and input the preprocessed reference image pair into the three-dimensional reconstruction model in the order of the second reference image to the first reference image.

[0199] In one possible implementation, the generation module 26 is used to:

[0200] Based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair, the optimized depth map corresponding to each reference image pair is determined.

[0201] Based on the optimized depth maps corresponding to each reference image pair, a 3D model corresponding to the target scene is generated.

[0202] In one possible implementation, the generation module 26 is used to:

[0203] By using an assisted 3D reconstruction model, a set of 3D key points corresponding to each reference image pair is generated;

[0204] Based on the point map, confidence map, depth scale adjustment parameter value, 3D key point set and correction parameter of each reference image pair, construct the joint optimization function corresponding to the reference image pair set;

[0205] By minimizing the joint optimization function, the corresponding correction parameter values ​​for each reference image pair are obtained;

[0206] Based on the corresponding point map, depth scale adjustment parameter value, and correction parameter value of each reference image pair, the corresponding optimized depth map is determined.

[0207] In one possible implementation, the generation module 26 is used to:

[0208] By fusing the optimized depth maps corresponding to each reference image pair using a truncated symbolic distance function, a 3D model corresponding to the target scene is obtained.

[0209] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and technical effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0210] This disclosure also provides a three-dimensional reconstruction apparatus, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0211] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0212] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0213] Figure 3 This is a block diagram illustrating a three-dimensional reconstruction apparatus 1900 according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 3The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0214] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0215] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0216] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0217] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0218] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0219] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0220] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0221] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0222] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0223] Computer program products can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0224] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0225] If the technical solution of this disclosure involves personal information, the product applying the technical solution of this disclosure has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this disclosure involves sensitive personal information, the product applying the technical solution of this disclosure has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to indicate that the user has entered the scope of personal information collection and that personal information will be collected. If the user voluntarily enters the collection scope, it is deemed to have consented to the collection of their personal information; or on the personal information processing device, with clear signs / information informing the user of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0226] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Obtain a set of reference image pairs for the target scene, wherein any reference image pair includes two reference images; The point map corresponding to each reference image pair in the reference image pair set is obtained by using a preset three-dimensional reconstruction model; Based on the corresponding point map and depth scale adjustment parameters of each reference image pair, construct the depth scale consistency optimization function corresponding to the set of reference image pairs; The depth scale consistency optimization function is decomposed into eigenvalues ​​to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set. By minimizing the depth scale consistency optimization function of the dimensionality reduction, the depth scale adjustment parameter values ​​corresponding to each reference image pair are obtained; Based on the reference images, the parameter values ​​of the corresponding point map and depth scale are adjusted to generate a 3D model corresponding to the target scene. The method further includes: obtaining confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model; The step of constructing the depth scale consistency optimization function corresponding to the reference image pair set based on the corresponding point map and depth scale adjustment parameters of each reference image pair includes: constructing the depth scale consistency optimization function corresponding to the reference image pair set based on the corresponding point map, confidence map and depth scale adjustment parameters of each reference image pair.

2. The method according to claim 1, characterized in that, The three-dimensional reconstruction model adopts a dense, unconstrained stereoscopic three-dimensional reconstruction model.

3. The method according to claim 1, characterized in that, The reference image pair satisfies at least two of the following conditions: The number of keypoint matches between two reference images is greater than or equal to a preset number; The difference in rotation angle between the two reference images is within a preset angle range; The ratio of the z-axis component of the translation vector between two reference images to the magnitude of the translation vector is less than or equal to a preset ratio.

4. The method according to claim 1, characterized in that, The step of performing eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set includes: Eigenvalue decomposition is performed on the depth scale consistency optimization function to obtain the equivalent point coordinates corresponding to the point map; Based on the equivalent point coordinates corresponding to the point map, the depth scale consistency optimization function for the dimensionality reduction of the reference image set is obtained.

5. The method according to claim 1, characterized in that, The reference image pair includes a first reference image and a second reference image; The point map and confidence map corresponding to each reference image pair in the reference image pair set are obtained through a preset 3D reconstruction model, including: The reference image pair is input into a preset 3D reconstruction model in the order of the first reference image to the second reference image. The 3D reconstruction model outputs a first point map and a first confidence map corresponding to the reference image pair. The first point map represents the 3D point coordinates of each pixel in the first reference image in the camera coordinate system of the first reference image. The reference image pair is input into the 3D reconstruction model in the order from the second reference image to the first reference image. The 3D reconstruction model outputs a second point map and a second confidence map corresponding to the reference image pair. The second point map represents the 3D point coordinates of each pixel in the second reference image in the camera coordinate system of the second reference image.

6. The method according to claim 5, characterized in that, Before inputting the reference image pair into a preset 3D reconstruction model in the order of the first reference image to the second reference image, the method further includes: performing distortion correction and rescaling on the first reference image and the second reference image to obtain a preprocessed reference image pair; The step of inputting the reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image includes: inputting the preprocessed reference image pair into a preset three-dimensional reconstruction model in the order of the first reference image to the second reference image; The step of inputting the reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image includes: inputting the preprocessed reference image pair into the 3D reconstruction model in the order of the second reference image to the first reference image.

7. The method according to claim 1, characterized in that, The step of generating a 3D model corresponding to the target scene by adjusting the parameter values ​​of the corresponding point map and depth scale based on each reference image includes: Based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​of each reference image pair, the optimized depth map corresponding to each reference image pair is determined. Based on the optimized depth maps corresponding to each reference image pair, a 3D model corresponding to the target scene is generated.

8. The method according to claim 7, characterized in that, The step of determining the optimized depth map corresponding to each reference image pair based on the corresponding point map, confidence map, and depth scale adjustment parameter values ​​includes: By using an assisted 3D reconstruction model, a set of 3D key points corresponding to each reference image pair is generated; Based on the point map, confidence map, depth scale adjustment parameter value, 3D key point set and correction parameter of each reference image pair, construct the joint optimization function corresponding to the reference image pair set; By minimizing the joint optimization function, the corresponding correction parameter values ​​for each reference image pair are obtained; Based on the corresponding point map, depth scale adjustment parameter value, and correction parameter value of each reference image pair, the corresponding optimized depth map is determined.

9. The method according to claim 7, characterized in that, The step of generating a 3D model of the target scene based on the optimized depth maps corresponding to each reference image includes: By fusing the optimized depth maps corresponding to each reference image pair using a truncated symbolic distance function, a 3D model corresponding to the target scene is obtained.

10. A three-dimensional reconstruction device, characterized in that, include: The first acquisition module is used to acquire a set of reference image pairs for the target scene, wherein any reference image pair includes two reference images; The second acquisition module is used to obtain the point map corresponding to each reference image pair in the reference image pair set through a preset three-dimensional reconstruction model; The construction module is used to construct the depth scale consistency optimization function corresponding to the set of reference image pairs by adjusting the parameters of the corresponding point map and depth scale according to each reference image pair. The eigenvalue decomposition module is used to perform eigenvalue decomposition on the depth scale consistency optimization function to obtain the dimension-reduced depth scale consistency optimization function corresponding to the reference image pair set. The minimization module is used to obtain the corresponding depth scale adjustment parameter values ​​for each reference image pair by minimizing the depth scale consistency optimization function of the dimensionality reduction. The generation module is used to adjust the parameter values ​​of the corresponding point map and depth scale based on each reference image to generate a three-dimensional model corresponding to the target scene. The second obtaining module is further configured to: obtain the confidence maps corresponding to each reference image pair through the three-dimensional reconstruction model; The construction module is used to: construct a depth scale consistency optimization function corresponding to the set of reference image pairs based on the corresponding point map, confidence map and depth scale adjustment parameters of each reference image pair.

11. A three-dimensional reconstruction apparatus, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images