A multi-view stereo matching reconstruction method

Abstract

The application discloses a kind of multi-view stereo matching reconstruction methods, comprising: obtaining the disparity map of multiple perspectives under the same scene;The disparity map obtained is input into the stereo matching reconstruction model trained, to obtain the corresponding depth estimation value;The depth estimation value obtained is converted into point cloud information;Point cloud information is visualized, and three-dimensional scene is reconstructed.The application enhances the conversion and modeling capacity of network, improves the accuracy and integrity of three-dimensional reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-view stereo matching and reconstruction method. Background Technology

[0002] 3D reconstruction technology has become a core technology in computer vision, enabling precise digital modeling of physical objects and providing crucial foundational support for many advanced applications, including but not limited to autonomous driving, robot navigation, and head-mounted displays. Particularly in autonomous driving, 3D reconstruction provides technical support for obstacle detection and accurate road surface modeling. Traditional 3D reconstruction methods are primarily based on geometric theories and feature matching algorithms, requiring a series of complex processing steps, such as feature extraction from views or point clouds, feature matching, and triangulation, to ultimately reconstruct the 3D scene. However, these traditional methods often encounter difficulties in handling specular reflections, varying lighting conditions, and low-texture areas, and they typically require substantial computational resources and expert-level human intervention.

[0003] In recent years, with the rapid advancement of deep learning technology, researchers have begun to explore combining deep learning to optimize 3D reconstruction techniques. This advanced 3D reconstruction method based on deep learning mainly involves the following four key steps: First, using 2D CNN technology to extract depth features from the view; next, using homography transformation to achieve variance-based cost aggregation; second, using 3D U-Net technology to perform depth regularization on the aggregated cost; and finally, performing depth regression to obtain the final 3D reconstruction result. In this series of steps, feature-based cost aggregation is clearly a crucial step. Therefore, how to effectively utilize multi-level cost volumes to ensure the accuracy and reliability of the final 3D reconstruction is the core issue in the development of this technology. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-view stereo matching reconstruction method that enhances the conversion and modeling capabilities of the network and improves the accuracy and completeness of 3D reconstruction.

[0005] To achieve the above-mentioned technical effects, the present invention is implemented using the following solution:

[0006] In a first aspect, the present invention provides a multi-view stereo matching and reconstruction method, comprising:

[0007] Obtain disparity maps from multiple perspectives within the same scene;

[0008] The acquired disparity map is input into the trained stereo matching and reconstruction model to obtain the corresponding depth estimate.

[0009] The obtained depth estimate is converted into point cloud information;

[0010] Visualize point cloud information and reconstruct a 3D scene;

[0011] The training of the stereo matching reconstruction model includes:

[0012] Obtain disparity maps and camera parameters from multiple perspectives in the same scene, and input the obtained disparity maps into the stereo matching and reconstruction model;

[0013] The input disparity map is scaled at different ratios and multi-scale feature maps are extracted.

[0014] The multi-scale feature maps are transformed in stages until the size of the generated depth map matches the size of the input disparity map and the loss of the stereo matching reconstruction model no longer decreases, thus obtaining the trained stereo matching reconstruction model.

[0015] Furthermore, disparity maps and camera parameters from multiple perspectives within the same scene are obtained, and the obtained disparity maps are input into the stereo matching and reconstruction model, including:

[0016] A stereo vision sensor is used to acquire disparity maps from multiple perspectives in the same scene, and the intrinsic parameter K and extrinsic parameter {R|t} of the stereo vision sensor are calculated.

[0017] N disparity maps Input the stereo matching reconstruction model and set I0 as the reference view. Set it as the source view.

[0018] Furthermore, the input disparity map is scaled at different ratios and multi-scale feature maps are extracted, including:

[0019] The input disparity map is scaled at different ratios using a feature pyramid network, and multi-scale feature maps are extracted. The resolutions are respectively And H×W×C3, where W, H, and C are the width, height, and number of channels of the view, respectively.

[0020] Furthermore, the multi-scale feature maps are transformed in stages, including:

[0021] Repeat the following steps for the multi-scale feature map in the scale partitioning stage:

[0022] Transform the feature map of the source view onto the depth assumption plane of the reference view to obtain the feature volume. And aggregated into a single cost body C:

[0023]

[0024] Where W, H, D, and C represent the width, height, and depth of the view, assuming the number of planes and channels, respectively. This is obtained by averaging all feature bodies;

[0025] The cost volume is regularized and normalized to obtain the probability volume, and the expected value is used to calculate the depth estimate and depth map for each pixel.

[0026] An adaptive range acquisition network is used to perform size correction and information constraints on the depth map. Based on the probability volume and depth estimate of the current stage, the initial depth hypothesis plane for the next stage is dynamically estimated for each pixel.

[0027] Furthermore, the homography transformation formula used to transform the feature map of the source view onto the depth assumption plane of the reference view is as follows:

[0028]

[0029] Where p is the pixel in the reference view, d j Let p' be the j-th depth hypothesis plane in the reference view, p' be the corresponding pixel in the source view, K0 be the camera's intrinsic parameters for capturing the reference view, and K be the depth hypothesis plane in the reference view. i R and t are the camera parameters for capturing the source view, and R and t are the relative rotation and translation matrices between the reference view and the source view, respectively.

[0030] Furthermore, the cost volume is regularized and normalized to obtain the probability volume, and the expected value is used to calculate the depth estimate and depth map for each pixel, including:

[0031] The cost volume is regularized using 3DCNNs based on the U-Net architecture, the channel dimension is reduced to 1, and the probability is normalized along the depth direction of the dimension using the softmax function to obtain the probability volume.

[0032] The depth estimate at each pixel p is calculated using the expected value, as shown in the following formula:

[0033]

[0034] Where, d k,j (p) represents the j-th depth hypothesis plane value of pixel p in stage k, P k,j (p) represents the probability of pixel p in the j-th depth hypothesis plane at stage k.

[0035] Furthermore, an adaptive range acquisition network is used to perform size correction and information constraints on the depth map. The number of depth assumption planes in the next stage is shifted according to the depth map of the current stage. The homography transformation formula used in the next stage is as follows:

[0036]

[0037] In the formula, Δ j This represents the offset depth learned in the j-th depth hypothesis plane;

[0038] The adaptive range acquisition network includes a fixed acquisition module and a scalable acquisition module connected in parallel. The fixed acquisition module includes a sigmoid activation function and multiple 2DCNNs, while the scalable acquisition module includes multiple deformable convolutional layers.

[0039] Furthermore, the initial depth assumption plane for the next stage is as follows:

[0040] d k+1 (p)=[D k (p)-σ(p),D k (p)+σ(p)]

[0041] Where σ(x) is the probability volume extracted through a deformable convolutional network, and D k (p) represents the depth estimate for the current stage.

[0042] Furthermore, the loss calculation for the stereo matching reconstruction model uses the L1 loss function formula, as follows:

[0043] Loss1 = Smooth L1 (D τ (p)-D gt )

[0044] Loss2 = Smooth L1 (D k (p)-D gt )

[0045]

[0046] Where, d τ (p) represents the depth estimate of the adaptive range acquisition network, λ τ λ represents the coefficients of the adaptive range acquisition network. k D represents the loss coefficient in stage k. gt D represents the ground truth value. k (p) represents the depth estimate for the k-th stage, Smooth L1 It is a smoothed mean absolute error function.

[0047] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0048] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0049] 1. This invention has high integrity: It proposes a new multi-stage, multi-view stereo reconstruction framework and adopts a novel 3D reconstruction network framework to predict a more accurate depth range. This framework determines the depth range of each pixel by collecting information of each pixel in the depth map in multiple directions and obtains a more accurate depth value.

[0050] 2. The present invention has strong stability: In the process of depth range estimation, each pixel is used to use an offset displacement coefficient to collect the distribution of uncertain pixels between pixels, so as to maintain the effectiveness of the network in the face of different parallax environments. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the multi-view stereo matching and reconstruction method provided by the present invention;

[0052] Figure 2 A schematic diagram of the multi-view stereo matching and reconstruction neural network model provided by the present invention;

[0053] Figure 3 The average depth error during training of the multi-view stereo matching and reconstruction neural network model provided by this invention;

[0054] Figure 4 The reconstruction effect diagram of the multi-view stereo matching reconstruction method provided by the present invention. Detailed Implementation

[0055] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0056] Example 1:

[0057] like Figure 1 As shown, this embodiment discloses a multi-view stereo matching and reconstruction method, which includes the following steps:

[0058] Step 1: Obtain disparity maps of multiple perspectives within the same scene.

[0059] Step 2: Input the acquired disparity map into the trained stereo matching and reconstruction model to obtain the corresponding depth estimate.

[0060] Step 3: Input the obtained depth estimate into the existing Gipuma open-source code. It can convert the depth estimate into point cloud information and use software tools such as MeshLab to process and reconstruct the point cloud information to generate an accurate 3D scene model.

[0061] Among them, such as Figure 2 As shown, the training of the stereo matching reconstruction model includes the following steps:

[0062] Step 1: Use a stereo vision sensor to acquire disparity maps from multiple perspectives of the same scene. Select N RGB disparity maps. Input the stereo matching reconstruction model, where I0 is the reference view. Given the source view, calculate the intrinsic parameters K and extrinsic parameters {R|t} for c cameras;

[0063] Step 2: The stereo matching reconstruction model is divided into three stages from coarse to fine, with each stage sharing a similar structure; all disparity maps are scaled at different ratios through a feature pyramid network, and multi-scale feature maps of all input disparity maps are extracted. The resolutions are respectively H×W×C3, where W, H, and C represent the width, height, and number of channels of the view, respectively;

[0064] Step 3: Step 2 provides multi-scale feature maps, which are then used to train the network from high to low in the feature pyramid network.

[0065] Step 301: First, process the high-level feature map of the first stage, i.e., at a resolution of... The feature map of the source view is warped into a forward parallel frustum of a reference view given a depth hypothesis. For pixel p in the reference image and the j-th depth hypothesis plane d... j Given that the intrinsic parameters of the camera in the reference view and the camera in the source view are K0 and K... i Given the relative rotation and translation matrices between the reference view and the source view as R and t, we can calculate the corresponding pixel p′ in the source view. Therefore, the homography transformation formula is as follows:

[0066]

[0067] Step 302: After warping the feature map onto the depth assumption plane of the reference camera, the feature volume is obtained as follows. Where D represents the number of depth hypothesis planes, and they are aggregated into a single cost volume C. The final cost volume can be calculated as follows:

[0068]

[0069] in, This indicates that the average value is obtained by averaging all feature volumes.

[0070] Step 4: Perform regularization and normalization operations on the cost volume to obtain the probability volume. And calculate the depth estimate and depth map for each pixel using the expected value method, including:

[0071] Step 401: Regularize the cost volume using 3DCNNs based on the U-Net architecture, reduce the channel dimension to 1, and normalize the probability along the depth direction of the dimension using the softmax function to obtain the probability volume;

[0072] Step 402: Calculate the depth estimate at each pixel p using the expected value, as shown in the following formula:

[0073]

[0074] Where, d k,j (p) represents the j-th depth hypothesis plane value of pixel p in stage k, P k,j (p) represents the probability of pixel p in the j-th depth hypothesis plane at stage k. The final depth estimate can be compressed into 0-1 and corresponding to different colors, thus obtaining a visual coarse depth map.

[0075] Step 5: The stereo matching reconstruction model is based on the feature pyramid network. When proceeding to the next stage of the network process, the initial data needs to be updated.

[0076] Step 501: An adaptive range acquisition network is used to perform size correction and information constraint on the coarse depth map. The adaptive range acquisition network consists of fixed acquisition modules with multiple 2D CNNs and sigmoid activation functions, and a scalable acquisition module based on deformable convolutions, which is then updated to the depth hypothesis plane in the next stage. Size correction and information constraint are performed on the current stage's depth map. The depth hypothesis plane in the next stage will shift according to the current stage's depth map, therefore the homography transformation formula is also transformed as follows:

[0077]

[0078] Where, Δ j This represents the offset depth learned in the hypothetical plane at the j-th depth.

[0079] Step 502: Using the probability volume and depth estimate of the current stage, dynamically estimate the initial depth hypothesis plane for the next stage for each pixel, resulting in the following dynamic depth hypothesis plane:

[0080] d k+1 (p)=[D k (p)-σ(p),D k (p)+σ(p)]

[0081] Here, σ(x) is extracted from the probability volume through a deformable convolutional network and is compared with the current stage depth estimate D. k (p) Combined to obtain the depth hypothesis plane for the next stage.

[0082] Step 6: Repeat steps 2-5 until the size of the final generated depth map is consistent with the size of the input disparity map and the loss of the stereo matching reconstruction model no longer decreases, thus obtaining the trained stereo matching reconstruction model.

[0083] The loss calculation process involves comparing depth estimates at different scales with the ground truth values ​​in the L1 loss function. The formula for the L1 loss function is as follows:

[0084] Loss1 = Smooth L1 (D τ (p)-D gt )

[0085] Loss2 = Smooth L1 (D k (p)-D gt )

[0086]

[0087] Where, d τ (p) represents the depth estimate of the adaptive range acquisition network, λ τ λ represents the coefficients of the adaptive range acquisition network. k D represents the loss coefficient in stage k. gt D represents the ground truth value. k (p) represents the depth estimate for the k-th stage, Smooth L1 It is a smoothed mean absolute error function.

[0088] according to Figure 3 and Figure 4 The average depth error and reconstruction results shown in this embodiment demonstrate that the multi-view stereo matching reconstruction method proposes a new multi-stage, multi-view stereo reconstruction framework. It employs a novel 3D reconstruction network framework to predict a more accurate depth range. This framework determines the depth range of each pixel by collecting information from each pixel in the depth map in multiple directions, thus obtaining a more accurate depth value. During the depth range estimation process, each pixel is used with an offset displacement coefficient to collect the distribution of uncertain pixels among pixels, thereby maintaining the effectiveness of the network in the face of different parallax environments.

[0089] Example 2:

[0090] This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method described in Embodiment 1.

[0091] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-view stereo matching and reconstruction method, characterized in that, include: Obtain disparity maps from multiple perspectives within the same scene; The acquired disparity map is input into the trained stereo matching and reconstruction model to obtain the corresponding depth estimate. The obtained depth estimate is converted into point cloud information; Visualize point cloud information and reconstruct a 3D scene; The training of the stereo matching reconstruction model includes: Obtain disparity maps and camera parameters from multiple perspectives in the same scene, and input the obtained disparity maps into the stereo matching and reconstruction model; The input disparity map is scaled at different ratios and multi-scale feature maps are extracted. The multi-scale feature map is transformed in stages until the size of the generated depth map is consistent with the size of the input disparity map and the loss of the stereo matching reconstruction model no longer decreases, thus obtaining the trained stereo matching reconstruction model. The multi-scale feature maps are transformed in stages, including: Repeat the following steps for the multi-scale feature map in the scale partitioning stage: Transform the feature map of the source view onto the depth hypothesis plane of the reference view to obtain the feature volume. And aggregated into a single cost body : ； in, W, H, D , C These represent the width, height, and depth of the view, assuming the number of planes and channels, respectively. This is obtained by averaging all feature bodies; The cost volume is regularized and normalized to obtain the probability volume, and the expected value is used to calculate the depth estimate and depth map for each pixel. An adaptive range acquisition network is used to perform size correction and information constraints on the depth map. Based on the probability volume and depth estimate of the current stage, the initial depth hypothesis plane for the next stage is dynamically estimated for each pixel. The homography transformation formula used to transform the feature map of the source view onto the depth assumption plane of the reference view is as follows: ； in, For the reference pixels in the view, Let the j-th depth assumption plane be in the reference view. For the corresponding pixel in the source view, To capture the in-camera parameters of the reference view, The camera's in-camera parameters for capturing the source view. and This represents the relative rotation and translation matrices between the reference view and the source view.

2. The multi-view stereo matching and reconstruction method according to claim 1, characterized in that, Obtain disparity maps and camera parameters from multiple viewpoints within the same scene, and input the obtained disparity maps into a stereo matching and reconstruction model, including: A stereo vision sensor is used to acquire disparity maps from multiple viewpoints in the same scene, and the intrinsic parameters of the stereo vision sensor are calculated. ; N disparity maps Input the stereo matching reconstruction model, and... Set as reference view. Set it as the source view.

3. The multi-view stereo matching and reconstruction method according to claim 1, characterized in that, The input disparity map is scaled at different ratios and multi-scale feature maps are extracted, including: The input disparity map is scaled at different ratios using a feature pyramid network, and multi-scale feature maps are extracted. The resolutions are respectively , and ,in W, H, C These represent the width, height, and number of channels of the view, respectively.

4. The multi-view stereo matching and reconstruction method according to claim 1, characterized in that, The cost volume is regularized and normalized to obtain the probability volume, and the expected value is used to calculate the depth estimate and depth map for each pixel, including: The cost volume is regularized using 3DCNNs based on the U-Net architecture, reducing the channel dimension to 1. softmax The function is normalized along the depth direction of the dimension to obtain the probability volume; The depth estimate at each pixel p is calculated using the expected value, as shown in the following formula: ； in, This represents the depth hypothesis plane value of pixel p at stage k. This represents the probability that pixel p is in the j-th depth hypothesis plane at stage k; By assigning different colors to the obtained depth estimates, a rough depth map can be visualized.

5. The multi-view stereo matching and reconstruction method according to claim 4, characterized in that, An adaptive range acquisition network is used to perform size correction and information constraints on the depth map. The number of depth assumption planes in the next stage is shifted according to the depth map in the current stage. The homography transformation formula used in the next stage is as follows: ； In the formula, This represents the offset depth learned in the depth hypothesis plane at the j-th depth; The adaptive range acquisition network includes a fixed acquisition module and a scalable acquisition module connected in parallel. The fixed acquisition module includes a sigmoid activation function and multiple 2DCNNs, while the scalable acquisition module includes multiple deformable convolutional layers.

6. The multi-view stereo matching and reconstruction method according to claim 5, characterized in that, The initial depth assumption plane for the next stage is as follows: ； in, The probability volume is extracted using a deformable convolutional network. This is the depth estimate for the current stage.

7. The multi-view stereo matching and reconstruction method according to claim 6, characterized in that, The loss calculation for the stereo matching reconstruction model uses the L1 loss function formula, as follows: ； ； ； in, This represents the depth estimate of the adaptive range acquisition network. These represent the coefficients of the adaptive range acquisition network. Indicates the first k The loss coefficient of the stage, Represents the ground truth value. Indicates the first k The depth estimate of the stage, It is a smoothed mean absolute error function.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images