Multi-view three-dimensional reconstruction method and device

Abstract

The application discloses a multi-view three-dimensional reconstruction method and device, comprising: processing multiple input images using multi-layer ordinary convolution to obtain multiple first feature maps of different scales for each input image; processing the multiple first feature maps of different scales using multiple deformable convolutions which do not share parameters, and obtaining multiple second feature maps of different scales for each input image after bilinear interpolation; splicing the multiple second feature maps of each input image to obtain an output feature map of each input image; constructing a cost volume; regularizing the cost volume to obtain a probability volume; obtaining an initial depth map according to a reference image; optimizing the initial depth map to obtain a depth optimized map; and generating a dense point cloud. The technical scheme provided by the application overcomes the technical problem that it is difficult to extract weak texture surface features, so that the weak texture region also has good reconstruction effect, and while ensuring accurate generation of the depth map, the GPU running loss is reduced.

Description

Technical Field

[0001] This application relates to the field of remote sensing and mapping geographic information technology, in particular to a multi-view Figure 3 Method and device for reconstruction. Background Art

[0002] Three-dimensional reconstruction refers to the process of depicting real scenes into mathematical models that conform to computer logical expression based on single-view or multi-view images through processes such as depth data acquisition, preprocessing, point cloud alignment and fusion, and surface generation. It is widely used in virtual reality, autonomous driving, game development, architectural design, clinical medicine and other fields.

[0003] Traditional multi-view stereo matching reconstruction methods use hand-crafted similarity metrics and photometric consistency to estimate depth maps and generate dense 3D point clouds. Although these methods show good reconstruction results in the ideal Lambertian case, they also have some common limitations. For example, the presence of problems such as weak textures, highlights, and specular reflections makes dense matching difficult to handle, resulting in incomplete reconstruction results. To overcome this limitation, deep learning has been introduced in recent years to improve stereo reconstruction methods. Deep learning-based methods have higher levels of accuracy and completeness than traditional methods in many MVS (Multi-view Stereo) indicators.

[0004] However, current deep learning-based 3D reconstruction methods still have some problems. Currently, 2D convolutional neural networks have difficulty extracting weak texture surface features on a regular pixel grid, which affects the integrity of 3D reconstruction. Summary of the Invention

[0005] The embodiment of the present application provides a multi-view Figure 3 The three-dimensional reconstruction method and device solve the technical problem that the existing three-dimensional reconstruction method has difficulty in extracting low-texture surface features, which affects the integrity of the three-dimensional reconstruction.

[0006] In the first aspect, the embodiment of the present application provides a multi-view Figure 3 A dimensional reconstruction method, the method comprising:

[0007] Processing multiple input images using multi-layer ordinary convolution to obtain multiple first feature maps of different scales for each input image; the multiple input images include one reference image and the rest are source images;

[0008] Using multiple deformable convolutions that do not share parameters to process multiple first feature maps of different scales, and obtaining multiple second feature maps of different scales for each input image after bilinear interpolation; wherein the deformable convolution is defined as follows:

[0009] f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in the ordinary convolution, Δp k and Δm k Respectively represent the offset and weight generated by the deformable convolution after learning;

[0010] splicing the multiple second feature maps of each input image to obtain an output feature map of each input image;

[0011] Constructing a cost volume based on the output feature map and camera parameters of each input image;

[0012] Regularizing the cost body to obtain a probability body;

[0013] determining an initial depth map of the reference image according to the probability volume;

[0014] Optimizing the initial depth map to obtain a depth-optimized map;

[0015] A plurality of the depth-optimized images are fused to generate a dense point cloud.

[0016] In conjunction with the first aspect, in a possible implementation, regularizing the cost volume to obtain a probability volume includes:

[0017] Regularizing the cost volume using a hierarchical recursive convolutional network module to obtain multiple regularized cost maps; the hierarchical recursive convolutional network module includes a plurality of sequentially arranged LSTM Conv cells, and a pooling layer and a deconvolution layer provided between two of the LSTM Conv cells;

[0018] Generate the corresponding probability volume according to the multiple cost maps.

[0019] With reference to the first aspect, in a possible implementation manner, determining an initial depth map of a reference image according to the probability volume includes:

[0020] An initial depth map of the reference image is determined along a depth direction of the probability volume using an expected depth value as a depth estimation value of each pixel.

[0021] With reference to the first aspect, in a possible implementation, optimizing the initial depth map includes:

[0022] Projecting the pixels on the reference image to corresponding positions of each of the source images using the initial depth map, and calculating the difference between the values ​​of the output feature map of the reference image and the values ​​of the output feature map of the source image;

[0023] minimizing the difference using a Gauss-Newton method;

[0024] The residual is calculated, and the calculation formula of the residual is as follows:

[0025] ri(p)=F i (p i ′)-F0(p);

[0026] Among them, F i (p i ′) is the value of the output feature map of the source image, and F0(p) is the value of the output feature map of the reference image;

[0027] Calculate the first-order derivative J of each residual with respect to the initial depth map i (p), and determine the increment δ of the current depth according to the following formula:

[0028] δ=-(J T J) -1 J T r;

[0029] Where J is the matrix {J i (p)}, r is the superposition of the residual vector {ri(p)};

[0030] The optimized depth map is obtained by adding the increment of the current depth to the value of the initial depth map.

[0031] In conjunction with the first aspect, in a possible implementation, fusing the multiple depth optimization images to generate a dense point cloud includes:

[0032] The dynamic matching consistency of all the depth optimization graphs is fused to obtain the global dynamic multi-view geometric consistency; the dynamic matching consistency is defined as follows:

[0033]

[0034] The global dynamic multi-view geometric consistency is defined as follows:

[0035]

[0036] Among them, ε p and ε d They represent pixel reprojection error and depth reprojection error respectively, and λ represents the coefficient affecting the two different reprojection errors;

[0037] Filter outliers using a preset filter coefficient.

[0038] In combination with the first aspect, in a possible implementation manner, a cross-entropy loss function is used between the probability volume and the one-hot encoded volume of the real depth map, and the cross-entropy loss function is defined as:

[0039]

[0040] wherein x valid represents a set of valid pixels, G(i, x) represents one-hot encoding generation of the real depth map at i depths of pixel x, and P(i, x) represents a pixel in the probability volume;

[0041] In the optimization of the initial depth map, the distance from the real depth map to the optimized depth map is taken as a loss, that is:

[0042]

[0043] wherein d(x) represents a pixel depth value of the real depth map, and d r (x) represents a pixel depth value of the optimized depth map.

[0044] The loss function in the training of the method is represented as follows:

[0045]

[0046] wherein λ determines whether the depth map optimization module is enabled.

[0047] In combination with the first aspect, in a possible implementation manner, the cost volume is constructed according to the output feature map of each input image and camera parameters, and the construction includes:

[0048] A conical volume is constructed for the reference image at the same interval using the over-plane scanning method with the principal axis of the reference image as the scanning direction.

[0049] The output feature map of each source image is projected onto each depth plane to form a feature volume according to the differentiable homography transformation, and the interpolation method is used to make the projection size of each image the same.

[0050] The cost volume corresponding to the reference image is determined based on the variance using the plurality of feature volumes corresponding to the reference image.

[0051] The second aspect, the embodiments of the present application provide a multi-view Figure 3 reconstruction device, the device comprises:

[0052] A first feature module is configured to process a plurality of input images using a plurality of layers of ordinary convolution to obtain a plurality of first feature maps of different scales for each input image. The plurality of input images includes a reference image and a plurality of source images.

[0053] The second feature module is configured to process the plurality of first feature maps of different scales using a plurality of deformable convolutions that do not share parameters, and obtain a plurality of second feature maps of different scales for each input image after bilinear interpolation; wherein the deformable convolution is defined as follows:

[0054]

[0055] f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in the ordinary convolution, Δp k and Δm k Respectively represent the offset and weight generated by the deformable convolution after learning;

[0056] an output feature module, configured to concatenate the plurality of second feature maps of each input image to obtain an output feature map of each input image;

[0057] A cost volume module, configured to construct a cost volume based on the output feature map and camera parameters of each input image;

[0058] A probability body module, used for regularizing the cost body to obtain a probability body;

[0059] an initial depth module, configured to determine an initial depth map of the reference image based on the probability volume;

[0060] A depth optimization module, configured to optimize the initial depth map to obtain a depth optimization map;

[0061] The fusion module is used to fuse multiple depth optimization images to generate a dense point cloud.

[0062] In conjunction with the second aspect, in one possible implementation, the probability body module is specifically used to: regularize the cost body using a hierarchical recursive convolutional network module to obtain multiple regularized cost maps; the hierarchical recursive convolutional network module includes a plurality of sequentially arranged LSTM Conv cells, and a pooling layer and a deconvolution layer provided between two of the LSTM Conv cells;

[0063] Generate the corresponding probability volume according to the multiple cost maps.

[0064] In combination with the second aspect, in a possible implementation, the initial depth module is specifically configured to determine an initial depth map of the reference image along a depth direction of the probability volume using an expected depth value as a depth estimation value of each pixel.

[0065] In conjunction with the second aspect, in a possible implementation, the depth optimization module is specifically configured to: project pixel points on the reference image to corresponding positions of each of the source images using the initial depth map, and calculate a difference between a value of the output feature map of the reference image and a value of the output feature map of the source image;

[0066] minimizing the difference using a Gauss-Newton method;

[0067] The residual is calculated, and the calculation formula of the residual is as follows:

[0068] ri(p)=F i (p i ′)-F0(p);

[0069] Among them, F i (p i ′) is the value of the output feature map of the source image, and F0(p) is the value of the output feature map of the reference image;

[0070] Calculate the first-order derivative J of each residual with respect to the initial depth map i (p), and determine the increment δ of the current depth according to the following formula:

[0071] δ=-(J T J) -1 J T r;

[0072] Where J is the matrix {J i (p)}, r is the superposition of the residual vector {ri(p)};

[0073] The optimized depth map is obtained by adding the increment of the current depth to the value of the initial depth map.

[0074] In conjunction with the second aspect, in one possible implementation, the fusion module is specifically configured to:

[0075] The dynamic matching consistency of all the depth optimization graphs is fused to obtain the global dynamic multi-view geometric consistency; the dynamic matching consistency is defined as follows:

[0076]

[0077] The global dynamic multi-view geometric consistency is defined as follows:

[0078]

[0079] Among them, ε p and ε dThey represent pixel reprojection error and depth reprojection error respectively, and λ represents the coefficient affecting the two different reprojection errors;

[0080] Filter outliers using a preset filter coefficient.

[0081] In conjunction with the second aspect, in one possible implementation, a cross entropy loss function is used between the probability volume and the one-hot encoding volume of the true depth map, and the cross entropy loss function is defined as:

[0082]

[0083] Among them, x valid represents a valid pixel set, G(i,x) represents the true depth map generated by one-hot encoding at the i-th depth of pixel x, and P(i,x) represents the pixel in the probability volume;

[0084] When optimizing the initial depth map, the distance from the real depth map to the optimized depth map is used as the loss, that is:

[0085]

[0086] Wherein, d(x) represents the pixel depth value of the real depth map, d r (x) represents the pixel depth value of the optimized depth map;

[0087] The loss function during training of the method is expressed as follows:

[0088]

[0089] Here, λ determines whether the method starts the depth map optimization module.

[0090] In conjunction with the second aspect, in one possible implementation, the cost volume module is specifically configured to:

[0091] Taking the principal optical axis of the reference image as a scanning direction, a through-plane scanning method is used to construct cones with the same spacing on the reference image;

[0092] Projecting the output feature map of each source image onto each depth plane to form a feature volume according to a differentiable homography transformation, and making each projection have the same size by using an interpolation method;

[0093] The cost volume corresponding to the reference image is determined based on variance using the plurality of feature volumes corresponding to the reference image.

[0094] In the third aspect, the embodiment of the present application provides a multi-view Figure 3 Maintenance and reconstruction equipment, the equipment comprising:

[0095] a memory for non-transitory storage of computer-readable instructions;

[0096] A processor configured to execute the computer-readable instructions, wherein the computer-readable instructions are executed by the processor to implement the multi-viewing method described in the first aspect and various possible implementations of the first aspect. Figure 3 Dimensional reconstruction method.

[0097] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the multi-viewing system described in the first aspect and various possible implementations of the first aspect is implemented. Figure 3 Dimensional reconstruction method.

[0098] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0099] The embodiment of the present invention provides a multi-view Figure 3 A dimensional reconstruction method is proposed. In the feature map acquisition stage, the method uses multiple layers of ordinary convolution and multiple deformable convolutions that do not share parameters to process the input image. After the multiple input images are processed by the multiple layers of ordinary convolution, multiple first feature maps of different scales are obtained. After the multiple deformable convolutions that do not share parameters process the multiple first feature maps of different scales, multiple second feature maps of different scales are obtained by bilinear interpolation. The multiple second feature maps are spliced ​​to obtain the output feature map, so that the multi-view Figure 3 The 3D reconstruction method realizes the multi-scale acquisition of image information, overcomes the technical problem of difficulty in extracting surface features with weak textures, and enables the weak texture areas to have better reconstruction effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present invention or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0101] Figure 1 A schematic diagram of generating a second feature map provided in an embodiment of the present application;

[0102] Figure 2 The multi-viewing embodiment provided in this application Figure 3 Flowchart of the dimensional reconstruction method;

[0103] Figure 3 A flowchart of generating a probability body provided in an embodiment of the present application;

[0104] Figure 4A schematic diagram of the structure of a layered recursive convolutional network module provided in an embodiment of the present application;

[0105] Figure 5 A flowchart for obtaining an optimized depth map provided in an embodiment of the present application;

[0106] Figure 6 A flowchart of the fusion optimization depth map provided in an embodiment of the present application;

[0107] Figure 7 A flowchart of obtaining a cost body provided in an embodiment of the present application;

[0108] Figure 8 The multi-viewing embodiment provided in this application Figure 3 Schematic diagram of the structure of the dimensional reconstruction device;

[0109] Figure 9 The multi-viewing embodiment provided in this application Figure 3 Schematic diagram of the structure of the maintenance and reconstruction equipment;

[0110] Figure 10 Comparison between the depth optimization graph provided in the embodiment of this application and the MVSNet depth graph;

[0111] Figure 11A 、 Figure 11B and Figure 11C The following is a comparison between the reconstruction results provided by the embodiments of the present application and the reconstruction results of R-MVSNet in different scenarios. DETAILED DESCRIPTION

[0112] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0113] The embodiment of the present application provides a multi-view Figure 3 dimensional reconstruction method, such as Figure 2 As shown, it includes S201 to S208.

[0114] S201: Process multiple input images using multi-layer ordinary convolution to obtain multiple first feature maps of different scales for each input image, wherein the multiple input images include one reference image and the rest are source images.

[0115] For example, Figure 1There are five layers of ordinary convolution 10a, 10b, 10c, 10d, and 10e, and the step size of the last two layers of ordinary convolution 10d and 10e is 2. After the input image is processed by the five layers of ordinary convolution 10a, 10b, 10c, 10d, and 10e, three first feature maps of different scales are obtained.

[0116] S202: Use multiple deformable convolutions that do not share parameters to process multiple first feature maps of different scales, and obtain multiple second feature maps of different scales for each input image after bilinear interpolation.

[0117] Among them, the deformable convolution is defined as follows:

[0118] In the above formula, f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in ordinary convolution, Δp k and Δm k They represent the offset and weight generated by the learning of deformable convolution respectively.

[0119] Figure 1 The middle layer of ordinary convolution obtains three first feature maps of different scales. Then, three deformable convolutions 20 without sharing parameters are used to process the first feature maps of three different scales. Then, after bilinear interpolation, three second feature maps 30a, 30b, and 30c of different scales are obtained. Figure 1 The scales of the three second feature maps 30a, 30b, and 30c are H×W×16, H / 2×W / 2×8, and H / 4×W / 4×8, respectively, where H and W are the scales of the input image.

[0120] S203: Splicing multiple second feature maps of each input image to obtain an output feature map of each input image.

[0121] like Figure 1 In the structure shown, each input image can obtain three second feature maps 30a, 30b, and 30c of different scales. Through S203, the three second feature maps 30a, 30b, and 30c with scales of H×W×16, H / 2×W / 2×8, and H / 4×W / 4×8 can be spliced ​​into an output feature map, and the scale of the output feature map is H×W×32.

[0122] certainly, Figure 1 This application is only a specific example provided by this application, and this application does not Figure 1The various numbers and scales shown in the figure are limited, as detailed below. The number of ordinary convolution layers is not limited to five; it can also be six, seven, eight, or other numbers. The number of first feature maps of different scales is not limited to three; it can also be four, five, or other numbers of first feature maps of different scales. The number of deformable convolutions that do not share parameters is not limited to three; it can also be other numbers. For example, when the number of first feature maps of different scales is four, the number of deformable convolutions that do not share parameters is also four; when the number of first feature maps of different scales is five, the number of deformable convolutions that do not share parameters is also five. In addition, the scale of the second feature map is not limited to H×W×16, H / 2×W / 2×8, or H / 4×W / 4×8, and the scale of the final output feature map is not limited to H×W×32. The scales of the second feature map and the output feature map can be set according to actual needs.

[0123] Through S201 to S203, multi-scale acquisition of the input image is achieved, the technical problem of difficulty in extracting surface features of weak texture areas is overcome, and better extraction of surface features of weak texture areas is achieved, so that the weak texture areas can obtain better reconstruction effects in subsequent steps.

[0124] S204: Construct a cost volume based on the output feature map of each input image and the camera parameters. The specific steps of implementing S204 in this embodiment of the application are as follows: Figure 5 As shown, it includes S701 to S703.

[0125] S701: Using the principal optical axis of the reference image as the scanning direction, a through-plane scanning method is used to construct a cone with the same spacing for the reference image.

[0126] S702: Project the output feature map of each source image onto each depth plane to form a feature volume according to the differentiable homography transformation, and use interpolation to make each projection have the same size. The differentiable homography transformation is defined as follows:

[0127]

[0128] In the above formula, {K, R, t} are camera parameters, which are camera intrinsic parameters, rotation and translation respectively; n is the main optical axis of the reference image, and θ is the depth value.

[0129] S703: Using multiple feature volumes corresponding to the reference image, determine a cost volume corresponding to the reference image based on the variance.

[0130] Specifically, let's assume that the number of source images is N. In theory, each reference image will have N corresponding feature volumes. These feature volumes are used to construct a cost volume based on the variance. The calculation method is as follows:

[0131]

[0132] In the above formula, V represents the feature volume of each source image.

[0133] S205: Regularize the cost volume to obtain the probability volume. This embodiment of the application provides a specific implementation method for obtaining the probability volume, such as Figure 3 As shown, it includes S301 and S302.

[0134] S301: Use the layered recursive convolutional network module to regularize the cost volume and obtain multiple regularized cost maps. Figure 4 The hierarchical recursive convolutional network module 40 includes: a plurality of LSTMConvCells 41, 43, 45, 47, 49 arranged in sequence, and pooling layers 42, 44 and deconvolution layers 46, 48 arranged between two LSTMConvCells.

[0135] LSTM (Long Short-Term Memory) is a neural network capable of memorizing both long-term and short-term information. When using deep learning to process time series, it can address the long-term dependency issues of RNNs (Recurrent Neural Networks). LSTMConvCell41, 43, 45, 47, and 49 incorporate convolutional computations into LSTM. This not only captures temporal relationships but also extracts spatial features similar to convolutional layers. This allows LSTMConvCell41, 43, 45, 47, and 49 to simultaneously extract temporal and spatial features, and switching between states is replaced by convolutional computations. This allows for both effective absorption of multi-scale contextual information and efficient processing of the cost volume.

[0136] The LSTM part in each LSTMConvCell41, 43, 45, 47, and 49 generates the following four variables:

[0137]

[0138] Finally, LSTMConvCell41, 43, 45, 47, and 49 output two variables. where h t is the output of the unit.

[0139] For example, the specific structure and parameters of the hierarchical recursive convolutional network module 40 may be shown in the following table:

[0140]

[0141] Figure 4The structure shown can also be shown in the table above, with the ResBlock module added after the last LSTMConvCell49.

[0142] S302: Generate corresponding probability volumes according to multiple cost maps.

[0143] The cost volume is divided into D layers in the depth direction, so the cost volume can be regarded as D 2D cost maps connected in the depth direction {C(i)} i＝0.....D-1 , in the sequential processing process, the regularized cost map is output {CH(i)} i＝0....D-1 , and finally the corresponding probability body is generated through the softmax layer.

[0144] Apart from Figure 3 While S301 and S302 can regularize the cost volume to obtain the probability volume, other specific implementations can also be used. For example, using a 3D CNN to directly regularize the cost volume, or using a stacked recursive structure and a state transfer mechanism to hierarchically process the 3D cost volume into sequentially connected 2D cost maps.

[0145] Directly regularizing the cost volume using a 3DCNN can effectively utilize the cost volume's local information and multi-scale context. However, due to the limited memory of the GPU (Graphics Processing Unit), especially for large-resolution images, it cannot directly regress the depth map of dense point clouds. Therefore, this method is mainly used to reconstruct objects in small scenes, such as the DTU dataset used in MVSNet.

[0146] By utilizing a stacked recursive structure and relying on a state transfer mechanism, the 3D cost volume can be hierarchically processed into sequentially connected 2D cost maps. Although this method can optimize GPU memory loss and greatly improve network operation efficiency without affecting reconstruction accuracy, it lacks the aggregation of multi-scale contextual information.

[0147] The specific implementation method of obtaining the probability body by regularizing the cost body given in the embodiment of the present application can not only well absorb multi-scale context information but also optimize the memory loss of the GPU, thereby improving processing accuracy and efficiency.

[0148] S206: Determine an initial depth map of the reference image according to the probability volume.

[0149] The initial depth map can be obtained by using the following method: Based on the winner-takes-all principle, the initial depth map can be directly obtained using the argmax method. However, the argmax method estimates depth at the sub-pixel level, which can cause sudden depth changes and unevenness.

[0150] When acquiring the initial depth map, the embodiment of the present application determines the initial depth map of the reference image along the depth direction of the probability volume using the expected depth value as the depth estimation value of each pixel, so that different parts of the initial depth map are smoother.

[0151] S207: Optimize the initial depth map to obtain a depth optimization map. Figure 5 The specific method for implementing S207 is shown, including S501 to S505.

[0152] S501: Project the pixels on the reference image to the corresponding positions of each source image using the initial depth map, and calculate the difference between the values ​​of the output feature map of the reference image and the values ​​of the output feature map of the source image.

[0153] S502: Minimize the difference value using the Gauss-Newton method.

[0154] S503: Calculate the residual. The residual calculation formula is as follows: ri(p) = F i (p i ′)-F0(p).

[0155] In the above formula, F i (p i ′) is the value of the output feature map of the source image, and F0(p) is the value of the output feature map of the reference image.

[0156] S504: Calculate the first-order derivative J of each residual with respect to the initial depth map i (p), and determine the current depth increment δ according to the following formula: δ=-(J T J) -1 J T r.

[0157] Where J is the matrix {J i (p)}, and r is the superposition of the residual vector {ri(p)}.

[0158] S505: Add the current depth increment to the value of the initial depth map to obtain an optimized depth map.

[0159] like Figure 10 As shown, by comparing the depth map obtained in the MVSNet method with the depth optimization map of the embodiment of the present application, it can be seen that the depth optimization map finally obtained by executing S201 to S207 in the embodiment of the present application is closer to the actual situation and has fewer defects.

[0160] S208: Fusion of multiple depth optimization images to generate a dense point cloud. The specific method of implementing S208 in this embodiment of the application is as follows: Figure 6 As shown, it includes S601 and S602.

[0161] S601: Fusing the dynamic matching consistency of all depth optimization graphs to obtain global dynamic multi-view geometric consistency.

[0162] Dynamic matching consistency is defined as follows:

[0163] Global dynamic multi-view geometric consistency is defined as follows:

[0164] Among them, ε p and ε d They represent pixel reprojection error and depth reprojection error respectively, and λ represents the coefficient that affects the two different reprojection errors.

[0165] S602: Filter outliers using a preset filter coefficient.

[0166] Apart from Figure 6 In addition to the fusion implementation shown, other methods can also be used to implement fusion, such as: following geometric constraints when fusing depth maps to measure the consistency of depth estimation of multiple views.

[0167] However, other fusion implementations use pre-fixed parameters, such as pixel reprojection error and depth reprojection error. These fixed parameters are not reliable for different scenes, and using fixed parameters may not filter out enough mismatched pixels in different scenes. Therefore, the embodiment of the present application applies dynamic consistency checking in S601 and S602 to fuse multiple optimized depth maps, dynamically constraining the consistency of adjacent views, thereby obtaining a more accurate and complete dense point cloud.

[0168] Figure 11A 、 Figure 11B and Figure 11C The multi-viewing method provided by the embodiment of the present application is shown in different scenarios. Figure 3 Comparison of the reconstruction results of the 3D reconstruction method and the reconstruction results of the R-MVSNet method shows that the point cloud in the reconstruction results of the embodiment of the present application is more accurate and complete, especially the framed area in the figure is more obvious.

[0169] The cross entropy loss function is used between the probability volume and the one-hot encoding volume of the true depth map. The cross entropy loss function is defined as:

[0170]

[0171] Among them, x valid Represents a valid pixel set, G(i,x) represents the true depth map generated by one-hot encoding at the i-th depth of pixel x, and P(i,x) represents the pixel in the probability volume.

[0172] When optimizing the initial depth map, the distance from the true depth map to the optimized depth map is used as the loss, that is:

[0173]

[0174] Among them, d(x) represents the pixel depth value of the real depth map, d r (x) represents the pixel depth value of the optimized depth map.

[0175] The multi-viewing provided in the embodiment of the present application Figure 3 The loss function during training of the dimensional reconstruction method is expressed as follows:

[0176]

[0177] Among them, λ determines whether this method turns on the depth map optimization module.

[0178] Currently, the 3D reconstruction method based on deep learning usually uses the distance function as the loss function, and regards the training as a regression problem. Figure 3 The dimensional reconstruction method is divided into two parts: the cross entropy loss function is used between the probability body and the real depth map, which is regarded as a multi-classification task; the distance function is used between the optimized depth map and the real depth map, which is regarded as a regression task; and the cross entropy loss function and the distance function are used as multi-view Figure 3 The loss function of the dimensional reconstruction method makes the training results of the method provided in the embodiment of the present application more accurate.

[0179] The multi-viewing provided in the embodiment of the present application Figure 3 The dimensional reconstruction method significantly reduces GPU memory consumption, with runtime consumption being 60.1% of R-MVSNet and only 26.1% of MVSNet. Therefore, this method can be applied to high-resolution images and reconstruct large-scale scenes. Quantitative results are shown in the following table.

[0180]

[0181] Although the present application provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in this embodiment is only one way of executing the steps among many steps and does not represent the only execution order. When an actual device or client product is executed, the method shown in this embodiment or the accompanying drawings may be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0182] The embodiment of the present application also provides a multi-view Figure 3 Maintenance and reconstruction devices, such as Figure 8As shown, the device includes a first feature module 81, a second feature module 82, an output feature module 83, a cost body module 84, a probability body module 85, an initial depth module 86, a depth optimization module 87 and a fusion module 88.

[0183] The first feature module 81 is used to process multiple input images using multi-layer ordinary convolution to obtain multiple first feature maps of different scales for each input image; the multiple input images include one reference image and the rest are source images.

[0184] The second feature module 82 is configured to process multiple first feature maps of different scales using multiple deformable convolutions that do not share parameters, and obtain multiple second feature maps of different scales for each input image after bilinear interpolation; wherein the deformable convolution is defined as follows:

[0185]

[0186] f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in ordinary convolution, Δp k and Δm k They represent the offset and weight generated by the learning of deformable convolution respectively.

[0187] The output feature module 83 is used to splice multiple second feature maps of each input image to obtain an output feature map of each input image.

[0188] The cost volume module 84 is used to construct a cost volume based on the output feature map of each input image and the camera parameters.

[0189] The probability volume module 85 is used to regularize the cost volume to obtain the probability volume.

[0190] The initial depth module 86 is configured to determine an initial depth map of the reference image according to the probability volume.

[0191] The depth optimization module 87 is used to optimize the initial depth map to obtain a depth optimization map.

[0192] The fusion module 88 is used to fuse multiple depth optimization images to generate a dense point cloud.

[0193] The cost volume module 84 is specifically used to: use the main optical axis of the reference image as the scanning direction, and use the plane scanning method to construct a cone with the same spacing for the reference image; project the output feature map of each source image onto each depth plane according to the differentiable homography transformation to form a feature volume, and use the interpolation method to make each projection size the same; use multiple feature volumes corresponding to the reference image to determine the cost volume corresponding to the reference image based on the variance.

[0194] The probability body module 85 is specifically used to: use a hierarchical recursive convolutional network module to regularize the cost body to obtain multiple regularized cost maps; the hierarchical recursive convolutional network module includes multiple LSTMConvcells arranged in sequence, and a pooling layer and a deconvolution layer arranged between two LSTMConvcells; generate corresponding probability bodies according to multiple cost maps.

[0195] The initial depth module 86 is specifically configured to determine an initial depth map of the reference image along the depth direction of the probability volume using the expected depth value as the depth estimation value of each pixel.

[0196] The depth optimization module 87 is specifically used to: project the pixel points on the reference image to the corresponding positions of each source image through the initial depth map, calculate the difference between the value of the output feature map of the reference image and the value of the output feature map of the source image; minimize the difference using the Gauss-Newton method; calculate the residual, and the residual calculation formula is as follows: ri(p) = F i (p i ′)-F0(p); where F i (p i ′) is the value of the output feature map of the source image, F0(p) is the value of the output feature map of the reference image; calculate the first-order derivative J of each residual to the initial depth map i (p), and determine the current depth increment δ according to the following formula: δ=-(J T J) -1 J T r; where J is the matrix {J i (p)}, r is the superposition of the residual vector {ri(p)}; the current depth increment is added to the value of the initial depth map to obtain the optimized depth map.

[0197] The fusion module 88 is specifically used to fuse the dynamic matching consistency of all depth optimization images to obtain global dynamic multi-view geometric consistency; the dynamic matching consistency is defined as follows: Global dynamic multi-view geometric consistency is defined as follows: Among them, ε p and ε d They represent pixel reprojection error and depth reprojection error respectively, and λ represents the coefficient affecting the two different reprojection errors; the preset filter coefficient is used to filter outliers.

[0198] The cross entropy loss function is used between the probability volume and the one-hot encoding volume of the true depth map. The cross entropy loss function is defined as: Among them, x valid Represents a valid pixel set, G(i,x) represents the true depth map generated by one-hot encoding at the i-th depth of pixel x, and P(i,x) represents the pixel in the probability volume.

[0199] When optimizing the initial depth map, the distance from the true depth map to the optimized depth map is used as the loss, that is: Among them, d(x) represents the pixel depth value of the real depth map, d r (x) represents the pixel depth value of the optimized depth map.

[0200] The multi-viewing provided in the embodiment of the present application Figure 3 The loss function during training of the dimensional reconstruction method is expressed as follows:

[0201]

[0202] Among them, λ determines whether this method turns on the depth map optimization module.

[0203] The devices or modules described in the above embodiments can be implemented by computer chips or physical devices, or by products with certain functions. For ease of description, the above devices are described separately by function and module. When implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0204] The embodiment of the present application also provides a multi-view Figure 3 Maintenance and reconstruction equipment 90, such as Figure 9 As shown, the multi-view Figure 3 The 3D reconstruction device 90 includes a memory 91 and a processor 92 connected via a bus 93. The memory 91 is used to store computer-readable instructions non-transiently. The processor 92 is used to execute the computer-readable instructions. When the computer-readable instructions are executed by the processor, the multi-viewing system provided in the embodiment of the present application is realized. Figure 3 Dimensional reconstruction method.

[0205] The embodiment of the present application also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the multi-viewing system provided in the embodiment of the present application. Figure 3 Dimensional reconstruction method.

[0206] The above-mentioned storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions.

[0207] The methods, devices, or modules described in this application can be implemented in the form of computer-readable program code. The controller can be implemented in any appropriate manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to implement the same function of the controller in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the means for implementing various functions may be considered to be both a software module for implementing the method and a structure within a hardware component.

[0208] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, or can be embodied through the implementation process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0209] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. All or part of this application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0210] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some or all of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present application.

Claims

1. A multi-view 3D reconstruction method, characterized in that: include: Processing multiple input images using multi-layer ordinary convolution to obtain multiple first feature maps of different scales for each input image; The plurality of input images include one reference image, and the rest are source images; Using multiple deformable convolutions that do not share parameters to process multiple first feature maps of different scales, and obtaining multiple second feature maps of different scales for each input image after bilinear interpolation; wherein the deformable convolution is defined as follows: f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in the ordinary convolution, Δp k and Δm k Respectively represent the offset and weight generated by the deformable convolution after learning; splicing the multiple second feature maps of each input image to obtain an output feature map of each input image; Constructing a cost volume based on the output feature map and camera parameters of each input image; Regularizing the cost body to obtain a probability body; determining an initial depth map of the reference image according to the probability volume; Optimizing the initial depth map to obtain a depth-optimized map; A plurality of the depth-optimized images are fused to generate a dense point cloud.

2. The multi-view 3D reconstruction method according to claim 1, wherein: Regularizing the cost volume to obtain a probability volume includes: Regularizing the cost volume using a hierarchical recursive convolutional network module to obtain multiple regularized cost maps; the hierarchical recursive convolutional network module includes a plurality of sequentially arranged LSTM Conv cells, and a pooling layer and a deconvolution layer provided between two of the LSTM Conv cells; Generate the corresponding probability volume according to the multiple cost maps.

3. The multi-view 3D reconstruction method according to claim 1, wherein: Determining an initial depth map of a reference image according to the probability volume includes: An initial depth map of the reference image is determined along a depth direction of the probability volume using an expected depth value as a depth estimation value of each pixel.

4. The multi-view 3D reconstruction method according to claim 1, wherein: Optimizing the initial depth map includes: Projecting the pixels on the reference image to corresponding positions of each of the source images using the initial depth map, and calculating the difference between the values ​​of the output feature map of the reference image and the values ​​of the output feature map of the source image; minimizing the difference using a Gauss-Newton method; The residual is calculated, and the calculation formula of the residual is as follows: to(p)=F i (p i ′)-F0(p); Among them, F i (p i ′) is the value of the output feature map of the source image, and F0(p) is the value of the output feature map of the reference image; Calculate the first-order derivative J of each residual with respect to the initial depth map i (p), and determine the increment δ of the current depth according to the following formula: δ=-(J T J) -1 I T r; Where J is the matrix {J i (p)}, r is the superposition of the residual vector {ri(p)}; The depth optimization map is obtained by adding the increment of the current depth to the value of the initial depth map.

5. The multi-view 3D reconstruction method according to claim 1, wherein: The fusing of the plurality of depth optimization images to generate a dense point cloud comprises: The dynamic matching consistency of all the depth optimization graphs is fused to obtain the global dynamic multi-view geometric consistency; the dynamic matching consistency is defined as follows: The global dynamic multi-view geometric consistency is defined as follows: Among them, ε p and ε d They represent pixel reprojection error and depth reprojection error respectively, and λ represents the coefficient affecting the two different reprojection errors; Filter outliers using a preset filter coefficient.

6. The multi-view 3D reconstruction method according to claim 1, wherein: A cross entropy loss function is used between the probability volume and the one-hot encoded volume of the true depth map, and the cross entropy loss function is defined as: Among them, x valid represents a valid pixel set, G(i,x) represents the true depth map generated by one-hot encoding at the i-th depth of pixel x, and P(i,x) represents the pixel in the probability volume; When optimizing the initial depth map, the distance from the real depth map to the depth optimization map is used as the loss, that is: Wherein, d(x) represents the pixel depth value of the real depth map, d r (x) represents the pixel depth value of the depth optimization map; The loss function during training of the method is expressed as follows: Here, λ determines whether the method starts the depth map optimization module.

7. The multi-view 3D reconstruction method according to claim 1, wherein: The constructing a cost volume according to the output feature map and camera parameters of each input image includes: Taking the principal optical axis of the reference image as a scanning direction, a through-plane scanning method is used to construct cones with the same spacing on the reference image; Projecting the output feature map of each source image onto each depth plane to form a feature volume according to a differentiable homography transformation, and making each projection have the same size by using an interpolation method; The cost volume corresponding to the reference image is determined based on variance using the plurality of feature volumes corresponding to the reference image.

8. A multi-view 3D reconstruction device, characterized in that: include: A first feature module is configured to process a plurality of input images using multi-layer ordinary convolution to obtain a plurality of first feature maps of different scales for each of the input images; The plurality of input images include one reference image, and the rest are source images; The second feature module is configured to process the plurality of first feature maps of different scales using a plurality of deformable convolutions that do not share parameters, and obtain a plurality of second feature maps of different scales for each input image after bilinear interpolation; wherein the deformable convolution is defined as follows: f(p) represents the characteristic value of pixel p, w k and p k They represent the convolution kernel parameters and fixed offset defined in the ordinary convolution, Δp k and Δm k Respectively represent the offset and weight generated by the deformable convolution after learning; an output feature module, configured to concatenate the plurality of second feature maps of each input image to obtain an output feature map of each input image; A cost volume module, configured to construct a cost volume based on the output feature map and camera parameters of each input image; A probability body module, used for regularizing the cost body to obtain a probability body; an initial depth module, configured to determine an initial depth map of the reference image based on the probability volume; A depth optimization module, configured to optimize the initial depth map to obtain a depth optimization map; The fusion module is used to fuse multiple depth optimization images to generate a dense point cloud.

9. A multi-view 3D reconstruction device, characterized in that: include: a memory for non-transitory storage of computer-readable instructions; A processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, implement the multi-view 3D reconstruction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the multi-view 3D reconstruction method according to any one of claims 1 to 7 is implemented.

Images