Multi-view stereoscopic three-dimensional reconstruction algorithm based on MHM novel model

Abstract

According to the method, a multi-cost body aggregation module is combined with a D-RFP network structure, a novel MHM model of deep separator convolution is added into the D-RFP network structure, and the D-RFP performs feature extraction and fusion by using a feature extraction network of an FPN structure for multiple times, so that the expression ability of the extracted features is enhanced; in addition, depth separable convolution is added on the basis of a conventional RFP network, and the operation can effectively reduce the parameter quantity of convolution operation, so that the memory requirement of a running model is reduced. According to the multi-cost aggregation module, the feature bodies are aggregated firstly, then the cost bodies are constructed according to the view angles one by one, then the final cost body is formed through aggregation, the similarity calculation mode of the related features provides effective cost body representation, excessive information loss is avoided, and therefore the precision of the whole network is improved. According to the model process, firstly, a feature body is obtained through a D-RFP structure, then, a cost body is constructed by using a multi-cost body aggregation module, then, an initial depth map is obtained through 3D convolution regularization, and finally, a network is refined through the depth map, and deep fusion is carried out to generate a final three-dimensional point cloud.

Description

Technical Field

[0001] The invention relates to the technical field of computer graphics and image processing, and in particular to a multi-view stereoscopic three-dimensional reconstruction algorithm based on a novel MHM model. Background Art

[0002] The goal of multi-view stereo 3D reconstruction is to restore the 3D model of the scene from images from multiple angles. This technology has a wide range of applications in autonomous driving, virtual reality, medical imaging and other fields, and as the demand for 3D models in these fields increases, 3D reconstruction technology is also constantly improving.

[0003] Three-dimensional reconstruction algorithms can be divided into active and passive types according to the different acquisition sensors. Active three-dimensional reconstruction requires the use of specific sensors, such as structured light or lidar, to obtain the three-dimensional structure information of the target by sending pulsed lasers or light to the target and receiving the reflected or echo information. This algorithm has the characteristics of high precision and high speed, but due to its dependence on acquisition equipment, it is costly and has poor versatility.

[0004] Passive 3D reconstruction uses the acquired image sequence to complete the 3D reconstruction. It uses knowledge from fields such as computer graphics and multi-view geometry to recover depth information from the 2D information in the image to achieve the final 3D reconstruction. Since it does not rely on special acquisition equipment, the data of passive 3D reconstruction is easy to obtain and has strong versatility, but its accuracy is often limited by the data itself. Compared with using depth sensors to collect 3D structural information, passive 3D reconstruction is more economical and reliable. Therefore, in tasks where real-time requirements are not high, passive 3D reconstruction is a better choice.

[0005] Multi-View Stereo (MVS) is an algorithm that uses images captured by a single camera or multiple cameras at different viewing angles to restore the three-dimensional model of a scene or object. Current multi-view stereo algorithms are mainly based on depth map algorithms. The multi-view stereo task refers to reconstructing a dense point cloud model of the scene by predicting the depth of the images given a series of images and their corresponding corrected camera parameters, fusing the predicted depth maps, and performing depth prediction on the images.

[0006] Therefore, the research focus of multi-view stereo algorithms is to generate accurate depth maps for each image in order to perform depth map fusion and generate dense 3D point cloud models. Traditional multi-view stereo algorithms rely on manually designed feature operators to extract features, which can achieve good reconstruction results under the ideal Lambertian reflectance model. However, in non-Lambertian areas and low-texture and repetitive texture areas, traditional algorithms perform poorly, resulting in incomplete reconstruction results.

[0007] With the widespread application of deep learning in the field of computer vision, especially the proven ability of convolutional neural networks in extracting multi-scale and multi-dimensional features of images, researchers have begun to introduce deep learning into multi-view stereo tasks. By using deep learning models, the reconstruction problems of non-Lambertian areas and low-texture and repeated-texture areas can be better handled, improving the accuracy and completeness of 3D reconstruction. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention proposes a multi-view stereo 3D reconstruction algorithm based on a new MHM model. The present invention improves and optimizes the MHM model based on casmvsnet as the basic network, which has relatively improved the rate, accuracy and completeness of 3D reconstruction, and is more convenient to deploy and run.

[0009] Technical solution: The present invention discloses a multi-view stereoscopic 3D reconstruction algorithm based on a new MHM model. The 3D reconstruction algorithm is as follows:

[0010] Step 1: The present invention proposes to use an improved loss function algorithm. By combining the L1 Loss plus the L2 Loss loss function, a new SmoothL1 loss function is obtained. This loss function combines the advantages of both and solves the disadvantages of both. At this time, there will be no gradient explosion and the model can be flexibly learned.

[0011] Step 2: The D-RFP network is formed by fusing the RFP feature extraction network with deep separable convolution and the SlowFast network with multi-cost volume aggregation module to obtain the MHM network model structure.

[0012] Step 2.1: In the D-RFP feature extraction network, the ResNet backbone network B is modified to take x and R(f) as input. ResNet has 4 stages, each consisting of multiple similar modules. Only the first module of each stage is changed, which calculates a 3-layer feature and adds it to the feature calculated by the shortcut. In order to use the feature R(f), another convolutional layer is added with a kernel size of 1. The weights of this layer are initialized to 0 to ensure that it will not have any impact when loading the pre-trained weights.

[0013] f i =F i (f i+1 ,x i ),x i =B i (x i-1 )

[0014] Step 2.2: Use Atrous Spatial Pyramid Pooling (ASPP) to implement the connection module R, which takes the feature fi^t as input and transforms it into RFP features. In this module, there are 4 parallel branches, which take fi^t as input, and then concat along the channel dimension to get the final output R. Three of them use a convolution layer and a ReLU layer, and the number of output channels is 1 / 4 of the number of input channels. The last branch uses a global average pooling layer to compress the features, followed by a 1×1 convolution layer and a ReLU layer to transform the compressed features into features of 1 / 4 size (channel dimension). Finally, these features are rescaled and concat with the features of the other three branches.

[0015] f i =F i (f i+1 ,x i ),x i =B i (x i-1 ,R i (f i ))

[0016] Step 2.3: Add depthwise convolution DSC. The calculation of Depthwise Convolution is very simple. It uses a convolution kernel for each channel of the input feature map, and then concatenates the outputs of all convolution kernels to get its final output. Compared with ordinary convolution, the ratio of DSC calculation to ordinary convolution is:

[0017]

[0018] Step 2.4: As an alternative to ordinary convolution, DSC has the biggest advantage of high computational efficiency. Therefore, it is very common to use DSC to build lightweight models. However, the high efficiency of DSC comes at the cost of low precision. At present, designing high-efficiency and high-precision DSC variant modules is still one of the hot research directions.

[0019] Step 3: Multiple feature bodies are grouped and vector inner products are performed to form multiple cost bodies. The weight information of different cost bodies is learned through an MLP, and the final cost body is obtained by weighted summation. Compared with the csaMVSNet network, the feature bodies are first grouped, the cost bodies are constructed from each perspective, and then the final cost body is formed by aggregation. This method of calculating the similarity of related features provides an effective cost body representation, avoids excessive information loss, and thus improves the accuracy of the overall network.

[0020] Beneficial Effects

[0021] In the present invention, we propose a model called MHM. The present invention utilizes the RFP network fused with deep separable convolution to form a D-RFP feature extraction network, and adds a new MHM model with a multi-cost volume aggregation module to preliminarily extract the multi-scale features of the feature image, which can not only obtain local semantic information but also obtain the expression of global semantic information. Then, the hole convolution layer and pooling layer with different sampling frequencies are introduced to obtain feature information under different receptive fields, and the extracted feature extraction and fusion of multiple FPN networks are fully utilized to enhance the feature expression ability. Then, through the multi-cost volume aggregation module, multiple feature maps are homographically transformed to obtain feature volumes, and multiple feature volumes are grouped and vector inner products are performed to form multiple cost volumes. The weight information learned by MLP is finally weighted and summed to obtain the final cost volume, providing an effective cost volume representation, avoiding excessive information loss, thereby improving the accuracy of the overall network, and updating the loss function of the original basic model, solving the shortcomings of the original loss function's fixed and inflexible gradient. The new loss function will not have gradient explosion and can be learned more flexibly. With the help of deep learning, compared with the traditional multi-view stereo three-dimensional reconstruction technology, the reconstruction efficiency of the present invention is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical description and implementation effects of the present invention, the accompanying drawings will be described in detail below.

[0023] See also Figure 1 , which is an overall flow chart of the three-dimensional reconstruction algorithm of the present invention.

[0024] See also Figure 2 , which is a structural diagram of the MHM model of the three-dimensional reconstruction algorithm of the present invention. DETAILED DESCRIPTION

[0025] The present invention is described in detail below based on the accompanying drawings and formulas. The specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0026] See also Figure 1 , which is a schematic diagram of the multi-view stereoscopic 3D reconstruction MHM model process provided by the present invention. After the multi-view images are captured by a camera, the steps are:

[0027] Step 1: Input a reference image and n source images, calibrate the camera to obtain the camera parameters, and use the device to collect images from various angles of the scene to determine the internal and external parameters of each view.

[0028] Step 2: After loading the semantic information, input it into the D-RFP feature extraction network, and extract and fuse the features through recursion. The sampling interval of the three dilated convolution operations of the ASPP module in the D-RFP network is 6.12.18. The upsampling operation is performed by bilinear interpolation, and the feature fusion operation is implemented using a 1*1 convolution layer to obtain the feature map.

[0029] Step 3: In the entire multi-scale feature extraction network D-RFP, N images with a resolution of H×W are input, including the reference view I0 and N-1 source views, and three feature maps of different sizes with resolutions of H / 2×W / 2, H / 4×W / 4, and H / 8×W / 8 are output. Then, the improved module can be used to perform depth estimation on the reference view from top to bottom and from coarse to fine, and the depth map of the reference view is output.

[0030] Step 4: The stereo 3D reconstruction method takes the feature maps of the reference view and the source view as input, first samples the depth values ​​within the set depth range, and uses the local plane consistency constraint to perform depth matching and propagation in an adaptive manner. Then, the method uses homography to map the features of the source view to the coordinate system of the reference view, and performs group-based similarity calculation together with the reference view features enhanced by the convolutional attention mechanism.

[0031] After obtaining the group similarity, the method further calculates the pixel-level view weights to aggregate the similarities from multiple source views. Then, combined with the group weight information, the group similarities of multiple views are synthesized into a cost volume through a simplified 3D convolutional network. This process effectively integrates information from different perspectives and provides accurate depth information for the final 3D reconstruction.

[0032] Step 5: When dealing with multiple feature maps with different resolutions, the process of constructing the cost volume follows the same steps. First, a series of parallel depth planes need to be created. Next, these feature maps are projected onto the corresponding depth planes using homography to form feature volumes. Finally, the final cost volume is constructed by aggregating these feature volumes. Once the cost volume is obtained, it is regularized to reduce the impact of noise in the feature map and reduce the resulting error.

[0033] Step 6: The input feature map of the first stage has the smallest scale but contains the depth information of the entire scene. Inspired by this, this paper chooses to preset the depth range of the feature map of the first stage, constructs the cost volume for the feature map of this stage, and selects the depth information obtained in the first stage as the preset depth range of the second stage to construct the cost volume of the second stage. Similarly, the depth information obtained in the second stage is selected as the preset depth range of the third stage to construct the cost volume of the third stage. Thus, the MHM model constructs cost volumes and depth maps of different scales in a coarse-to-fine manner.

[0034] Step 7: Point cloud depth fusion. The preliminary depth map is further refined through the depth map refinement network. The depth estimation of all views is completed through the above steps to obtain the depth map, and the depth map is fused to generate the spatial point cloud 3D expression of the scene.

[0035] Those skilled in the art can understand that the above is only a preferred example of the invention and is not intended to limit the invention. Although the invention is described in detail, those skilled in the art can still modify the technical solutions recorded in the examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A multi-view 3D reconstruction algorithm based on a new MHM model, characterized in that: The multi-view 3D reconstruction algorithm comprises the following steps: Step 1: Determine the required depth map reference image and n original images, and use the required camera internal and external parameters. Step 2: Pass it through the D-RFP network structure with deep separable convolution to obtain multi-scale feature maps and generate corresponding feature maps. Step 3: Then, the feature map is passed through the multi-cost volume aggregation module, and multiple feature maps are transformed homographically to synthesize multiple feature volumes. Then, multiple feature volumes are grouped and vector inner products are performed to form multiple cost volumes. Different weight information is learned through an MLP, and finally the weighted sum is obtained to obtain the final cost volume. Step 4: The initially constructed cost volume is regularized using the 3D-UNet network to generate a probability volume. Subsequently, a softmax function is applied to the probability volume in the first dimension to calculate the depth probability distribution of each pixel. Next, the depth map corresponding to the reference image is obtained by calculating the expected value of these probabilities. Finally, the depth map is fed into a refinement network for further optimization to improve the accuracy and resolution of the depth estimation. Step 5: Perform the above operations on the images in the corresponding data set in sequence to obtain the depth map corresponding to each image, and then obtain the point cloud image of the three-dimensional model through depth fusion. Finally, the three-dimensional reconstruction requirements of the target are realized through this point cloud image.

2. The MHM model algorithm for multi-view 3D reconstruction according to claim 1 is characterized in that: The feature extraction network FPN was changed on the original basic model casmvsnet and replaced with RFP. The recursive feature pyramid structure uses the feature extraction network of the FPN structure multiple times in a recursive way to extract and fuse features, so as to enhance the expressive power of the extracted features. However, the RFP structure will greatly increase the amount of calculation and parameters in the feature advance process, so the depth-wise separable convolution is added to effectively reduce the number of parameters of the convolution operation and reduce the memory required for running the model.

3. The multi-view three-dimensional reconstruction model according to claim 1, characterized in that: Depthwise separable convolution reduces the amount of computation and model parameters by decomposing the traditional convolution operation into two smaller operations: channel-wise convolution and pointwise convolution. In the channel-wise convolution step, each convolution kernel is independently applied to the corresponding channel of the input feature map without mixing the information of any other channels, thereby keeping the number of channels of the output feature map the same as the input. In the pointwise convolution step, a small 1×1 convolution kernel is used to process the output of the previous step to achieve interaction between feature maps and allow the number of feature channels of the output to change. By adding depthwise separable convolution to the original basic model, the change in structure can reduce the amount of computation when the model is running, thereby achieving the memory requirements when running the model.

4. The multi-view 3D reconstruction MHM model algorithm according to claim 1, characterized in that: In the step of constructing the cost volume, the algorithm used by the original basic model is to extract feature maps from the input reference image and a series of source images, and convert these feature maps into feature volumes corresponding to the camera cone direction. Then, it uses a variance-based algorithm to measure the similarity between feature volumes and generates a cost volume by merging them. In the present invention, we adopt a different strategy: first, multiple feature volumes are grouped, and then vector inner product operations are performed in each group. Next, a multi-layer perceptron (MLP) is used to learn the weight information of different cost volumes. Finally, the final cost volume is obtained by weighted summation. Compared with the original basic model, the present invention first groups the feature volumes, constructs a cost volume separately for each perspective, and finally aggregates them to form the final cost volume. This calculation algorithm based on the similarity of related features provides an effective cost volume representation, reduces information loss, and thus improves the accuracy of the entire network.

5. The multi-view 3D reconstruction MHM model algorithm according to claim 1, characterized in that: The original basic model is the L1 Loss loss function of three scales. This loss function is the mean absolute error. The disadvantage is that the gradient of MAE is fixed. Whether it is a large model or a small model, the fixed gradient will cause a big difference. Therefore, this loss function is not flexible and will affect the learning ability of the model. Therefore, the present invention proposes to use an improved loss function algorithm. By combining the L1 Loss plus the L2Loss loss function, a new SmoothL1 loss function is obtained. This loss function combines the advantages of both and solves the disadvantages of both. At this time, there will be no gradient explosion and the model can be flexibly learned.

Images