Unsupervised three-dimensional reconstruction method based on comparative learning

Abstract

The invention discloses an unsupervised three-dimensional reconstruction method based on comparative learning. The method belongs to the field of computer artificial intelligence, and can solve the problems of difficulty in obtaining a labeled data set with supervision and low precision without supervision. The method comprises the following steps: selecting a group of images of a public data set, and inputting the images and corresponding camera poses into an unsupervised three-dimensional reconstruction network; obtaining a depth map according to the unsupervised three-dimensional reconstruction network; and performing three-dimensional reconstruction in combination with the camera pose to obtain an object three-dimensional model. According to the method, feature extraction is performed through the selective structured state space model, the scene-level comparison network is improved, and the more reasonable dynamic loss function is used, so that the object depth map can be obtained more accurately, the three-dimensional reconstruction effect in the prior art is optimized, and the three-dimensional reconstruction effect of the target image is improved.

Description

Technical Field

[0001] The invention belongs to the field of computer artificial intelligence, and in particular relates to an unsupervised three-dimensional reconstruction method based on contrastive learning. Background Art

[0002] Multi-view stereo (MVS) is an important task in computer vision and image processing, and is widely used in artificial intelligence, autonomous driving, cultural heritage protection, virtual reality and other fields. Traditional 3D reconstruction methods usually rely on geometric feature matching and disparity estimation, but these methods are sensitive to adverse conditions such as noise and occlusion, and are not robust enough to changes in perspective. With the rapid development of deep learning technology, 3D reconstruction methods based on deep learning have shown significant advantages in accuracy and robustness. However, most existing deep learning methods rely on a large amount of labeled data for supervised training, which not only increases the difficulty of data acquisition, but also limits the generalization ability of the model in different scenarios.

[0003] Using unsupervised methods, the MVS network can be trained without a labeled dataset. However, due to weak texture and reflection factors, the assumption that relies on photometric consistency sometimes fails, resulting in low reconstruction accuracy. At the same time, the accuracy of the 3D reconstruction model depends on a good feature extraction process. Weak texture areas lack obvious features and rely more on feature context information at a longer distance for auxiliary expression. Traditional CNN uses a fixed-size convolution kernel for feature extraction, which may result in blurred features extracted from weak texture areas, thereby reducing the robustness of the match. Summary of the invention

[0004] The present invention aims to solve the technical problems in existing three-dimensional reconstruction methods, such as difficulty in obtaining labeled data sets, insufficient generalization ability of the model in different scenarios, reliance on intensive feature extraction and fusion processes, and difficulty in capturing long-distance dependencies and global context information.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is as follows:

[0006] An unsupervised 3D reconstruction method based on contrastive learning comprises the following steps:

[0007] S1, input the dataset image and camera pose into the comparative learning unsupervised 3D reconstruction network;

[0008] S2, use contrastive learning unsupervised 3D reconstruction network to obtain depth image;

[0009] S3. Using geometric features and photometric consistency evaluation, remove depth points with low confidence and use the depth map to generate the final 3D point cloud.

[0010] 1. In the above technical solution, step S1 is specifically:

[0011] S11, set the input image number N to 5, adjust the image size and crop it, and use the Adam optimizer to optimize the model;

[0012] S12. Convert the known internal and external parameters of the camera into matrix forms respectively and input them into the unsupervised 3D reconstruction network.

[0013] 2. In the above technical solution, step S2 is specifically:

[0014] S21, using the selective structured state space model to extract two-dimensional pixel feature expressions at different scales;

[0015] S22. Use image-level and scene-level contrast learning branches to generate contrast consistency loss to enhance the robustness of the model in low-texture areas, repeated textures, and view dependence.

[0016] 3. In the above technical solution, step S21 is specifically:

[0017] S21-1. Use the state space model to capture the long-distance pixel dependencies. In the state space model, the hidden state h(t) is modeled by a linear ordinary differential equation. A, B, C, and D are the parameter matrices of the system, x(t) represents the features of the input image, h(t) is the hidden state, and y(t) is the output feature. The design of the state space system enables the network to effectively capture the dependencies between long-distance pixels. In reflective surfaces and low-texture areas, effective image features can be extracted. The modeling formula is:

[0018] h′(t)=Ah(t)+Bx(t)

[0019] y(t)=Ch(t)+Dx(t)

[0020] S21-2, the image is processed recursively to further optimize the dependency modeling of long-distance pixels, and the formula is:

[0021] h n+1 =Ah n +Bx n

[0022] y n =Ch n +Dx n

[0023] 4. In the above technical solution, step S22 specifically includes:

[0024] S22-1. Use a binary mask to randomly block some pixels of the source image to generate image-level contrast samples, simulate the failure of local photometric consistency, and guide the model to learn more contextual information by maximizing the similarity of depth estimation.

[0025] S22-2, using the context-sensitive attention LUF module to improve the model's ability to process global information and local details in the scene-level comparison stage;

[0026] S22-3. Randomly select multiple source images to form scene-level comparison samples. By comparing the depth estimation results of the branch with the reference branch, simulate the perspective dependence effect and occlusion problem, and use a dynamic scene-level loss function to enable the model to learn stronger feature representation capabilities.

[0027] 5. In the above technical solution, step S22-2 is specifically:

[0028] S22-2-1. Use discrete Fourier transform to convert spatial features into frequency domain, and use inverse Fourier transform to convert frequency features back to spatial domain. f[m,n] represents the input spatial domain matrix, and F[k,l] represents the spectrum distribution in Fourier domain. The LUF module converts image features into frequency domain to capture global spatial patterns and textures. The formulas for discrete Fourier transform and inverse Fourier transform are:

[0029]

[0030] S22-2-2. Use frequency range transformation and inverse transformation, combined with dynamic jump connection, to improve the ability to capture frequency information, which can effectively reduce spectrum drift and enhance the fusion of frequency features and spatial features. F[k,l] is the spectrum converted by the spatial matrix f[m,n]. c is the interpolation rate, which corresponds to the expansion rate in convolution. The formulas for frequency range transformation and inverse transformation are:

[0031]

[0032] S22-2-3. Use the adaptive clipping method to normalize the iDFT output and then limit its value to between the maximum and minimum values ​​of the input feature. The normalization formula is:

[0033]

[0034] S22-2-4. The features of different scales are weighted through the selective attention mechanism. σ is the Sigmoid function, which is used to generate attention weights. The attention weight formula is:

[0035]

[0036] S22-2-5. The weights are fused with the convolution features to obtain the weighted attention features. The weighted attention feature formula is:

[0037]

[0038] S22-2-6. Multiply the input feature by the attention feature element by element to obtain the output feature. The output feature formula is:

[0039] Y=X·S

[0040] 6. In the above technical solution, step S22-3 is specifically:

[0041] S22-3-1. According to the complexity of different regions, different weights are assigned to the loss function. R and D IC Represent the depth estimation of the original image and the occluded image, M c is the confidence mask, W is the dynamic weight, and the dynamic scene level loss function formula is:

[0042]

[0043] 7. In the above technical solution, step S3 is specifically:

[0044] S31, back-projecting the pixels on the depth map through the camera internal and external parameters to convert them into points in three-dimensional space. If the depth values ​​of the points reprojected from different viewing angles are consistent with the original depth values, then these points are considered to be consistent, and these high-confidence points are retained;

[0045] S32. Project the depth values ​​of multiple perspectives into a point cloud, integrate the depths under different perspectives, and finally form a unified three-dimensional point cloud.

[0046] The beneficial effects of the present invention are:

[0047] (1) The present invention introduces the selective structured state space model into the framework of MVS and proposes a pyramid structure feature extraction module based on the selective structured state space model. Combined with the selective structured state space model, it filters redundant information, highlights key features, and reduces noise interference through dynamic updating of the state space. At the same time, it uses state vectors to represent information at different levels, thereby enhancing the ability to process long-distance dependencies and obtain global context information, and has better feature extraction capabilities for weak texture areas.

[0048] (2) This paper proposes a new contextual attention mechanism, which enhances the image-level detail expression capability through multi-scale convolution and cross-dimensional information integration in the image-level contrastive learning stage.

[0049] (3) The present invention proposes a new dynamic scene-level loss function, which assigns different weights to each pixel in the loss function according to the complexity or uncertainty of different regions, thereby improving the depth estimation accuracy of these regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of the unsupervised 3D reconstruction method based on contrastive learning of the present invention;

[0051] Figure 2 This is a framework diagram of the unsupervised 3D reconstruction method based on contrastive learning according to the present invention;

[0052] Figure 3 This is the depth map result of the present invention tested under the public data set;

[0053] Figure 4 This is the 3D reconstruction effect tested by the present invention under the public data set.

Claims

1. An unsupervised 3D reconstruction method based on contrastive learning, characterized in that: The following steps are involved: S1, input the dataset image and camera pose into the comparative learning unsupervised 3D reconstruction network; S2, use contrastive learning unsupervised 3D reconstruction network to obtain depth image; S3. Using geometric features and photometric consistency evaluation, remove depth points with low confidence and use the depth map to generate the final 3D point cloud.

2. The unsupervised 3D reconstruction method based on contrastive learning according to claim 1, characterized in that: The step S1 is specifically as follows: S11, set the input image number N to 5, adjust the image size and crop it, and use the Adam optimizer to optimize the model; S12. Convert the known internal and external parameters of the camera into matrix forms respectively and input them into the unsupervised 3D reconstruction network.

3. The unsupervised 3D reconstruction method based on contrastive learning according to claim 1, characterized in that: The step S2 is specifically as follows: S21, using the selective structured state space model to extract two-dimensional pixel feature expressions at different scales; S22. Use image-level and scene-level contrast learning branches to generate contrast consistency loss to enhance the robustness of the model in low-texture areas, repeated textures, and view dependence.

4. The unsupervised 3D reconstruction method based on contrastive learning according to claim 3, characterized in that: In step S21, the selective structured state space model is used to enhance feature extraction specifically as follows: S21-1. Use the state space model to capture the long-distance pixel dependencies. In the state space model, the hidden state h(t) is modeled by a linear ordinary differential equation. A, B, C, and D are the parameter matrices of the system, x(t) represents the features of the input image, h(t) is the hidden state, and y(t) is the output feature. The design of the state space system enables the network to effectively capture the dependencies between long-distance pixels. In reflective surfaces and low-texture areas, effective image features can be extracted. The modeling formula is: h′(t)=Ah(t)+Bx(t) y(t)=Ch(t)+Dx(t) S21-2, the image is processed recursively to further optimize the dependency modeling of long-distance pixels, and the formula is: h n+1 =Ah n +Bx n y n =Ch n +Dx n 5. The unsupervised 3D reconstruction method based on contrastive learning according to claim 3, characterized in that: In step S22, the image-level and scene-level contrast learning branches are used to enhance the model robustness as follows: S22-1. Use a binary mask to randomly block some pixels of the source image to generate image-level contrast samples, simulate the failure of local photometric consistency, and guide the model to learn more contextual information by maximizing the similarity of depth estimation. S22-2, using the context-sensitive attention LUF module to improve the model's ability to process global information and local details in the scene-level comparison stage; S22-3. Randomly select multiple source images to form scene-level comparison samples. By comparing the depth estimation results of the branch with the reference branch, simulate the perspective dependence effect and occlusion problem, and use a dynamic scene-level loss function to enable the model to learn stronger feature representation capabilities.

6. The unsupervised 3D reconstruction method based on contrastive learning according to claim 5, characterized in that: In step S22-2, the context-sensitive attention module is used to improve the model's ability to process global information and local details: S22-2-1. Use discrete Fourier transform to convert spatial features into frequency domain, and use inverse Fourier transform to convert frequency features back to spatial domain. f[m,n] represents the input spatial domain matrix, and F[k,l] represents the spectrum distribution in Fourier domain. The LUF module converts image features into frequency domain to capture global spatial patterns and textures. The formulas for discrete Fourier transform and inverse Fourier transform are: S22-2-2. Use frequency range transformation and inverse transformation, combined with dynamic jump connection, to improve the ability to capture frequency information, which can effectively reduce spectrum drift and enhance the fusion of frequency features and spatial features. F[k,l] is the spectrum converted by the spatial matrix f[m,n]. c is the interpolation rate, which corresponds to the expansion rate in convolution. The formulas for frequency range transformation and inverse transformation are: S22-2-3. Use the adaptive clipping method to normalize the iDFT output and then limit its value to between the maximum and minimum values ​​of the input feature. The normalization formula is: S22-2-4. The features of different scales are weighted through the selective attention mechanism. σ is the Sigmoid function, which is used to generate attention weights. The attention weight formula is: S22-2-5. The weights are fused with the convolution features to obtain the weighted attention features. The weighted attention feature formula is: S22-2-6. Multiply the input feature by the attention feature element by element to obtain the output feature. The output feature formula is: Y=X·S 7. The unsupervised 3D reconstruction method based on contrastive learning according to claim 5, characterized in that: In step S22-3, the dynamic scene-level loss function enables the model to learn stronger feature representation capabilities: S22-3-1. According to the complexity of different regions, different weights are assigned to the loss function. R and D IC Represent the depth estimation of the original image and the occluded image, M c is the confidence mask, W is the dynamic weight, and the dynamic scene level loss function formula is:

8. The unsupervised 3D reconstruction method based on contrastive learning according to claim 1, characterized in that: In step S3, the depth map is fused to generate the final three-dimensional model as follows: S31, back-projecting the pixels on the depth map through the camera internal and external parameters to convert them into points in three-dimensional space. If the depth values ​​of the points reprojected from different viewing angles are consistent with the original depth values, then these points are considered to be consistent, and these high-confidence points are retained; S32. Project the depth values ​​of multiple perspectives into a point cloud, integrate the depths under different perspectives, and finally form a unified three-dimensional point cloud.

Images