Three-dimensional reconstruction method based on self-attention mechanism
By introducing a dual attention network based on self-attention mechanism in the three-dimensional reconstruction technology, the problem of insufficient feature extraction of convolutional neural networks in weak texture areas is solved, and efficient three-dimensional reconstruction in complex scenarios is achieved, and more accurate depth is obtained Graphs and more stable results.
Patent Information
- Application Number
- CN202310791448.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-30
- Publication Date
- 2024-12-31
AI Technical Summary
Existing three-dimensional reconstruction techniques are difficult to obtain accurate depth information in weak textures and complex scenarios, especially due to insufficient feature extraction caused by the local receptive field of convolutional neural networks, resulting in incomplete or fuzzy reconstruction results.
Using a dual attention network based on deformable convolution, combining self-attention mechanism and block matching algorithm, feature extraction is enhanced through the spatial attention module and channel attention module, context features are captured and long-distance correlation is established in weak texture areas. Supervised training is performed using focus loss and minimum absolute error.
It improves the expressiveness and robustness of three-dimensional reconstruction in weak texture areas, obtains clearer object boundaries, and adapts to multi-view stereoscopic visual semi-global matching tasks, with fast running speed, small resource consumption and stable effect.
Smart Images
Figure CN119228982A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, mainly to the field of three-dimensional reconstruction. Specifically, the present invention provides a three-dimensional reconstruction method based on a self-attention mechanism. Background Art
[0002] Vision-based 3D reconstruction is an important part of the computer vision field. It refers to the use of relevant instruments to obtain the two-dimensional image data information of an object, and then analyze and process the acquired data information through computer vision technology, and finally reconstruct the contour information of the object surface in the real environment to obtain the 3D model of the object. In recent years, with the rapid development of artificial intelligence and the high-quality and diversified needs in the field of human-computer interaction, 3D reconstruction technology has been rapidly developed and widely used as one of the core technologies in the fields of augmented reality, mixed reality, robot navigation, and autonomous driving.
[0003] From the perspective of technical classification, 3D reconstruction can be divided into active vision-based technology and passive vision-based technology. Active means that the sensor actively irradiates the object with a signal, and then obtains the 3D information of the object by analyzing the returned signal. There are mainly laser scanning method, structured light, TOF and radar methods. The structured light method relies on the projector to project the coded structured light onto the photographed object, and then the camera shoots and captures the changes in the size and shape of the coded pattern on the photographed object, and then converts it into depth information through the calculation unit to obtain the 3D contour information of the object. The TOF time of flight method relies on continuously sending light pulses to the target and calculating the distance of the target based on the time or phase difference when the sensor receives the returned light. The passive method relies on geometric principles to calculate based on parallax. According to the number of cameras, it can be divided into monocular vision method, binocular vision method and multi-view stereo vision. Since the monocular vision method only uses a single view, it needs to use empirical knowledge to supplement the information. Binocular vision draws on the human visual system, and multi-view vision is an extension of binocular vision. On its basis, one or more perspectives are added to obtain multiple pairs of images of an object at different angles. Compared with active methods, passive methods obtain depth values by matching 2D images, which is more economical and easier to implement. In addition, images contain real scene information such as texture and lighting, which is also an indispensable condition for reconstructing 3D model details.
[0004] Three-dimensional reconstruction methods based on multi-view stereo vision are divided into traditional algorithms and algorithms based on deep learning. Traditional MVS methods use photometric consistency to calculate dense three-dimensional information. Although these methods have high accuracy in ideal Lambertian scenes, some complex situations such as weak textures, reflections, and mirrors will make reconstruction difficult or incomplete. The successful application of deep learning in the field of computer vision in recent years has made people begin to consider introducing deep learning into the field of three-dimensional reconstruction. Learning-based methods can introduce global semantic information such as mirror priors or reflection priors to make matching more robust, thereby providing solutions to problems that traditional methods cannot solve. In an environment where convolutional neural networks continue to make major breakthroughs in the field of computer vision, researchers have introduced convolutional neural networks as a method of feature extraction and feature fusion into the process of depth map calculation. For example, the MVS network based on the plane scanning algorithm uses CNN for feature extraction, and then constructs a cost volume through a differentiable homography transformation. Finally, the network is used to regularize the cost volume and regress to obtain the depth map of the corresponding perspective of the scene. In addition, the researchers also combined deep learning with the traditional PatchMatch algorithm, using the local correlation of the image to achieve fast block matching. This method does not require three-dimensional convolution to regularize the cost volume, saves memory, shortens the running time, and is more suitable for devices with limited resources. However, due to the local receptive field of convolutional neural networks, using only convolutional neural networks for feature extraction may not obtain enough contextual information, which may easily lead to reconstruction loss problems on low-texture surfaces.
[0005] The MVS task is different from the global field of view of the image classification task and the local field of view of the target detection task. Instead, it expects a local field of view at strong textures and expands the field of view at weak textures. In order to adapt to such tasks, the present invention includes a three-dimensional reconstruction method based on a dual attention network based on deformable convolution, which applies the self-attention mechanism to the feature extraction link of the MVS task, enhances contextual association, establishes long-range correlation in low-texture areas, and captures contextual feature dependencies, so that the network can perform better on reflective and weakly textured surfaces. Summary of the invention
[0006] The present invention aims to make up for the problem of lack of context features caused by the existing technology that only uses convolutional neural networks in the feature extraction process. With scientific thinking and rigorous attitude, a 3D reconstruction method based on self-attention mechanism is proposed. The present invention designs a dual attention network based on deformable convolution, captures context feature dependencies on local features, and establishes long-distance correlations in weak texture areas. It is well adapted to the task of semi-global matching of multi-view stereo vision. It is a 3D reconstruction method with fast running speed, low resource consumption, excellent and stable effect.
[0007] The technical solution of the present invention is as follows:
[0008] A 3D reconstruction method based on self-attention mechanism and block matching, characterized by a feature extraction network that integrates a self-attention module based on deformable convolution, including a spatial attention module and a channel attention module based on deformable convolution. The network is based on a block matching algorithm, including feature extraction using a feature pyramid network with a self-attention mechanism, adaptive propagation to obtain a deep hypothesis plane, cost volume construction based on homography transformation and adaptive cost aggregation, and supervised training using focal loss and minimum absolute error.
[0009] For weak texture areas, there are fewer feature points and the changes are not obvious. It is expected to expand the field of view to reduce the ambiguity of depth calculation; for strong texture areas, such as object boundaries, local field of view is expected to obtain clearer object boundaries. A dual attention module is added to the feature extraction network. The spatial attention module and the channel attention module are used to aggregate context-related features at the lowest resolution, thereby avoiding the limitations of the local field of view of the convolutional neural network in feature extraction. The obtained features well contain the context feature dependency, which facilitates subsequent regression to obtain a more accurate depth map with better performance on weak texture surfaces.
[0010] This method first collects and processes data, uses cameras to shoot objects with overlap, and inputs the images into COLMAP to obtain camera poses and sparse point clouds.
[0011] Then the camera pose and the captured image are input into the optimized feature extraction network. First, the image goes through the bottom-up process of the feature pyramid network to obtain feature maps of 1 / 2, 1 / 4, and 1 / 8 of the original image. The 1 / 8 feature map goes through the designed dual attention module before entering the top-down process. The context information is aggregated by the spatial attention module and the channel attention module. The calculation results of the two modules are fused, and the obtained feature map is used as the input of the first stage of the block matching network. The feature map of the subsequent stage is obtained by adding the interpolation of the feature map of the previous stage to the feature map obtained in the bottom-up process. The feature maps of different stages obtained in this way contain rich context information. At the same time, adding the self-attention module at the lowest resolution can greatly reduce the amount of calculation and increase the operation efficiency. The feature map obtained by the feature extraction network obtains the depth hypothesis plane through adaptive propagation, constructs the cost volume based on the homography transformation, and obtains the depth map through adaptive cost aggregation regression. Considering that effective supervision cannot be obtained when only using L1 loss for weak texture areas, the loss function is replaced by focal loss to strengthen the constraints on the blurred area and improve the robustness of the network.
[0012] The present invention has the following beneficial effects:
[0013] The present invention provides a 3D reconstruction method based on the self-attention mechanism, which enables the network to model rich contextual information, establish long-distance correlation in weak texture areas, and better obtain clear boundaries at the boundaries of objects. It not only adapts to the task of semi-global matching of multi-view stereo vision, but also improves the robustness of the network, runs faster, consumes less resources, and has excellent and stable effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 Workflow diagram of the 3D reconstruction method based on self-attention mechanism and block matching;
[0015] Figure 2 The structure diagram of the dual attention module designed by the present invention; DETAILED DESCRIPTION
[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0017] The operating environment required for this example is pytorch1.11, ubuntu18.04, cuda11.3, and training on NVIDIA RTX A4000 server. The batch size used for training is 4, the number of iterations is 8, the initial learning rate is set to 0.001, and the learning rate is reduced as the number of iterations increases. It takes 15 hours to complete a whole task.
[0018] This example provides a 3D reconstruction method based on the self-attention mechanism. The workflow diagram is as follows: Figure 1 As shown, the first step is to collect and process the data. The object is photographed with overlapping cameras, and the images are input into COLMAP to obtain the camera pose and sparse point cloud.
[0019] Then the neural network structure is optimized. First, N input images of size W×H are input into the FPN network, including a reference image and N-1 source images. The FPN network is a pyramid structured network, so it is able to obtain hierarchically extracted features at multiple resolutions. The input image first passes through two 3×3 convolution kernels for feature extraction to obtain a feature map with 8 channels and unchanged size. After that, each layer passes through a convolution layer with a convolution kernel size of 5×5 and a step size of 2 and two convolution layers with a convolution kernel size of 3×3 and a step size of 1 to obtain feature maps of sizes W / 2×H / 2, W / 4×H / 4, and W / 8×H / 8, respectively. Apply the following on the feature map of size W / 8×H / 8: Figure 2 The dual attention module shown in Figure 1 aggregates context features as the first layer input of the block matching network, and obtains feature maps at a more refined stage through interpolation and fusion. Figure 2As shown in the figure, it includes a position attention module and a channel attention module. The position attention module first passes the feature map through three 3×3 deformable convolution layers with unshared weights to obtain three new feature maps, which can be regarded as Q, K and V of the self-attention mechanism. Then Q and K are adjusted to the size of C×N, where N=H / 8×W / 8, Q is transposed and multiplied with K, and the result is then subjected to a softmax operation to obtain a feature map S of size N×N. V is then adjusted to C×N and multiplied with the transposed S. The result is adjusted back to the size of C×H / 8×W / 8, multiplied by a coefficient to be learned, and added to the original feature map. The result thus obtained encodes rich contextual information while retaining the abstract features extracted by the feature pyramid network. The channel attention module is used to explicitly model the interdependence between channels and is a module parallel to the position attention module. The channel attention module directly operates on the W / 8×H / 8 feature map. First, the feature map is resized to C×N, then multiplied by the transpose of the feature map itself, and then subjected to a softmax operation to obtain a feature map P of size C×C. The transpose of P is multiplied by the feature map of size C×N, and finally the result is resized to C×W / 8×H / 8, multiplied by a coefficient to be learned, and added to the original feature map. The results of the position attention module and the channel attention module are added and feature fused through a convolutional layer to obtain the final output result as the input of the first stage of the block matching network. The feature map of the subsequent stage is obtained by adding the interpolation of the feature map of the previous stage to the feature map obtained in the bottom-up process.
[0020] The feature map obtained by the optimized feature extraction network then enters the block matching network to obtain depth hypotheses through adaptive propagation. Then, the cost volume is constructed and the depth map is regressed using homography transformation and adaptive cost aggregation. Adaptive propagation and adaptive cost aggregation use deformable convolution to obtain neighborhood coordinates, which can achieve the dispersion of sampling points in weak texture areas to a larger space, while the sampling points at the edge of the object are gathered on the surface of the object, thereby reducing the uncertainty of the estimation. Since the network is a cascade structure, the depth prediction value obtained by the final regression also has different resolutions at different stages. In the first three stages, the focus loss is calculated between the depth prediction value and the true value to strengthen the constraints on the blurred area. In the last stage, the minimum absolute error is calculated between the depth prediction value and the true value. The camera pose and the captured image are input into the network, and the predicted depth map is fused to obtain the three-dimensional point cloud of the object.
[0021] Obviously, the described embodiment is only one possible embodiment of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of the present invention.
Claims
1. A 3D reconstruction method based on self-attention mechanism and block matching, characterized in that The feature extraction network that integrates the self-attention module based on deformable convolution includes the spatial attention module and the channel attention module based on deformable convolution. The network is based on the block matching algorithm, including the feature pyramid network using the self-attention mechanism for feature extraction, adaptive propagation to obtain the deep hypothesis plane, cost volume construction based on homography transformation and adaptive cost aggregation, and supervised training using focal loss and minimum absolute error. For weak texture areas, there are fewer feature points and the changes are not obvious. It is expected that the field of view can be expanded to reduce the ambiguity of depth calculation; For strong texture areas, such as object boundaries, local vision is expected to obtain clearer object boundaries. A dual attention module is added to the feature extraction network, and context-related features are aggregated through the spatial attention module and the channel attention module at the lowest resolution, thereby avoiding the limitations of the local vision of the convolutional neural network in feature extraction. The obtained features well contain the context feature dependency, which facilitates subsequent regression to obtain a more accurate depth map with better performance on weak texture surfaces. The specific steps include: The first step is to collect and process the data. The camera takes overlapping photos of the object and inputs the images into COLMAP to obtain the camera pose and sparse point cloud. Then the camera pose and the captured image are input into the optimized feature extraction network. First, the image goes through the bottom-up process of the feature pyramid network to obtain feature maps of 1 / 2, 1 / 4, and 1 / 8 of the original image. The 1 / 8 feature map goes through the designed dual attention module before entering the top-down process. The context information is aggregated by the spatial attention module and the channel attention module. The calculation results of the two modules are fused, and the obtained feature map is used as the input of the first stage of the block matching network. The feature map of the subsequent stage is obtained by adding the interpolation of the feature map of the previous stage to the feature map obtained in the bottom-up process. The feature maps of different stages obtained in this way contain rich context information. At the same time, adding the self-attention module at the lowest resolution can greatly reduce the amount of calculation and increase the operation efficiency. The feature map obtained by the feature extraction network obtains the depth hypothesis plane through adaptive propagation, constructs the cost volume based on the homography transformation, and obtains the depth map through adaptive cost aggregation regression. Considering that effective supervision cannot be obtained when only using L1 loss for weak texture areas, the loss function is replaced by focal loss to strengthen the constraints on the blurred area and improve the robustness of the network. Finally, the depth map calculated by the network is fused to obtain the three-dimensional point cloud of the object.
2. The three-dimensional reconstruction method based on the self-attention mechanism according to claim 1, characterized in that: Adapting to the needs of 3D reconstruction tasks, which expect local vision in strong textures and expand the field of view in weak textures, the self-attention mechanism is applied to the feature extraction link of the MVS task to enhance contextual association, establish long-distance correlation in low-texture areas, and capture contextual features. Dependence enables the network to perform better on reflective and weakly textured surfaces.