An intelligent multi-view stereo matching method and device
By building the global context and three-dimensional perceived Transformer module, Transformer captures long-distance dependencies inside and outside the view, solving the problem of insufficient information exploration in multi-view stereo matching, and achieving higher quality depth map generation.
Patent Information
- Application Number
- CN202210919810.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-01
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2042-08-01
AI Technical Summary
The existing multi-view stereo matching method fails to effectively explore the global context information within the view and the three-dimensional consistency information between views, resulting in poor depth map quality.
Transformer is used to capture long-distance dependencies, build a global context Transformer module and a three-dimensional sense Transformer module based on polar line geometry, dig up the global context and three-dimensional consistency information in the view, and generate a depth map through differentiable deformation operations and cost regularization modules.
Improved the depth map quality of multi-view stereo matching, and demonstrated better performance over existing methods on multiple datasets through experiments.
Smart Images
Figure CN115222977B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of deep learning and depth estimation, and particularly to an intelligent multi-view stereo matching method and device. Background Art
[0002] As one of the basic tasks of computer vision, multi-view stereo matching can be widely applied in fields such as robot navigation, autonomous driving, and virtual reality. Multi-view stereo matching aims to estimate the dense depth of a scene based on multi-view images, thereby reconstructing the three-dimensional model of the scene. In the past decade, relevant scholars have conducted in-depth research on multi-view stereo matching, which has greatly promoted the development of this field.
[0003] Traditional multi-view stereo matching methods usually estimate the scene depth based on manually designed features and cost regularization methods. Tola et al. proposed an efficient multi-view stereo matching method that uses the DAISY descriptor for feature matching, effectively improving the quality of the depth map. Xu et al. improved the cost regularization process using structured region information and multi-scale geometric consistency to optimize the depth map. Campbell et al. designed a multi-view stereo matching method based on the optimization of discrete label Markov random fields, effectively reducing the outliers in the depth map. Although traditional methods have made great progress, they are limited by various problems such as poor robustness and low generalization, and it is difficult to be effectively applied in actual scenarios.
[0004] In recent years, multi-view stereo matching methods based on deep learning have gradually shown more superior performance than traditional methods. Yao et al. proposed a multi-view stereo matching method based on a neural network, which extracts multi-view features through a convolutional neural network to construct a matching cost volume, and infers the depth map based on a recurrent neural network to learn the cost regularization process. Wei et al. proposed an adaptive aggregation multi-view stereo matching method, which adaptively extracts features through deformable convolution and uses an attention mechanism to adaptively integrate multi-view matching costs to improve the quality of the depth map. Gu et al. proposed a multi-view stereo matching method from coarse to fine, which gradually infers the depth map by constructing a cascaded matching cost volume. However, existing methods usually use a convolutional neural network to extract the features of each view separately, and do not effectively explore the global context information within the view and the three-dimensional consistency information between views, so problems such as matching ambiguity and matching errors often occur, resulting in poor quality of the depth map.
[0005] Therefore, in multi-view stereo matching methods, how to effectively explore the global context information within the view and the three-dimensional consistency information between views is crucial for improving the quality of the depth map. Summary of the Invention
[0006] The present invention provides an intelligent multi-view stereo matching method and apparatus. The present invention utilizes the ability of Transformer to capture long-distance dependencies, explores the global context relationship within the view and mines the three-dimensional consistency information between views to achieve multi-view stereo matching, as described in detail below:
[0007] An intelligent multi-view stereo matching method, the method comprising:
[0008] Construct a global context Transformer module for exploring the global context relationship between the reference view features and the source view features at the smallest scale within the view;
[0009] Construct a three-dimensional perception Transformer module based on epipolar geometry to mine the globally perceived reference view feature C r and the globally perceived source view feature for three-dimensional consistency information therebetween;
[0010] Use differentiable deformation operations to map the large-scale features and three-dimensionally consistent features to multiple forward planes of the reference view to obtain the mapped multi-scale source view features, and perform group-level correlation operations on the multi-scale source view features to obtain a multi-scale matching cost volume;
[0011] Construct a multi-scale cost regularization module to realize the mapping between the matching cost volume and the depth map of the reference view; use the L1 loss function to constrain the difference between the multi-scale depth map and the true depth map to realize the training of the network, and based on the training results, use one reference view and any number of source views as inputs, and finally output the depth map of the reference view.
[0012] The global context Transformer module is:
[0013] Input the reference view sequence X r into the Transformer-S r layer, capture the long-distance dependencies between the pixels within the reference view based on multi-head self-attention to obtain the self-attended reference view features, and realize the mapping of the features based on the fully connected feed-forward network to obtain the globally perceived reference view features;
[0014] Input the source view sequence into the Transformer-S s layer to obtain the globally perceived source view features.
[0015] The three-dimensional perception Transformer module based on epipolar geometry is:
[0016] In the Transformer-C r layer, first construct an epipolar feature sampler to sample from The epipolar features of each source view are obtained, and then the pixel dependency between the reference view and the source view is modeled based on multi-head mutual attention to obtain the mutually-attended reference view features. The features are then mapped based on a fully connected feedforward network to obtain 3D consistent reference view features.
[0017] In Transformer-C s In the layer, firstly, the epipolar feature sampler is used to sample the r The epipolar features of the reference view are obtained from the multi-head mutual attention model, and then the pixel dependencies between each source view and the reference view are modeled to obtain the mutually-attended source view features. The feature mapping is implemented based on a fully connected feedforward network to obtain 3D consistent source view features.
[0018] The Transformer-S r The layers are:
[0019] Q r =K r =V r =X r
[0020] X′ r =Concat(LN(MSA(Q r ,K r ,V r )),X r )
[0021] C r =LN(FFN(X′ r ))+X r
[0022] Among them, Concat(,) represents the concatenation operation, LN represents layer normalization, MSA(Q r ,K r ,V r ) indicates that Q r For query, K r As key, V r is the multi-head self-attention value, FFN represents the fully connected feedforward network, X r ′ represents the reference view feature of self-attention, C r Represents the globally aware reference view features.
[0023] The Transformer-S s The layers are:
[0024]
[0025]
[0026]
[0027] Among them, represents the source view feature of self-attention, represents the i-th source view feature of global perception, represents the query when the i-th source view is fed into the multi-head self-attention, represents the key when the i-th source view is fed into the multi-head self-attention, represents the value when the i-th source view is fed into the multi-head self-attention, represents the i-th source view sequence.
[0028] The Transformer-C r layer is:
[0029]
[0030]
[0031]
[0032]
[0033] Among them, EFS represents the epipolar feature sampler, which consists of a differentiable mapping layer and a sampling layer, represents the multi-head cross-attention with Q r as the query, as the key, and as the value, is the epipolar feature of the i-th source view, is the reference view feature of cross-attention, T r is the three-dimensional consistent reference view feature.
[0034] The Transformer-C s layer is:
[0035]
[0036]
[0037]
[0038]
[0039] Among them, represents the multi-head cross-attention with as the query, as the key, and as the value, is the epipolar feature of the reference view, is the mutually-attended i-th source view feature, is the 3D consistent i-th source view feature.
[0040] An intelligent multi-view stereo matching device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the method steps described.
[0041] The beneficial effects of the technical solution provided by the present invention are:
[0042] 1. This paper leverages the Transformer's ability to capture long-range dependencies, fully exploiting the global contextual information within a view and the 3D consistency information between views, thereby achieving multi-view stereo matching.
[0043] 2. This paper designs a global context Transformer module that captures long-range dependencies between pixels within a view, facilitating the extraction of globally context-aware multi-view features. Furthermore, a 3D-aware Transformer module based on epipolar geometry is designed to facilitate information exchange between views and help obtain consistent 3D multi-view features.
[0044] 3. The present invention has been experimentally verified on multiple data sets and can achieve performance that is superior to existing multi-view stereo matching methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 Flowchart of an intelligent multi-view stereo matching method. DETAILED DESCRIPTION
[0046] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.
[0047] 1. Constructing a local feature extraction module
[0048] For a reference view and N source views, the embodiment of the present invention constructs a local feature extraction module to obtain multi-scale reference view features. and multi-scale source view features Where l represents the scale, and its value range is 1 to 3.
[0049] In particular, the local feature extraction modules of different views have the same structure and shared weights, and are composed of 8 convolutional layers.
[0050] 2. Building a Global Context Transformer Module
[0051] For the smallest scale reference view feature and the source view features of the minimum scale In the embodiments of the present invention, a global context Transformer module is constructed to explore the global context relationship within the view. Specifically, and First, an addition operation is performed with the learnable position encoding P to obtain position information, where r represents the reference view. Then, through a flattening operation, the reference view features with position information and the source view features are transformed into a reference view sequence X r and the source view sequence The above process is expressed by the following formula:
[0052]
[0053]
[0054] where j represents the number of pixels in each view feature map, and P j represents the position encoding of the j-th pixel, represents the feature value of the j-th pixel in represents the feature value of the j-th pixel.
[0055] Then, a Transformer-S r layer is designed to obtain the global information of the reference view. Specifically, the reference view sequence X r is input into the Transformer-S r layer. Based on the multi-head self-attention (well-known to those skilled in the art), the long-range dependence relationship between the pixel points within the reference view is captured to obtain the self-attention reference view features, and the mapping of the features is realized based on the fully connected feed-forward network to obtain the globally perceived reference view features.
[0056] where the Transformer-S r layer is expressed by the following formula:
[0057] Q r = K r = V r = X r (३)
[0058] X′ r = Concat(LN(MSA(Q r , K r , V r ))), X r ) (४)
[0059] C r = LN(FFN(X′ r))+X r (5)
[0060] Among them, Concat(,) represents the concatenation operation, and LN represents layer normalization. MSA(Q r ,K r ,V r ) represents taking Q r as the query, taking K r as the key, and taking V r as the value of the multi-head self-attention (Q (query), K (key), V (value). The self-attention mechanism is essentially a weighted summation process. First, the similarity between Q and K is calculated as the weight, and then all Vs are weighted and summed to obtain the output result). FFN represents the fully connected feed-forward network (well-known to those skilled in the art). X r ′ represents the reference view feature of self-attention, and C r represents the reference view feature of global perception.
[0061] Meanwhile, design a Transformer-S s layer to obtain the global information of the source view. Specifically, input the source view sequence into the Transformer-S s layer to obtain the globally perceived source view feature. The Transformer-S s layer is expressed by the formula as follows:
[0062]
[0063]
[0064]
[0065] Among them, represents the self-attention source view feature, represents the i-th globally perceived source view feature, represents the query when the i-th source view is fed into the multi-head self-attention, represents the key when the i-th source view is fed into the multi-head self-attention, represents the value when the i-th source view is fed into the multi-head self-attention, represents the i-th source view sequence, and MSA represents the multi-head self-attention.
[0066] III. Construct a 3D perception Transformer module based on epipolar geometry
[0067] For the globally perceived reference view feature C r and the globally perceived source view feature In the embodiment of the present invention, a 3D perception Transformer module based on epipolar geometry is constructed to mine the 3D consistency information between views.
[0068] In the 3D perception Transformer module based on epipolar geometry, a Transformer-C r layer is designed to enable the reference view to obtain useful information along the epipolar lines of all source views, so as to obtain 3D consistent reference view features. Specifically, in the Transformer-C r layer, first, an epipolar feature sampler is constructed to obtain the epipolar features of each source view from , and then the pixel dependence relationship from the reference view to the source views is modeled based on multi-head mutual attention to obtain the mutually attended reference view features, and the mapping of features is realized based on a fully connected feed-forward network to obtain 3D consistent reference view features. The Transformer-C r layer is expressed by the following formula:
[0069]
[0070]
[0071]
[0072]
[0073] Among them, EFS represents the epipolar feature sampler, which consists of a differentiable mapping layer and a sampling layer. denotes the multi-head mutual attention with Q r as the query, as the key, and as the value, is the epipolar feature of the i-th source view, is the mutually attended reference view feature, and T r is the 3D consistent reference view feature.
[0074] Among them, the epipolar lines of the above-mentioned source views are concepts in multi-view geometry. For a certain pixel point in a view, the corresponding (matched) pixel points of this pixel point in other views must be on a straight line, and this straight line is called the epipolar line. The differentiable mapping layer and the sampling layer are well-known to those skilled in the art.
[0075] Meanwhile, in the 3D perception Transformer module based on epipolar geometry, a Transformer-C s layer is designed to enable the source views to obtain useful information along the epipolar lines of the reference view, so as to obtain 3D consistent source view features. Specifically, in the Transformer-Cs In the layer, first, using the epipolar feature sampler, the epipolar features of the reference view are obtained from T r Then, based on the multi-head mutual attention, the pixel dependencies of each source view to the reference view are modeled to obtain the mutually attentive source view features, and based on the fully connected feed-forward network, the mapping of the features is implemented to obtain the three-dimensionally consistent source view features. The Transformer-C s layer is expressed by the following formula:
[0076]
[0077]
[0078]
[0079]
[0080] Wherein, represents the multi-head mutual attention with as the query, as the key, and as the value, is the epipolar feature of the reference view, is the i-th mutually attentive source view feature, is the i-th three-dimensionally consistent source view feature.
[0081] IV. Constructing the matching cost volume
[0082] Based on the large-scale features obtained by the local feature extraction module and and the three-dimensionally consistent features T r and (i.e., formulas (12) and (16)), the embodiment of the present invention uses the differentiable deformation operation to map to multiple forward planes of the reference view to obtain the mapped multi-scale source view features Then, for the multi-scale source view features and a group-level correlation operation is performed to obtain the multi-scale matching cost volume {V l =V 1 ,V 2 ,V 3}.
[0083] Taking as an example, the embodiment of the present invention first uses the differentiable deformation operation to map it to multiple forward planes of the reference view to obtain the mapped source view features at the 3rd scale Then, for Perform a group-level correlation operation with T r to obtain the matching cost volume V at the third scale 3 . Similar to the above process, based on and the matching cost volume V at the first scale can be obtained 1 ; based on and the matching cost volume V at the first scale can be obtained 2 .
[0084] V. Constructing a cost regularization module
[0085] To obtain the multi-scale reference view depth maps The embodiments of the present invention construct a multi-scale cost regularization module to implement the mapping from V l to . Specifically, the cost regularization modules at different scales are all composed of a 3D U-Net and a regression operation. Among them, the 3D U-Net is used to optimize the initial matching cost volume, and the regression operation maps the optimized matching cost volume to the depth map of the reference view.
[0086] Taking the cost regularization module at the third scale as an example, the initial matching cost volume V at the third scale 3 is first input into the 3D U-Net to implement the optimization process, and then the optimized matching cost volume is subjected to a regression operation to obtain the reference view depth map at the third scale Similarly, V 1 can be obtained through the 3D U-Net and the regression operation V 2 can be obtained through the 3D U-Net and the regression operation
[0087] VI. Training a multi-view stereo matching network based on Transformer
[0088] To obtain high-quality depth maps, the embodiments of the present invention train the network in a multi-scale manner. Specifically, the embodiments of the present invention use the L1 loss function to constrain the difference between the multi-scale depth maps and the ground truth depth maps to implement the training of the network, and the formula is expressed as follows:
[0089]
[0090] where represents the ground truth of the depth maps at different scales, and α l represents the weight coefficients at different scales.
[0091] After training the network, an end-to-end multi-view stereo matching model is obtained. This model takes a reference view and any number of source views as inputs and finally outputs the depth map of the reference view.
[0092] An intelligent multi-view stereo matching device, which includes: a processor and a memory. Program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the following method steps:
[0093] Construct a global context Transformer module for exploring the global context relationship between the reference view features and the source view features at the smallest scale within the view;
[0094] Construct a 3D perception Transformer module based on epipolar geometry to mine the globally perceived reference view feature C r and the globally perceived source view features for 3D consistency information;
[0095] Use differentiable deformation operations to map large-scale features and 3D-consistent features to multiple forward planes of the reference view to obtain the mapped multi-scale source view features, and perform group-level correlation operations on the multi-scale source view features to obtain a multi-scale matching cost volume;
[0096] Construct a multi-scale cost regularization module to realize the mapping between the matching cost volume and the depth map of the reference view; use the L1 loss function to constrain the difference between the multi-scale depth map and the ground truth depth map to realize the training of the network. Based on the training results, take a reference view and any number of source views as inputs and finally output the depth map of the reference view.
[0097] Among them, the global context Transformer module is:
[0098] Input the reference view sequence X r into the Transformer-S r layer, capture the long-range dependencies between pixel points within the reference view based on multi-head self-attention to obtain the self-attended reference view features, and realize the mapping of features based on a fully connected feed-forward network to obtain the globally perceived reference view features;
[0099] Input the source view sequence into the Transformer-S s layer to obtain the globally perceived source view features.
[0100] Among them, the 3D perception Transformer module based on epipolar geometry is:
[0101] In the Transformer-C rIn the layer, first, an epipolar feature sampler is constructed to obtain the epipolar features of each source view from , then the pixel dependencies from the reference view to the source views are modeled based on multi-head mutual attention to obtain the mutually attentive reference view features, and the mapping of the features is implemented based on a fully connected feed-forward network to obtain the three-dimensionally consistent reference view features;
[0102] In the Transformer-C s layer, first, the epipolar feature sampler is used to obtain the epipolar features of the reference view from T r , then the pixel dependencies from each source view to the reference view are modeled based on multi-head mutual attention to obtain the mutually attentive source view features, and the mapping of the features is implemented based on a fully connected feed-forward network to obtain the three-dimensionally consistent source view features.
[0103] In the embodiments of the present invention, unless otherwise specified for the models of each device, the models of other devices are not limited, and any device that can perform the above functions is acceptable.
[0104] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0105] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.
Claims
1. An intelligent multi-view stereo matching method, characterized in that, The method includes: Constructing a global context Transformer module for exploring the global context relationship between the reference view features and the source view features at the smallest scale within the view; Construct a 3D perception Transformer module based on epipolar geometry to mine the reference view features C with global perception r and the source view features with global perception for the 3D consistency information therebetween; Using differentiable deformation operations to map large-scale features and three-dimensionally consistent feature maps to multiple forward planes of the reference view to obtain the mapped multi-scale source view features, and performing group-level correlation operations on the multi-scale source view features to obtain a multi-scale matching cost volume; Constructing a multi-scale cost regularization module to implement the mapping from the matching cost volume to the depth map of the reference view; using the L1 loss function to constrain the difference between the multi-scale depth map and the true depth map to implement the training of the network, and based on the training results, taking one reference view and any number of source views as inputs, and finally outputting the depth map of the reference view.
2. The intelligent multi-view stereo matching method according to claim 1, wherein The global context Transformer module is: The reference view sequence X will be used r as the input to Transformer-S r which has multiple layers. The long-range dependencies between pixel points within the reference view are captured based on multi-head self-attention to obtain self-attention reference view features, and the mapping of features is realized based on a fully connected feed-forward network to obtain globally perceived reference view features; Input the source view sequence into Transformer-S s layers to obtain source view features with global perception.
3. An intelligent multi-view stereo matching method according to claim 1, characterized in that The three-dimensional perception Transformer module based on epipolar geometry is: In Transformer-C r layer, first construct an epipolar feature sampler to obtain the epipolar features of each source view from , then model the pixel dependencies from the reference view to the source views based on multi-head cross-attention to obtain the cross-attention reference view features, and implement the mapping of features based on a fully-connected feed-forward network to obtain three-dimensionally consistent reference view features; In Transformer-C s layer, first, the epipolar feature sampler is used to obtain the epipolar features of the reference view from T r , then the pixel dependencies of each source view to the reference view are modeled based on the multi-head mutual attention to obtain the mutually attentive source view features, and the mapping of features is realized based on the fully connected feed-forward network to obtain the three-dimensionally consistent source view features.
4. An intelligent multi-view stereo matching method according to claim 2, characterized in that, The Transformer-S r layer is as follows: Q r = K r = V r = X r X r ′ = Concat(LN(MSA(Q r , K r , V r ))), X r ) C r = LN(FFN(X r ')) + X r Among them, Concat(,) represents the concatenation operation, LN represents layer normalization, MSA(Q r ,K r ,V r ) represents the multi-head self-attention with Q r as the query, K r as the key, and V r as the value, FFN represents the fully-connected feed-forward network, X r ' represents the reference view feature of self-attention, and C r represents the reference view feature of global perception.
5. An intelligent multi-view stereo matching method according to claim 4, characterized in that, The Transformer-S s layer is as follows: Among them, represents the source view feature of self-attention, represents the i-th source view feature of global perception, represents the query when the i-th source view is fed into the multi-head self-attention, represents the key when the i-th source view is fed into the multi-head self-attention, represents the value when the i-th source view is fed into the multi-head self-attention, represents the i-th source view sequence.
6. The intelligent multi-view stereo matching method according to claim 3, wherein The Transformer-C r layer is as follows: Among them, EFS represents the epipolar feature sampler, which consists of a differentiable mapping layer and a sampling layer. denotes the multi-head mutual attention with Q r as the query, as the key, and as the value. is the epipolar feature of the i-th source view, is the reference view feature of the mutual attention, and T r is the three-dimensional consistent reference view feature.
7. An intelligent multi-view stereo matching method according to claim 3, characterized in that The Transformer-C s layer is as follows: T r e = EFS(T r ) Among them, denotes the multi-head mutual attention with as the query, as the key, and V r e as the value, r e is the epipolar feature of the reference view, is the i-th source view feature of the mutual attention, is the i-th source view feature of the three-dimensional consistency.
8. An intelligent multi-view stereo matching device, characterized in that, The device includes: a processor and a memory, and program instructions are stored in the memory. The processor calls the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-7.