Depth estimation method for multi-view vision system, electronic device and medium

Abstract

The application discloses a kind of depth estimation methods for multi-view visual system, electronic equipment and medium, it is related to multi-view visual system depth estimation technical field, the method mainly includes the target view image and the auxiliary view image are input to the depth estimation model for multi-view image repair, obtain the parallax map for estimating scene depth value;Wherein, the depth estimation model for multi-view image repair includes feature extraction module based on deformable convolution and hollow pyramid convolution, matching cost calculation module based on multi-scale structure, multi-scale cost aggregation module based on self-attention mechanism and parallax regression operation module.The application can accurately estimate scene depth value, provide reliable technology for unmanned system to realize high-precision scene depth estimation.

Description

Technical Field

[0001] This invention relates to the field of depth estimation technology for multi-view vision systems, and in particular to a depth estimation method, electronic device, and computer-readable storage medium for multi-view vision systems. Background Technology

[0002] In recent years, with the rapid development of technologies such as computers and wireless communications, and the successful application of artificial intelligence methods in various fields, unmanned systems, represented by autonomous vehicles, drones, and unmanned surface vessels, have increasingly become a key research focus, keeping pace with the times. Unmanned systems require various sensors, such as lidar, millimeter-wave radar, and visual sensors, to perceive unknown scenes. Considering the inherent advantages and disadvantages of various sensors and ensuring the stability and safety of unmanned systems, they typically incorporate multiple different types of sensors to achieve comprehensive scene perception. Furthermore, to obtain rich scene structural information and enhance the ability to reconstruct scene information, unmanned systems usually require the inclusion of visual sensors.

[0003] Because the lenses of visual sensors are in constant contact with the external environment, their surfaces are easily obscured by dust, water stains, and other contaminants, leading to a loss of scene information. These uncontrollable external factors significantly degrade the camera's image quality, causing deviations in the unmanned system's perception of complex environments, affecting the correctness of its decisions, and even causing accidents. To ensure the accuracy of visual system data acquisition and high scene reproduction, multi-view image inpainting methods are used to repair contaminated images. However, the repaired images cannot provide distance information between the system and objects in the environment, easily leading to collisions between the unmanned system and its surroundings. To avoid collisions, the unmanned system needs to measure its distance from objects in the scene in real time to maintain a certain distance. Therefore, a method for measuring the distance between the unmanned system and objects in the scene based on multi-view image inpainting is needed. Summary of the Invention

[0004] To address the problem of distance measurement between unmanned systems with multiple viewpoints and objects in a scene, this invention provides a depth estimation method, electronic device, and medium for multi-view vision systems.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] In a first aspect, the present invention provides a depth estimation method for multi-view vision systems, comprising:

[0007] Acquire the repaired target view image and auxiliary view image;

[0008] The target view image and the auxiliary view image are input into a depth estimation model for multi-view image inpainting to obtain a disparity map; the disparity map is used to estimate scene depth values.

[0009] The depth estimation model for multi-view image inpainting includes a feature extraction module based on deformable convolution and dilated pyramid convolution, a matching cost calculation module based on multi-scale structure, a multi-scale cost aggregation module based on self-attention mechanism, and a disparity regression operation module.

[0010] Optionally, the feature extraction module based on deformable convolution and dilated pyramid convolution is a U-shaped network structure; the main body of the U-shaped network structure is divided into an encoder-decoder structure.

[0011] The encoder-decoder structure includes an encoder, a connection layer, and a decoder connected in sequence.

[0012] The encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and each encoding layer and the decoding layer is embedded with a residual module combined with deformable convolution; the connection layer is embedded with a spatial pyramid convolution with increasing dilation rate.

[0013] Optionally, the encoder is used to process the target view image and the auxiliary view image to obtain the encoder output result;

[0014] The connection layer is used to process the encoder output results in parallel using spatial pyramid convolution with increasing dilation rate, and then splices and fuses the output results of different branches obtained after parallel processing along the channel direction to obtain a fused result.

[0015] The decoder is used to process the fusion result to obtain target view feature maps and auxiliary view feature maps at different resolutions.

[0016] Optionally, the matching cost calculation module based on multi-scale structure is used to perform initial matching cost calculation on multiple pairs of feature maps output by the feature extraction module based on deformable convolution and dilated pyramid convolution respectively to obtain multiple initial matching cost bodies;

[0017] The number of initial matching cost bodies is the same as the number of pairs of feature maps, and each pair of feature maps includes a target view feature map and an auxiliary view feature map corresponding to the target view feature map; wherein, different pairs of feature maps correspond to different resolutions, and different initial matching cost bodies correspond to different scales.

[0018] Optionally, the calculation process of the initial matching cost body is as follows:

[0019] When the disparity k of the search is 0, the target view feature map and the corresponding auxiliary view feature map are multiplied element-wise channel by channel to construct a single disparity cost body with a disparity of 0.

[0020] When the disparity k of the search is greater than 0 and less than the maximum disparity value, the similarity of the pixels in the width direction of the target view feature map with the pixels in the width direction of the auxiliary view feature map corresponding to the target view feature map is calculated, and a single disparity cost body with disparity k is constructed based on the similarity.

[0021] The initial matching cost body is obtained by combining single disparity cost bodies with different disparities.

[0022] Optionally, the multi-scale cost aggregation module based on the self-attention mechanism is used to perform cost aggregation operations on the initial matching cost volumes of different scales output by the matching cost calculation module based on the multi-scale structure to obtain multiple aggregated cost volume feature maps.

[0023] The multi-scale cost aggregation module based on the self-attention mechanism includes at least three branches, wherein the first and second branches each include an encoder, a self-attention mechanism module and a decoder connected in sequence, and the third branch includes an encoder and a decoder connected in sequence.

[0024] The encoder is composed of stacked residual 3D convolutional modules with a stride of 2 connected in series; the decoder is composed of stacked 3D transposed convolutional modules with an upsampling rate of 2 connected in series.

[0025] Optionally, the self-attention mechanism module is used for:

[0026] The initial matching cost body processed by the encoder is preprocessed using three 1×1 3D convolution kernels with non-shared parameters to obtain three cost bodies Q, K, and V.

[0027] The cost body Q, the cost body K, and the cost body V are respectively transformed by dimension to obtain cost body Q′, cost body K′, and cost body V′;

[0028] Perform matrix multiplication on the cost body Q′ and the cost body K′ to obtain the correlation description matrix S of all sampling points of the cost body;

[0029] The horizontal values ​​of the correlation description matrix S are mapped to a weight sequence using the softmax function to obtain the cost attention map S′.

[0030] Perform a matrix multiplication operation between the cost body V′ and the cost attention map S′ to obtain the weighted cost body;

[0031] The dimensions of the weighted cost volume are restored by dimensional transformation to obtain the output result O, and the output result O is input into the decoder.

[0032] Optionally, the disparity regression operation module is used for:

[0033] The feature map of the aggregated cost volume with a downsampling rate of k, output by the multi-scale cost aggregation module based on the self-attention mechanism, is amplified by a factor of k using trilinear interpolation.

[0034] The softmax function is used to normalize the amplified aggregate cost volume feature map to obtain the weighted cost volume.

[0035] The weighted cost volume is searched along the disparity dimension of the weighted cost volume and the discrete disparity search range [0, D]. max The final disparity map is obtained by performing a weighted summation within the range / k); D max This represents the maximum value of the parallax search range.

[0036] In a second aspect, the present invention provides an electronic device including a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the depth estimation method for multi-view vision systems according to the first aspect.

[0037] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the depth estimation method for multi-view vision systems as described in the first aspect.

[0038] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] This invention provides a depth estimation method, electronic device, and medium for multi-view vision systems. The invention inputs a target view image and an auxiliary view image into a depth estimation model for multi-view image inpainting to obtain a disparity map. The disparity map is used to estimate scene depth values. The depth estimation model for multi-view image inpainting includes a feature extraction module based on deformable convolution and dilated pyramid convolution, a matching cost calculation module based on a multi-scale structure, a multi-scale cost aggregation module based on a self-attention mechanism, and a disparity regression operation module. This invention, through a multi-scale network structure, enables the network to capture objects of different scales and generate clearer, sharper disparity edges, providing a reliable technology for high-precision scene depth estimation in unmanned systems. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the depth estimation method for multi-view vision systems provided in an embodiment of the present invention.

[0042] Figure 2 A schematic diagram of the main structure of the depth estimation model for multi-view image inpainting provided in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a feature extraction module based on deformable convolution and dilated pyramid convolution provided in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the residual module combined with deformable convolution provided in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the multi-scale matching cost calculation process provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the initial matching cost calculation process provided in an embodiment of the present invention;

[0047] Figure 7 A schematic diagram of the structure of the multi-scale cost aggregation module and the disparity regression operation module based on the self-attention mechanism provided in an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the self-attention mechanism calculation process in the multi-scale cost aggregation module provided in an embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of the softmax disparity regression calculation process provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] like Figure 1 As shown in the figure, the depth estimation method for multi-view vision systems provided by this embodiment of the invention includes the following steps.

[0054] Step 100: Obtain the repaired target view image and auxiliary view image. The target view image and the auxiliary view image correspond one-to-one.

[0055] Step 200: Input the target view image and the auxiliary view image into a depth estimation model for multi-view image inpainting to determine a disparity map. The disparity map is used to estimate scene depth values.

[0056] like Figure 2 As shown in the embodiment of the present invention, the depth estimation model for multi-view image inpainting includes a feature extraction module based on deformable convolution and dilated pyramid convolution, a matching cost calculation module based on multi-scale structure, a multi-scale cost aggregation module based on self-attention mechanism, and a disparity regression operation module.

[0057] As a preferred embodiment, the feature extraction module based on deformable convolution and dilated pyramid convolution provided in this invention can effectively capture multi-scale contextual information, and the feature extraction module has a U-shaped network structure; this U-shaped network structure can enhance the network's ability to capture both detailed information and global semantic information by fusing shallow features with high-level features.

[0058] The main body of the U-shaped network structure is divided into an encoder-decoder structure, which includes an encoder, a connection layer, and a decoder connected in sequence. The encoder includes multiple encoding layers, and the decoder includes multiple decoding layers. Each encoding layer and decoding layer embeds a residual module combined with deformable convolution to enhance feature capture capability. The connection layer embeds a spatial pyramid convolution with an increasing dilation rate, that is, the encoder output is processed by a spatial pyramid convolution with an increasing dilation rate to enhance the network's ability to represent contextual information at different scales and preserve the edge information of features to the greatest extent. At different stages of the decoder, feature maps with different resolutions are output to prepare for constructing the cost volume in subsequent operations.

[0059] Furthermore, the encoder is used to process the target view image and the auxiliary view image to obtain the encoder output result; the connection layer is used to process the encoder output result in parallel using spatial pyramid convolution with increasing dilation rate, and to stitch and fuse the output results of different branches obtained after parallel processing along the channel direction to obtain the fused result; the decoder is used to process the fused result to obtain target view feature maps and auxiliary view feature maps of different resolutions.

[0060] One example is: using the repaired target view image and the auxiliary view image as... Figure 3 The input to the feature extraction module is shown. The feature extraction module uses an encoder-decoder structure as its backbone. Each encoding and decoding layer is composed of residual modules combined with deformable convolutions, as shown in the diagram. Figure 4 As shown. In the feature extraction module, the input image is first processed by the encoder. Then, the encoder's output is processed in parallel using spatial pyramid convolutions with dilation rates of {2, 4, 8, 16}. The outputs of different branches are then concatenated along the channel direction and fused using a 1×1 convolution kernel as the input to the decoder. Finally, feature maps with resolutions of {1 / 16, 1 / 8, 1 / 4} are output at different stages of the decoder, resulting in target and auxiliary viewpoint feature maps with resolutions of {1 / 16, 1 / 8, 1 / 4}. Figure 4 In this context, DCN represents deformable convolution.

[0061] As a preferred embodiment, the multi-scale structure-based matching cost calculation module provided in this invention is used to perform initial matching cost calculations on multiple pairs of feature maps output by the feature extraction module based on deformable convolution and dilated pyramid convolution, respectively, to obtain multiple initial matching cost bodies (i.e., four-dimensional cost bodies), thereby enhancing the network's ability to perceive objects at different scales; the four dimensions refer to the height H of the input feature map, the width W of the input feature map, the number of channels C of the feature map, and the maximum disparity search range D. max .

[0062] The number of initial matching cost bodies is the same as the number of pairs of feature maps, and each pair of feature maps includes a target view feature map and an auxiliary view feature map corresponding to the target view feature map; wherein, different pairs of feature maps correspond to different resolutions, and different initial matching cost bodies correspond to different scales.

[0063] Furthermore, the calculation process of the initial matching cost body described in this embodiment of the invention is as follows:

[0064] The similarity of all matching point pairs along the epipolar direction is calculated by element-wise multiplication to construct the initial matching cost body; in [0, D]max Within the disparity range, based on the different disparities, a single disparity cost body is constructed by multiplying corresponding elements. This calculation process is highly parallelized. The single disparity cost bodies of different disparities are used together to construct the initial matching cost body. The detailed process is as follows.

[0065] Step (1): When the disparity k of the search is 0, the target view feature map and the corresponding auxiliary view feature map are multiplied element by element through each channel to construct a single disparity cost body with a disparity of 0.

[0066] Step (2): When the searched disparity k is greater than 0 and less than the maximum disparity value, the similarity of the pixels in the width direction of the target view feature map with the pixels in the width direction of the auxiliary view feature map corresponding to the target view feature map is calculated, and a single disparity cost body with disparity k is constructed based on the similarity.

[0067] Step (3): Combine the single disparity cost volumes with different disparities to obtain the initial matching cost volume.

[0068] An example is: Performing multi-scale matching cost calculation on three pairs of feature maps (target view feature map and corresponding auxiliary view feature map) with downsampling rates {1 / 16, 1 / 8, 1 / 4} output by the feature extraction module. The calculation process is as follows: Figure 5 As shown. The multi-scale matching cost calculation consists of three initial matching cost calculations, the process of which is as follows: Figure 6 As shown. When the search disparity is 0, the target view feature map and the corresponding auxiliary view feature map are directly multiplied element-wise channel by channel to construct a single disparity cost body with 0 disparity; when the search disparity k is greater than 0 and less than the maximum disparity search range D, ... max At this time, it is necessary to calculate the similarity between the pixels of the target view feature map in the width direction [n:W) and the pixels of the auxiliary view feature map in the width direction [0:Wn), and construct a single disparity cost body with disparity k. Then, the disparity in [0, D] is calculated. max The single disparity cost volumes within the range are combined to form the initial matching cost volume. After multi-scale matching cost calculation, three pairs of feature maps with downsampling rates of {1 / 16, 1 / 8, 1 / 4} are used to obtain three initial matching cost volumes Cost1, Cost2, and Cost3 at different scales.

[0069] The multi-scale matching cost calculation structure designed in this invention takes into account that the perception capabilities of matching cost volumes of different scales are different for objects of different scales. It takes into account the characteristics of large-scale cost volumes and small-scale cost volumes, improves the global continuity of disparity estimation and retains a large amount of detailed information, which is conducive to improving the overall accuracy of disparity estimation.

[0070] As a preferred embodiment, the multi-scale cost aggregation module based on the self-attention mechanism provided in this invention is used to perform cost aggregation operations on multiple initial matching cost volumes of different scales using a 3D convolution-based encoder-decoder structure and an improved 3D convolution-based encoder-decoder structure, to obtain multiple aggregated cost volume feature maps.

[0071] The multi-scale cost aggregation module based on the self-attention mechanism includes at least three branches. The first and second branches are both improved 3D convolution-based encoder-decoder structures, each comprising an encoder, a self-attention mechanism module, and a decoder connected in sequence. The third branch is also a 3D convolution-based encoder-decoder structure, comprising an encoder and a decoder connected in sequence. This 3D convolution-based encoder-decoder structure increases network capacity while stabilizing the network training process.

[0072] The encoder is composed of stacked residual 3D convolutional modules with a stride of 2 connected in series, and the decoder is composed of stacked 3D transposed convolutional modules with an upsampling rate of 2 connected in series. The encoder and decoder are connected by residual connections to ensure the normal training process of deep networks.

[0073] The self-attention mechanism module is used to explicitly model the correlation between each initial matching cost body in order to achieve dynamic aggregation of cost bodies.

[0074] Furthermore, this self-attention mechanism module is used for:

[0075] The initial matching cost volume processed by the encoder is used as input, and it is preprocessed using three 1×1 3D convolution kernels with non-shared parameters to obtain three cost volumes Q, K, and V.

[0076] The cost bodies Q, K, and V are transformed by dimensionality to obtain cost bodies Q′, K′, and V′, respectively.

[0077] Perform matrix multiplication on cost bodies Q′ and K′ to obtain the correlation description matrix S of all sampling points of the cost body.

[0078] By using the softmax function to map the values ​​in the horizontal direction of the correlation description matrix S to a weight sequence, the cost attention map S′ can be obtained.

[0079] Perform a matrix multiplication operation between the cost body V′ and the cost attention map S′ to obtain the weighted cost body.

[0080] The dimensions of the weighted cost volume are restored by dimensional transformation to obtain the output result O, and the output result O is input into the decoder.

[0081] One example is that, in order to reduce the computational time and space complexity of the self-attention mechanism module, the self-attention mechanism module is only embedded at the junction of the 1 / 16 scale encoder-decoder structure and the 1 / 8 scale encoder-decoder structure.

[0082] The initial matching cost body is used as... Figure 7 The input of the multi-scale cost aggregation module is shown. This multi-scale cost aggregation module includes at least three branches. The first and second branches each include an encoder, a self-attention mechanism module, and a decoder connected in sequence. The third branch includes an encoder and a decoder connected in sequence. The encoder is composed of stacked residual 3D convolutional modules with a stride of 2, and the decoder is composed of stacked 3D convolutional modules with an upsampling rate of 2. For the initial matching cost body Cost3 with a downsampling rate of 1 / 4, it is directly input to the decoder after processing by the encoder to obtain the corresponding output result. For the initial matching cost bodies Cost1 and Cost2 with downsampling rates of 1 / 16 and 1 / 8, respectively, the encoder output is processed by the self-attention mechanism module before being input to the decoder. The calculation process of the self-attention mechanism module is as follows: Figure 8 As shown.

[0083] like Figure 8 As shown, the initial matching cost body I, after being processed by the encoder, is preprocessed using three 1×1 3D convolutional kernels with non-shared parameters to obtain three cost bodies Q, K, and V. Cost bodies Q, K, and V are then subjected to dimensionality transformations to obtain cost bodies Q′, K′, and V′, respectively. Matrix multiplication is performed on cost bodies Q′ and K′ to obtain the correlation description matrix S of all sampled points of the cost body. The horizontal values ​​of the correlation description matrix S are mapped to a weight sequence using the softmax function to obtain the cost attention map S′. Matrix multiplication is then performed between cost body V′ and cost attention map S′ to obtain the weighted cost body. Dimensional transformation is then used to restore the dimensions of the weighted cost body, resulting in the output O. The output O of the self-attention mechanism module is used as the input to the decoder to obtain the feature maps of the aggregated cost bodies corresponding to the initial matching cost bodies Cost1, Cost2, and Cost3.

[0084] As a preferred embodiment, the disparity regression operation module of this invention is used to: enlarge the aggregated cost volume feature map with a downsampling rate of k output by the multi-scale cost aggregation module by a factor of k in all three dimensions using trilinear interpolation; then normalize the enlarged aggregated cost volume feature map using the softmax function to obtain a weighted cost volume; and then apply the weighted cost volume along its disparity dimension and the discrete disparity search range [0, D]. max We perform a weighted summation within / k) to obtain the final disparity map.

[0085] One example is:

[0086] To improve the ability of cost volumes at different scales to capture semantic information at different levels, this embodiment of the invention jointly trains three aggregated cost volume feature maps {1 / 16, 1 / 8, 1 / 4}, which are respectively aggregated by encoder and decoder. Figure 7 As shown in the right half, cost aggregation outputs different results during the training and inference phases.

[0087] During the training phase, disparity regression is used to independently process the aggregate cost volume feature maps of the three different branches, outputting three disparity maps Disp1, Disp2, and Disp3 at different resolutions. Then, smoothed L1 loss is used to calculate the loss value between the disparity estimation maps at the three scales and their true disparity. The three loss values ​​are then added together according to certain weights to obtain the total loss value.

[0088] disparity regression calculation as follows Figure 9 As shown, the aggregated cost volume feature map with a downsampling rate of k output by the multi-scale cost aggregation module is first processed using trilinear interpolation. The dimensions in all three dimensions were magnified by a factor of k, resulting in the cost body. Then, the softmax function is used to optimize the amplified cost volume. Perform a normalization operation to obtain the weighted cost volume. Finally, the weighted cost body Along its disparity dimension and discrete disparity search range [0, D] max We perform a weighted summation within / k) to obtain the final disparity map Disp.

[0089] This describes the softmax function on the amplified cost volume. The calculation process of performing normalization operations; among which, The cost volume pixel p with a downsampling rate of k, output by the cost aggregation module, represents the i-th cost value in the disparity dimension, where k∈{4, 8, 16}; for The weight values ​​are obtained by mapping using the softmax function; ε is a very small number that ensures the stability of the operation and is the upper limit of the relative error caused by floating-point operations in the computer. The cost of downsampling at rate k is the sum of all costs for pixel p in the disparity dimension.

[0090] The calculation process of disparity regression is described; where k∈{4,8,16}, and j is the disparity search range [0,D]. max Discrete values ​​of / k), The weight value d is the j-th weight value of pixel p in the disparity dimension of the weight sequence cost volume after processing by the above formula. k (p) represents the disparity regression result at pixel p in the cost volume, i.e., the final disparity estimate.

[0091] Example 2

[0092] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a depth estimation device for multi-view vision systems is provided below.

[0093] This invention provides a depth estimation device for multi-view vision systems, comprising:

[0094] The data acquisition unit is used to acquire the repaired target view image and auxiliary view image.

[0095] The disparity map calculation unit is used to input the target view image and the auxiliary view image into a depth estimation model for multi-view image inpainting to obtain a disparity map. The disparity map is used to estimate scene depth values.

[0096] The depth estimation model for multi-view image inpainting includes a feature extraction module based on deformable convolution and dilated pyramid convolution, a matching cost calculation module based on multi-scale structure, a multi-scale cost aggregation module based on self-attention mechanism, and a disparity regression operation module.

[0097] Example 3

[0098] This invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform a depth estimation method for a multi-view vision system according to Embodiment 1.

[0099] Alternatively, the aforementioned electronic device may be a server.

[0100] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements a depth estimation method for a multi-view vision system according to Embodiment 1.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0102] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A depth estimation method for multi-view vision systems, characterized in that, include: Acquire the repaired target view image and auxiliary view image; The target view image and the auxiliary view image are input into a depth estimation model for multi-view image inpainting to obtain a disparity map; The disparity map is used to estimate scene depth values; The depth estimation model for multi-view image inpainting includes a feature extraction module based on deformable convolution and dilated pyramid convolution, a matching cost calculation module based on multi-scale structure, a multi-scale cost aggregation module based on self-attention mechanism, and a disparity regression operation module. The matching cost calculation module based on multi-scale structure is used to calculate the initial matching cost of multiple pairs of feature maps output by the feature extraction module based on deformable convolution and dilated pyramid convolution, and obtain multiple initial matching cost bodies. The number of initial matching cost bodies is the same as the number of logarithms of the feature maps, and each pair of feature maps includes a target view feature map and an auxiliary view feature map corresponding to the target view feature map; wherein, different pairs of feature maps correspond to different resolutions, and different initial matching cost bodies correspond to different scales; The multi-scale cost aggregation module based on the self-attention mechanism is used to perform cost aggregation operations on the initial matching cost bodies of different scales output by the matching cost calculation module based on the multi-scale structure, and obtain multiple aggregated cost body feature maps. The multi-scale cost aggregation module based on the self-attention mechanism includes at least three branches, wherein the first and second branches each include an encoder, a self-attention mechanism module and a decoder connected in sequence, and the third branch includes an encoder and a decoder connected in sequence. The encoder is composed of stacked residual 3D convolutional modules with a stride of 2 connected in series; the decoder is composed of stacked 3D transposed convolutional modules with an upsampling rate of 2 connected in series.

2. The depth estimation method for multi-view vision systems according to claim 1, characterized in that, The feature extraction module based on deformable convolution and dilated pyramid convolution has a U-shaped network structure; the main body of the U-shaped network structure is divided into an encoder-decoder structure. The encoder-decoder structure includes an encoder, a connection layer, and a decoder connected in sequence. The encoder includes multiple encoding layers, the decoder includes multiple decoding layers, and each encoding layer and the decoding layer is embedded with a residual module combined with deformable convolution; the connection layer is embedded with a spatial pyramid convolution with increasing dilation rate.

3. The depth estimation method for multi-view vision systems according to claim 2, characterized in that, The encoder is used to process the target view image and the auxiliary view image to obtain the encoder output result; The connection layer is used to process the encoder output results in parallel using spatial pyramid convolution with increasing dilation rate, and then splices and fuses the output results of different branches obtained after parallel processing along the channel direction to obtain a fused result. The decoder is used to process the fusion result to obtain target view feature maps and auxiliary view feature maps at different resolutions.

4. The depth estimation method for multi-view vision systems according to claim 1, characterized in that, The calculation process for the initial matching cost body is as follows: When the disparity k of the search is 0, the target view feature map and the corresponding auxiliary view feature map are multiplied element-wise channel by channel to construct a single disparity cost body with a disparity of 0. When the disparity k of the search is greater than 0 and less than the maximum disparity value, the similarity of the pixels in the width direction of the target view feature map with the pixels in the width direction of the auxiliary view feature map corresponding to the target view feature map is calculated, and a single disparity cost body with disparity k is constructed based on the similarity. The initial matching cost body is obtained by combining single disparity cost bodies with different disparities.

5. The depth estimation method for multi-view vision systems according to claim 1, characterized in that, The self-attention mechanism module is used for: The initial matching cost body processed by the encoder is preprocessed using three 1×1 3D convolution kernels with non-shared parameters to obtain three cost bodies Q, K, and V. The cost body Q, the cost body K, and the cost body V are respectively transformed by dimension to obtain cost body Q′, cost body K′, and cost body V′; Perform matrix multiplication on the cost body Q′ and the cost body K′ to obtain the correlation description matrix S of all sampling points of the cost body; The horizontal values ​​of the correlation description matrix S are mapped to a weight sequence using the softmax function to obtain the cost attention map S′. Perform a matrix multiplication operation between the cost body V′ and the cost attention map S′ to obtain the weighted cost body; The dimensions of the weighted cost volume are restored by dimensional transformation to obtain the output result O, and the output result O is input into the decoder.

6. The depth estimation method for a multi-view vision system according to claim 1, characterized in that, The disparity regression operation module is used for: The feature map of the aggregated cost volume with a downsampling rate of k, output by the multi-scale cost aggregation module based on the self-attention mechanism, is amplified by a factor of k using trilinear interpolation. The softmax function is used to normalize the amplified aggregate cost volume feature map to obtain the weighted cost volume. The weighted cost volume is searched along the disparity dimension of the weighted cost volume and the discrete disparity search range [0, D]. max The final disparity map is obtained by performing a weighted summation within the range / k); D max This represents the maximum value of the parallax search range.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform a depth estimation method for a multi-view vision system according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the depth estimation method for multi-view vision systems as described in any one of claims 1 to 6.

Images