A distortion-aware outdoor panoramic depth estimation method

Abstract

The present invention provides a distortion-aware outdoor panoramic depth estimation method, which belongs to the field of outdoor panoramic depth estimation, and comprises: constructing an estimation network, wherein the estimation network comprises a convolution branch, a collaborative network of an attention mechanism branch, n A fusion module and a decoding module; based on the panoramic spherical imaging model, the distortion perception map corresponding to the panoramic image is calculated; the equirectangular projection map and the distortion perception map of the panoramic image are used as the input of the convolution branch, and the slice projection map and the distortion perception map are used as the input of the attention mechanism branch; the decoding module includes sequentially connected n decoding layer, the convolution branch i +1 layer output encoding features and attention mechanism branch i The encoded features output by the layer are input into the i Fusion module, i The output of the fusion module is input i The present invention can effectively improve the estimation accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of outdoor panoramic depth estimation, and in particular relates to a distortion-aware outdoor panoramic depth estimation method. Background Art

[0002] Outdoor panoramic images have a 360° viewing angle with large distortion, more complex scene structures, and a wider range of depth information, so the depth estimation of outdoor panoramic images has unique challenges. There are two existing panoramic depth estimation methods: 1. Use equirectangular projection and cubic projection fusion, or double cubic projection fusion strategy; 2. Design a special distortion-aware convolution filter (such as spherical distortion-aware convolution filter). Both methods have defects: for the former, equirectangular projection cannot fully process the distorted area of ​​the panoramic image; cubic projection is to project the panoramic image onto a cube consisting of six faces: top, bottom, left, right, front, and back. Although this fusion strategy can reduce panoramic distortion to a certain extent, it cannot completely eliminate distortion and the additional cross-projection fusion module increases the computational burden. For the latter, the fixed sampling position limits their performance. The spherical distortion-aware convolution filter performs filtering operations in a spherical space, which leads to higher computational complexity, especially for high-resolution panoramic images. The computational cost is higher. Secondly, the spherical perception filter is sensitive to the distortion in the panoramic image. When there is large distortion in the panoramic image, the filter may produce large errors, which will seriously affect the accuracy of depth estimation. In addition, the convolution filter has a limited field of view and cannot understand the overall information of the panoramic image, which will lead to overall misestimation. It is still difficult to apply the convolution filters of deeper networks to depth estimation tasks. Summary of the invention

[0003] The purpose of the present invention is to propose a distortion-aware outdoor panoramic depth estimation method, which can effectively improve the estimation accuracy.

[0004] The present invention is achieved through the following technical solutions:

[0005] A distortion-aware outdoor panoramic depth estimation method comprises the following steps:

[0006] Step S1, constructing an estimation network, the estimation network includes a convolution branch, a collaborative network of an attention mechanism branch, n fusion modules and a decoding module, the convolution branch is used to extract local features of the equirectangular projection of the panoramic image, and the attention mechanism branch is used to extract global features of the slice projection map of the panoramic image;

[0007] Step S2: based on the panoramic spherical imaging model, the area stretching ratio of the panoramic spherical surface to the projection surface is calculated, and the distortion value of each pixel point on the panoramic image is obtained accordingly, and then the distortion perception map corresponding to the panoramic image is obtained;

[0008] Step S3, using the equirectangular projection map and the distortion perception map of the panoramic image as inputs of the convolution branch, and using the slice projection map and the distortion perception map as inputs of the attention mechanism branch, wherein the slice projection map is obtained by using the equirectangular projection of the panoramic image;

[0009] Step S4: The decoding module includes sequentially connected n decoding layer, the convolution branch i +1 layer output encoding features and attention mechanism branch i The encoded features output by the layer are input into the i Fusion module, i The output of the fusion module is input i A decoding layer is used to decode and generate a depth map, where the fusion module is implemented based on the dual attention mechanism of space and channel. n , i are all integers, i =1,2,…, n .

[0010] Furthermore, in step S1, the backbone network of the convolution branch is composed of a ResNet encoder, and the backbone network of the attention mechanism branch is a visual attention mechanism network.

[0011] Furthermore, in step S2, the distortion perception map W It is composed of the distortion values ​​of all pixels in the panoramic image, where the pixel ( x , y ) is expressed as , .

[0012] Furthermore, in step S3, the equirectangular projection image of size H×W and the distortion perception image W of the same size are used as inputs of the convolution branch. i The encoded features of the layer output are .

[0013] Furthermore, in step S3, the equirectangular projection image is projected onto the panoramic sphere, and for each point on the panoramic sphere, each point is m Points form a slice block, and the slice projection image is composed of slice blocks. The attention mechanism branch applies the attention mechanism in each slice block. i The encoded features of the layer output are .

[0014] Furthermore, in step S4, the convolution branch i Encoded features of layer output , the layer corresponding toi The fusion module encodes the features After performing maximum pooling and mean pooling respectively, they are concatenated to form a feature map, and then a convolutional layer is used to learn the feature map. Finally, the Sigmoid function is used to obtain the channel attention result. .

[0015] Furthermore, in step S4, the attention mechanism branch i Encoded features of layer output , the layer corresponding to i The fusion module encodes the features Perform global pooling, then pass through a fully connected layer, and finally use the Sigmoid function to get the spatial attention result .

[0016] Further, in step S4, i The attention results of the fusion module on the channel And spatial attention results Using the formula Fusion to get output ,in, , W 1. W 2 represents two 3×3 convolutional layers, Residual represents the residual connection function, and Conv represents the convolution function.

[0017] Furthermore, in step S4, the decoding layer includes an upsampling block and a decoding block output by the upsampling module, the decoding module is implemented by an attention mechanism, and the output of the fusion module is used as the input of the upsampling module of the corresponding decoding layer.

[0018] Furthermore, in step S4, the depth map generated by decoding is transformed to obtain a three-dimensional reconstructed map with complete spatial information.

[0019] The present invention has the following beneficial effects:

[0020] The estimation network constructed by the present invention includes a collaborative network of a convolution branch, an attention mechanism branch, a plurality of fusion modules and a decoding module. The equirectangular projection map and the distortion perception map of the panoramic image are used as inputs of the convolution branch, the slice projection map and the distortion perception map are used as inputs of the attention mechanism branch, the coding features output by each layer of the convolution branch and the coding features output by each layer of the attention mechanism branch are input into the corresponding fusion module, and the outputs of each fusion module are respectively input into the corresponding decoding layer to decode and generate a depth map, which can fully extract the panoramic image features and perform panoramic distortion perception on the panoramic image, thereby solving the problem of limited field of view in the spherical convolution filter, and in the fusion module, the coding features of each layer are fused with attention, that is, the coding features of the lower layer are retained, and the coding features of the lower layer provide more detailed spatial information, which helps to better capture details during decoding, thereby effectively improving the estimation accuracy, and the features of the lower layer fusion are jump-connected to the decoding layer, which helps to alleviate the gradient vanishing problem in the deep network and makes the training more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention is further described in detail below with reference to the accompanying drawings.

[0022] Figure 1 It is a flow chart of the present invention.

[0023] Figure 2 It is a schematic diagram of the connection block diagram of the estimation network of the present invention.

[0024] Figure 3 It is a schematic diagram of the connection block diagram of the fusion module and the decoding module of the present invention.

[0025] Figure 4 It is a schematic diagram of the panoramic spherical forming model of the present invention. DETAILED DESCRIPTION

[0026] like Figures 1 to 3 As shown, the distortion-aware outdoor panoramic depth estimation method includes the following steps:

[0027] Step S1, constructing an estimation network, the estimation network includes a convolution branch, a collaborative network of an attention mechanism branch, n fusion modules and a decoding module, the convolution branch is used to extract local features of the equirectangular projection of the panoramic image, and the attention mechanism branch is used to extract global features of the slice projection map of the panoramic image;

[0028] Step S2: based on the panoramic spherical imaging model, the area stretching ratio of the panoramic spherical surface to the projection surface is calculated, and the distortion value of each pixel point on the panoramic image is obtained accordingly, thereby obtaining a distortion perception map corresponding to the panoramic image;

[0029] The panoramic spherical imaging model is an existing technology. Figure 4As shown: The panoramic sphere is an image formed by scanning and photographing with a panoramic camera. The points on the panoramic sphere are projected using the projection formula to obtain the projection plane.

[0030] The specific process includes:

[0031] Step S21: The coordinates of any point on the panoramic sphere S are expressed in longitude and latitude. Indicates that the point corresponds to the pixel point on the panoramic image ( x , y ) is expressed as ,in ;

[0032] Step S22, a point on the projection plane P ( x , y ) is located at its microelement dxdy The center of , and then the microelement dxdy Mapped onto the panoramic sphere S, the area is The microelement , micro yuan dxdy and microelement The relationship between can be expressed as ,in It is the Jacobian determinant, which can be used to express the calculation relationship of the mapping area from the projection plane to the sphere

[0033] ;

[0034] Step S23: Area on projection plane P The mathematical formula can be expressed as , the area on the panoramic sphere S The mathematical formula can be expressed as , so the area stretching ratio from the panoramic sphere S to the projection plane P is for ;

[0035] Step S24: For the equirectangular projection of the panoramic image, the corresponding relationship between the latitude and longitude coordinates in the image and the projection plane coordinates is: , then the area stretching ratio can be obtained as , The area stretching ratio is the distortion value of the pixel, and the distortion values ​​of all pixels constitute the distortion perception map W .

[0036] Step S3, using the equirectangular projection map and the distortion perception map of the panoramic image as inputs of the convolution branch, and using the slice projection map and the distortion perception map as inputs of the attention mechanism branch, wherein the slice projection map is obtained by using the equirectangular projection of the panoramic image;

[0037] The backbone network of the convolutional branch is composed of a ResNet encoder, which uses residual connections and multi-scale feature extraction to effectively capture local features and details, which is crucial for accurate depth perception of outdoor scene details. The equirectangular projection image of size H×W and the distorted perception image W of the same size are used as the input of the convolutional branch. , the convolution branch i The encoded features of the layer output are In this embodiment, the convolution branch has 5 layers of convolution. The convolution kernel used in the first layer is 7×7, and the convolution kernels of the remaining four layers are all 3×3. In the first convolution of the input convolution branch, the output encoding feature is , channel C is 32, and the encoding features output by the 2nd, 3rd, 4th and 5th layers are: , , , .

[0038] The backbone network of the attention mechanism branch is the visual attention mechanism network, which is good at capturing long-range dependencies. The equirectangular projection image is projected onto the panoramic sphere. For each point on the panoramic sphere, each point is associated with its nearest m = 8 points form a slice block, the slice projection map consists of slice blocks, linear embedding is applied to flatten the slice block to generate the initial feature representation, which is used as the input of the first layer of the attention mechanism branch, and then a continuous multi-layer attention mechanism is applied to generate multi-layer encoding features. Each layer of the attention mechanism includes a multi-head self-attention (MSA) module and a multi-layer perceptron (MLP), and then layer normalization (LN) is applied before MSA and MLP, and jump connections are used within each module. i The encoded features of the layer output are ,in, , MSA stands for multi-head attention mechanism, MLP stands for multi-layer perceptron, and LN stands for layer normalization. express i The output features of the multi-head self-attention module, express i -1 layer of attention features. In this embodiment, the attention mechanism branch has four layers of output.

[0039] Step S4: The decoding module includes sequentially connected n decoding layer, the convolution branch i +1 layer output encoding features and attention mechanism branch i The encoded features output by the layer are input into the i Fusion module, i The output of the fusion module is input iA decoding layer is used to generate a depth map by decoding, and the depth map generated by decoding is transformed to obtain a 3D reconstruction map with complete spatial information. Among them, the fusion module is implemented based on the dual attention mechanism of space and channel. n , i are integers, i =1,2,…, n In this embodiment, n =4.

[0040] The fusion module is implemented based on a dual attention mechanism of space and channels, which can enhance the estimation network's ability to capture spatial and channel information from panoramic images and help to more fully understand the scene's 3D structure. In the convolution branch, spatial attention is used to obtain key information from different position areas. By weighting the features of each position, it adaptively focuses on image areas related to the depth estimation task while weakening irrelevant background areas, which helps the estimation network to more effectively utilize spatial position information and enhance its perception and understanding of panoramic images. In the attention mechanism branch, a channel attention mechanism is used to enhance the encoded features on different channels of the image. The channel attention mechanism enables the network to better capture the feature representations of different channels in the panoramic image by adaptively weighting the encoded features of each channel.

[0041] For the convolution branch i Encoded features of layer output , the layer corresponding to i The fusion module encodes the features After performing maximum pooling and mean pooling respectively, they are concatenated to form a feature map, and then a convolutional layer is used to learn the feature map. Finally, the Sigmoid function is used to obtain the channel attention result. .

[0042] For the attention mechanism branch i Encoded features of layer output , the layer corresponding to i The fusion module encodes the features Perform global pooling, then pass through a fully connected layer, and finally use the Sigmoid function to get the spatial attention result .

[0043] No. i The attention results of the fusion module on the channel And spatial attention results Using the formula Fusion to get output F i ,in, , Conv() represents convolution, W 1. W2 represents two 3×3 convolutional layers, Re sidual Represented as residual connection, it is the bottleneck block for feature compression in ResNet. For each residual function, three layers of stacking are used, namely 1×1, 3×3 and 1×1 convolutions. The 1×1 convolution layer is responsible for first reducing and then increasing the dimension, so that the number of channels of the 3×3 convolution layer is reduced, the number of parameters is reduced, and the computing bottleneck is reduced.

[0044] The decoding layer includes an upsampling block and a decoding block with the output of the upsampling module. The decoding module is implemented by the attention mechanism. The output of the fusion module is used as the input of the upsampling module of the corresponding decoding layer to finally obtain a size of H × W Depth map of .

[0045] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. A distortion-aware outdoor panoramic depth estimation method, characterized in that: The steps include: Step S1: construct an estimation network, the estimation network includes a collaborative network, n The collaborative network includes a convolution branch and an attention mechanism branch. The convolution branch is used to extract local features from the equirectangular projection of the panoramic image, and the attention mechanism branch is used to extract global features from the slice projection of the panoramic image. Step S2: based on the panoramic spherical imaging model, the area stretching ratio of the panoramic spherical surface to the projection surface is calculated, and the distortion value of each pixel point on the panoramic image is obtained accordingly, and then the distortion perception map corresponding to the panoramic image is obtained; Step S3, using the equirectangular projection map and the distortion perception map of the panoramic image as inputs of the convolution branch, and using the slice projection map and the distortion perception map as inputs of the attention mechanism branch, wherein the slice projection map is obtained by using the equirectangular projection of the panoramic image; Step S4, the decoding module includes sequentially connected n decoding layer, the convolution branch i +1 layer output encoding features and attention mechanism branch i The encoded features output by the layer are input into the i Fusion module, i The output of the fusion module is input i A decoding layer is used to decode and generate a depth map, where the fusion module is implemented based on the dual attention mechanism of space and channel. n , i are integers, i =1,2,…, n ; In step S3, the equirectangular projection image is projected onto the panoramic sphere, and for each point on the panoramic sphere, each point is m Points form a slice block, and the slice projection image is composed of slice blocks. The attention mechanism branch applies the attention mechanism in each slice block. i The encoded features of the layer output are .

2. The method for outdoor panoramic depth estimation based on distortion perception according to claim 1, characterized in that: In step S1, the backbone network of the convolution branch is composed of a ResNet encoder, and the backbone network of the attention mechanism branch is a visual attention mechanism network.

3. The distortion-aware outdoor panoramic depth estimation method according to claim 2, characterized in that: In step S2, the distortion perception map is composed of distortion values ​​of all pixels in the panoramic image, where the pixel ( x , y ) is expressed as , .

4. The distortion-aware outdoor panoramic depth estimation method according to claim 3, characterized in that: In step S3, the equirectangular projection image of size H×W and the distortion perception image of the same size are used as the input of the convolution branch. i The encoded features of the layer output are .

5. The distortion-aware outdoor panoramic depth estimation method according to claim 4, characterized in that: In step S4, the convolution branch i Encoded features of layer output , the layer corresponding to i The fusion module encodes the features After performing maximum pooling and mean pooling respectively, they are concatenated to form a feature map, and then a convolutional layer is used to learn the feature map. Finally, the Sigmoid function is used to obtain the channel attention result. .

6. The distortion-aware outdoor panoramic depth estimation method according to claim 5, characterized in that: In step S4, the attention mechanism branch i Encoded features of layer output , the layer corresponding to i The fusion module encodes the features Perform global average pooling, then pass through a fully connected layer, and finally use the Sigmoid function to obtain the spatial attention result .

7. The distortion-aware outdoor panoramic depth estimation method according to claim 6, characterized in that: In step S4, i The attention results of the fusion module on the channel And spatial attention results Using the formula Fusion to get output ,in, , W 1. W 2 represents two 3×3 convolutional layers, Residual represents the residual connection function, and Conv represents the convolution function.

8. The distortion-aware outdoor panoramic depth estimation method according to claim 7, characterized in that: In step S4, the decoding layer includes an upsampling block and a decoding block output by the upsampling module, the decoding module is implemented by an attention mechanism, and the output of the fusion module is used as the input of the upsampling module of the corresponding decoding layer.

9. The distortion-aware outdoor panoramic depth estimation method according to claim 8, characterized in that: In the step S4, the depth map generated by decoding is transformed to obtain a three-dimensional reconstructed map with complete spatial information.

Images