Isometric cylindrical projection distortion correction method and device for monocular panorama depth estimation

Abstract

The invention discloses an equidistant cylindrical projection distortion correction method and device for monocular panorama depth estimation, which can capture a smaller object, have a smoother and more accurate depth result and have a better distortion restoration effect. The method comprises the following steps: (1) taking a color panoramic image shot by a panoramic camera in an actual scene as input, carrying out adaptive selection distortion correction, and introducing a learnable mask while reducing discontinuous features in a cubic projection branch, so that an equidistant columnar projection branch learns valuable contents from the cubic projection branch; (2) executing a self-structure smoothing mechanism, adopting a structure information perception strategy of equidistant columnar projection, and designing a depth distribution fusion strategy to correct a distorted area of the equidistant columnar projection; and (3) supervising the training process of the network through a Berhu loss function.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to an equirectangular projection distortion correction method for monocular panoramic image depth estimation, and an equirectangular projection distortion correction device for monocular panoramic image depth estimation. Background Art

[0002] As a basic technology of computer vision, panoramic depth estimation is a fundamental task in computer vision, which aims to assign depth values ​​to each pixel in a panoramic image to reconstruct the three-dimensional structure of the scene. Panoramic images are usually represented in three main forms: equirectangular projection, cubic projection and tangent projection.

[0003] Each representation has its own advantages and disadvantages: Equirectangular projection projects a panoramic scene onto a rectangular image, providing a wider field of view, thus capturing the entire panoramic scene. However, this projection produces distortions when mapping a spherical panoramic scene onto a rectangular image. These distortions are most noticeable at the edges, top, and bottom of the image, which may cause distortion of the panoramic image. On the other hand, cubic projection involves mapping a spherical scene onto a cube map, thus avoiding the distortions associated with equirectangular projection. However, cubic projection produces discontinuities at the edges of each cube face and has a limited field of view. Tangent images are a two-dimensional local planar representation formed by mapping points in a panoramic image onto a plane tangent to a cylinder or sphere. By mapping to tangent images, geometric distortions at the edges of the panoramic image can be reduced, improving the accuracy of depth estimation. However, the depth values ​​predicted by multiple tangent images for the same part may be inconsistent, and how to resolve this conflict in the process of fusing tangent images is also a research difficulty. Therefore, in order to generate high-quality depth information, it is necessary to deal with these discontinuities and distortions in the panoramic image as well as the problem of depth value conflicts. Existing depth estimation methods can be roughly divided into the following three categories according to the models adopted.

[0004] The first type of method is the panorama depth estimation method based only on equirectangular projection. This method directly obtains the depth map through the panorama of equirectangular projection. Some methods will correct the distortion of equirectangular projection first and then predict the depth map. The ACDNet model uses a set of dilated convolutions to adjust the receptive field. The OmniDepth model uses rectangular convolutions arranged in rows to reduce distortions at different latitudes. Tateno et al. studied the method of training perspective images using standard convolutions, adjusting the shape of the sampling grid to adapt to spherical distortions during reasoning. In contrast, the SliceNet model vertically divides the equirectangular projection image into multiple slices and directly applies standard convolutions to predict the depth map of the equirectangular projection. However, due to the inherent distortion of the equirectangular projection, the results of depth estimation are limited.

[0005] The second category of methods is the panorama depth estimation method based only on tangent images. This method first maps the panorama of equirectangular projection into multiple tangent images, then predicts the depth value of each tangent image separately, and finally fuses multiple tangent images to obtain the final panoramic depth map. A recent work by Eder et al. proposed the use of tangent images, which are rendered on the faces of an icosahedron to represent 360-degree images. The advantage of using tangent images is that they have reduced distortion and can effectively utilize pre-trained convolutional neural network models originally developed for perspective imaging. Consistent with this, the frameworks of the 360MonoDepth model and the OmniFusion model use tangent images as input, and add geometric features to the fusion process to solve the problem of depth conflict. Different from the above methods, the HRDFuse model predicts the depth map of equirectangular projection format by collaboratively learning the global and regional depth distributions.

[0006] The third type of method is a panoramic depth estimation method that fuses equirectangular projection and cubic projection. This method refers to repairing the distorted parts of the equirectangular projection through cubic projection. The BiFuse model proposes a bidirectional fusion method that integrates features from equirectangular projection and cubic projection at the encoding and decoding stages. In contrast, the UniFuse model chooses to fuse these features only in the encoding stage because it believes that the equirectangular projection features play a more critical role in the final depth prediction. However, this fusion scheme cannot discard the useless parts of the cubic projection. There are problems of slow convergence and lack of global analysis when regressing pixel depth values, and directly regressing the depth value of each pixel will result in an unsmooth and inaccurate depth estimation result. Summary of the invention

[0007] In order to overcome the defects of the prior art, the technical problem to be solved by the present invention is to provide an equirectangular projection distortion correction method for monocular panoramic image depth estimation, which can capture smaller objects, has smoother and more accurate depth results, and has a better effect of repairing distortion.

[0008] The technical solution of the present invention is: the equirectangular projection distortion correction method for monocular panoramic image depth estimation comprises the following steps:

[0009] (1) Taking the color panoramic image captured by the panoramic camera in the actual scene as input, adaptive selective distortion correction is performed to reduce the discontinuity features in the cubic projection branch while introducing a learnable mask so that the equirectangular projection branch can learn valuable content from the cubic projection branch;

[0010] (2) Implement a self-structure smoothing mechanism, adopt the structural information perception strategy of equirectangular projection, and design a depth distribution fusion strategy to correct the distorted areas of equirectangular projection;

[0011] (3) The network training process is supervised by the Berhu loss function.

[0012] The selective distortion correction mechanism of the present invention is designed to capture effective information in the cubic projection to correct the distorted area; the self-structure smoothing mechanism is designed to perceive the structural information of the entire equidistant cylindrical projection, and then correct the distorted area of ​​the equidistant cylindrical projection through a depth distribution fusion strategy; therefore, the present invention can capture smaller objects, has smoother and more accurate depth results, and has a better effect of repairing distortion.

[0013] A device for correcting equirectangular projection distortion of a monocular panoramic image depth estimation is also provided, the device comprising:

[0014] An adaptive selective distortion correction module configured to take a color panoramic image captured by a panoramic camera in the actual scene as input and perform adaptive selective distortion correction, while mitigating discontinuities in the cubic projection branch and introducing a learnable mask so that the equirectangular projection branch learns valuable content from the cubic projection branch;

[0015] A self-structured smoothing module, which is configured to perform a self-structured smoothing mechanism, adopts a structural information perception strategy of equirectangular projection, and designs a deep distribution fusion strategy to correct the distorted areas of equirectangular projection;

[0016] The loss module is configured to supervise the training process of the network through the Berhu loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A flow chart of an equirectangular projection distortion correction method for monocular panoramic image depth estimation according to the present invention is shown. DETAILED DESCRIPTION

[0018] The purpose of the present invention is to solve the problem of distortion of equirectangular projection panoramas, so as to obtain a panoramic depth map with smaller errors. Different from the above-mentioned other methods, the present invention proposes a DistortionCorrection Network of Equirectangular Projection (DCN-ERP), which mainly includes the following parts: First, an adaptive selection distortion correction mechanism (ADCM) is proposed, which enables the equirectangular projection branch to adaptively select useful information from the cubic projection branch to correct the distortion during the fusion process. In addition, a self-structured smoothing mechanism (SSM) module is proposed, which uses its own structural information and a depth distribution design to correct the distortion in depth estimation. The main technologies involved in the present invention are: monocular depth estimation suitable for panoramas; adaptive panoramic distortion correction mechanism; self-structured smoothing mechanism.

[0019] like Figure 1 As shown, the equirectangular projection distortion correction method for monocular panoramic image depth estimation includes the following steps:

[0020] (1) Taking the color panoramic image captured by the panoramic camera in the actual scene as input, adaptive selective distortion correction is performed to reduce the discontinuity features in the cubic projection branch while introducing a learnable mask so that the equirectangular projection branch can learn valuable content from the cubic projection branch;

[0021] (2) Implement a self-structure smoothing mechanism, adopt the structural information perception strategy of equirectangular projection, and design a depth distribution fusion strategy to correct the distorted areas of equirectangular projection;

[0022] (3) The network training process is supervised by the Berhu loss function.

[0023] The selective distortion correction mechanism of the present invention is designed to capture effective information in the cubic projection to correct the distorted area; the self-structure smoothing mechanism is designed to perceive the structural information of the entire equidistant cylindrical projection, and then correct the distorted area of ​​the equidistant cylindrical projection through a depth distribution fusion strategy; therefore, the present invention can capture smaller objects, has smoother and more accurate depth results, and has a better effect of repairing distortion.

[0024] Preferably, in step (1), the formula for adaptively selecting the distortion correction network is as follows:

[0025] F′ res =ReLU(BN(Conv 1×1 (Cat(F equi , F c2e )))

[0026] F res =BN(Conv 3×3 (F′ res )) (2)

[0027] F′ c2e =F c2e +F res (3)

[0028] Mask=Sigmoid(Conv 1×1 (F res )) (4)

[0029] F fused =SE(F equi +Mask·F′ c2e ) (5)

[0030] Among them, Cat represents the concatenation operation, Conv 1×1 Represents a 1×1 convolution operation, Conv 3×3 represents a 3×3 convolution operation, BN is batch normalization, ReLU is the rectified linear unit activation function, Sigmoid is the Sigmoid activation function, SE is the compression and excitation network, which acts on the equirectangular projection feature map that integrates the information of the cubic projection feature map. It adaptively recalibrates the channel feature response, thereby enhancing the effectiveness of the fusion.

[0031] Preferably, the step (1) comprises the following sub-steps:

[0032] (1.1) Obtain the encoding feature F from the equidistant cylindrical projection branch equi , the encoded feature F from the cube projection branch cube ;

[0033] (1.2) Project the feature map F of the cube cuve F converted to equirectangular projection format by mapping c2e , F equi and F c2e Put them together to get F cat , allowing F c2e Get global information within the equirectangular projection;

[0034] (1.3)Fcat After convolution, F res , used to compensate F c2e Discontinuous features within

[0035] (1.4) Introduce a mask, through F res The convolution and sigmoid function inside the mask are generated. The value of this mask is between 0 and 1. The purpose of the mask is to emphasize learning from the relevant parts of the cube projection as much as possible while minimizing the impact of discontinuous areas;

[0036] (1.5)F′ c2e Multiply by Mask and add to F equi On, so that F equi Selectively learn F′ c2e The fused features are passed through SE to obtain the final result;

[0037] (1.6) Repair the distorted equirectangular projection format feature map from the cubic projection feature.

[0038] Preferably, the step (2) comprises the following sub-steps:

[0039] (2.1) The feature map of the equirectangular projection branch is encoded by the Transformer encoder to obtain an embedding vector containing rich structural information. Through the self-attention mechanism, the Transformer encoder captures the long-range dependencies in the feature map and learns the global structural information of the image;

[0040] (2.2) Using the embedding vector output by the encoder, the center point of the depth distribution histogram is predicted. The self-structured smoothing mechanism consists of two branches: the lower branch involves passing the feature map through the Transformer encoder to obtain the query, which is then passed to the MLP head to obtain the depth distribution; the upper branch generates a keymap through convolution; the query from the lower branch and the keymap from the upper branch perform a dot product operation to produce a range attention map, which captures the global context information and is spatially aligned with the equirectangular projection feature map;

[0041] (2.3) The softmax activation function is applied to predict the probability score map, and the final depth value is a linear combination of the center points of the depth distribution to smooth the depth estimation task.

[0042] Preferably, the step (2) comprises the following sub-steps:

[0043] (i) Input is the equirectangular projection feature map fused with the cube projection information, denoted as F ec ;

[0044] (ii)F ec The features are convolved to obtain E ti , E ti After the transformer encoder, we get E to ;

[0045] (iii)E to The first token E in to [0] is used to predict the distribution center C of the depth distribution histogram, and the rest is used as the query;

[0046] (iv)F ec After convolution, the keymap is obtained, and the dot product of the query and the keymap is calculated to obtain the range attention map R. R undergoes a 1×1 convolution and then generates a probability score map P through a softmax activation function;

[0047] (v) With the complete depth distribution center C and probability score map P, the final depth map is a linear combination of the depth distribution centers.

[0048] Preferably, in step (ii),

[0049] E to The first token E in to [0] is used to predict the distribution center of the depth distribution histogram

[0050] C, the specific formula is as follows:

[0051]

[0052] For i and j, the range is from 1 to B, where B represents the number of depth distribution intervals, W represents the depth range in the distribution histogram, mlp is a multilayer perceptron head with a ReLU activation function, (d min ,d max ) represents the depth range of the dataset and ε is a small constant used to ensure that each W value is positive.

[0053] Preferably, in step (v), the range attention map R undergoes a 1×1 convolution, and then generates a probability score map P through a softmax activation function, having a complete depth distribution center and a probability score map, and the following formula is used to calculate the overall depth map

[0054]

[0055] Preferably, in step (3), the formula of BerHu loss is as follows:

[0056]

[0057] The Berhu loss function is expressed as is defined as a piecewise function that depends on the predicted value and the actual value d i The absolute difference between |Δ| and there is a threshold c; if the absolute difference |Δ| is less than or equal to c, the loss is the absolute difference; if |Δ| is greater than c, the loss becomes a combination of the squared difference and the threshold; for each input image, the parameter c is set to:

[0058]

[0059] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, CD, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes an equirectangular projection distortion correction device for monocular panoramic image depth estimation, which is usually represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0060] An adaptive selective distortion correction module configured to take a color panoramic image captured by a panoramic camera in the actual scene as input and perform adaptive selective distortion correction, while mitigating discontinuities in the cubic projection branch and introducing a learnable mask so that the equirectangular projection branch learns valuable content from the cubic projection branch;

[0061] A self-structured smoothing module, which is configured to perform a self-structured smoothing mechanism, adopts a structural information perception strategy of equirectangular projection, and designs a deep distribution fusion strategy to correct the distorted areas of equirectangular projection;

[0062] The loss module is configured to supervise the training process of the network through the Berhu loss function.

[0063] Preferably, the adaptive selection distortion correction module performs the following steps:

[0064] (1.1) Obtain the encoding feature F from the equidistant cylindrical projection branch equi , the encoded feature F from the cube projection branch cube ;

[0065] (1.2) Project the feature map F of the cube cube F converted to equirectangular projection format by mapping c2e , F equi and F c2e Put them together to get F cat , allowing Fc2e Get global information within the equirectangular projection;

[0066] (1.3)F cat After convolution, F res , used to compensate F c2e Discontinuous features within

[0067] (1.4) Introduce a mask, through F res The convolution and sigmoid function inside the mask are generated. The value of this mask is between 0 and 1. The purpose of the mask is to emphasize learning from the relevant parts of the cube projection as much as possible while minimizing the impact of discontinuous areas;

[0068] (1.5)F′ c2e Multiply by Mask and add to F equi On, so that F equi Selectively learn F′ c2e The fused features are passed through SE to obtain the final result;

[0069] (1.6) Repair the distorted equirectangular projection format feature map from the cubic projection feature.

[0070] The method proposed in the present invention is suitable for solving the problem of equirectangular projection distortion in the panoramic depth estimation task. The proposed equirectangular projection distortion correction network contains two novel designs: a selective distortion correction mechanism and a self-structural smoothing mechanism. The selective distortion correction mechanism is designed to capture effective information in the cube projection to correct the distorted area. The self-structural smoothing mechanism is designed to perceive the structural information of the entire equirectangular projection, and then correct / refine the distorted part through a smooth depth regression model. Table 1 shows a comparison between some advanced panoramic depth estimation methods and this method on the Mattport3D dataset. Table 2 shows a comparison between some advanced panoramic depth estimation methods and this method on the Standford2D3D dataset. Table 3 shows the role of the ADCM and SSM modules. Table 4 shows a comparison of the number of parameters of this method with other methods.

[0071]

[0072] Table 1

[0073]

[0074] Table 2

[0075]

[0076] Table 3

[0077]

[0078] Table 4

[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. An equirectangular projection distortion correction method for monocular panoramic image depth estimation, characterized by: The method comprises the following steps: (1) Taking the color panoramic image captured by the panoramic camera in the actual scene as input, adaptive selective distortion correction is performed to reduce the discontinuity features in the cubic projection branch while introducing a learnable mask so that the equirectangular projection branch can learn valuable content from the cubic projection branch; (2) Implement a self-structure smoothing mechanism, adopt the structural information perception strategy of equirectangular projection, and design a depth distribution fusion strategy to correct the distorted areas of equirectangular projection; (3) The network training process is supervised by the Berhu loss function.

2. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 1, characterized in that: In the step (1), the formula for adaptively selecting the distortion correction network is as follows: F′ res =ReLU(BN(Conv 1×1 (Cat(F equi ,F c2e )))) (1) F res =BN(Conv 3×3 (F′ res )) (2) F′ c2e =F c2e +F res (3) Mask=Sigmoid(Conv 1×1 (F res )) (4) F fused =SE(F equi +Mask·F′ c2e ) (5) Among them, Cat represents the concatenation operation, Conv 1×1 Represents a 1×1 convolution operation, Conv 3×3 represents a 3×3 convolution operation, BN is batch normalization, ReLU is the rectified linear unit activation function, Sigmoid is the Sigmoid activation function, SE is the compression and excitation network, which acts on the equirectangular projection feature map that integrates the information of the cubic projection feature map. It adaptively recalibrates the channel feature response, thereby enhancing the effectiveness of the fusion.

3. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 2, characterized in that: The step (1) comprises the following sub-steps: (1.1) Obtain the encoding feature F from the equidistant cylindrical projection branch equi , the encoded feature F from the cube projection branch cube ; (1.2) Project the feature map F of the cube cube F converted to equirectangular projection format by mapping c2e , F equi and F c2e Put them together to get F c2e , allowing F c2e Get global information within the equirectangular projection; (1.3)F cat After convolution, F res , used to compensate F c2e Discontinuous features within (1.4) Introduce a mask, through F res The convolution and sigmoid function inside the mask are generated. The value of this mask is between 0 and 1. The purpose of the mask is to emphasize learning from the relevant parts of the cube projection as much as possible while minimizing the impact of discontinuous areas; (1.5)F′ c2e Multiply by Mask and add to F equi On, so that F equi Selectively learn F′ c2e The fused features are passed through SE to obtain the final result; (1.6) Repair the distorted equirectangular projection format feature map from the cubic projection feature.

4. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 3, characterized in that: The step (2) comprises the following sub-steps: (2.1) The feature map of the equirectangular projection branch is encoded by the Transformer encoder to obtain an embedding vector containing rich structural information. Through the self-attention mechanism, The Transformer encoder captures long-range dependencies in feature maps and learns the global structural information of the image; (2.2) Using the embedding vector output by the encoder, the center point of the depth distribution histogram is predicted. The self-structured smoothing mechanism consists of two branches: the lower branch involves passing the feature map through The Transformer encoder obtains the query, which is then passed to the MLP head to obtain the deep distribution; the upper branch generates a keymap through convolution; the query from the lower branch and the keymap from the upper branch perform a dot product operation to produce a range attention map, which captures the global context information and is spatially aligned with the equirectangular projection feature map; (2.3) The softmax activation function is applied to predict the probability score map, and the final depth value is a linear combination of the center points of the depth distribution to smooth the depth estimation task.

5. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 3, characterized in that: The step (2) comprises the following sub-steps: (i) Input is the equirectangular projection feature map fused with the cube projection information, denoted as F ec ; (ii) F ec The features are convolved to obtain E ti , E ti After the transformer encoder, we get E to ; (iii)E to The first token E in to [0] is used to predict the distribution center C of the depth distribution histogram, and the rest is used as the query; (iv)F ec After convolution, the keymap is obtained, and the dot product of the query and the keymap is calculated to obtain the range attention map R. R undergoes a 1×1 convolution and then generates a probability score map P through a softmax activation function; (v) With the complete depth distribution center C and probability score map P, the final depth map is a linear combination of the depth distribution centers.

6. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 5, characterized in that: In the step (ii), E to The first token E in to [0] is used to predict the distribution center C of the depth distribution histogram. The specific formula is as follows: For i and j, the range is from 1 to B, where B represents the number of depth distribution intervals, W represents the depth range in the distribution histogram, mlp is a multilayer perceptron head with a ReLU activation function, (d min , d max ) represents the depth range of the dataset and ε is a small constant used to ensure that each W value is positive.

7. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 6, characterized in that: In step (v), the range attention map R undergoes a 1×1 convolution and then generates a probability score map P through a softmax activation function. With a complete depth distribution center and probability score map, the overall depth map is calculated using the following formula:

8. The equirectangular projection distortion correction method for monocular panoramic image depth estimation according to claim 7, characterized in that: In step (3), the formula of BerHu loss is as follows: The Berhu loss function is expressed as is defined as a piecewise function that depends on the predicted value and the actual value d i The absolute difference between |Δ| and there is a threshold c; if the absolute difference |Δ| is less than or equal to c, the loss is the absolute difference; if |Δ| is greater than c, the loss becomes a combination of the squared difference and the threshold; for each input image, the parameter c is set to:

9. An equirectangular projection distortion correction device for monocular panoramic image depth estimation, characterized in that: The device includes: An adaptive selective distortion correction module configured to take a color panoramic image captured by a panoramic camera in the actual scene as input and perform adaptive selective distortion correction, while mitigating discontinuities in the cubic projection branch and introducing a learnable mask so that the equirectangular projection branch learns valuable content from the cubic projection branch; A self-structured smoothing module, which is configured to perform a self-structured smoothing mechanism, adopts a structural information perception strategy of equirectangular projection, and designs a deep distribution fusion strategy to correct the distorted areas of equirectangular projection; The loss module is configured to supervise the training process of the network through the Berhu loss function.

10. The equirectangular projection distortion correction device for monocular panoramic image depth estimation according to claim 9, characterized in that: The adaptive selection distortion correction module performs the following steps: (1.1) Obtain the encoding feature F from the equidistant cylindrical projection branch equi , the encoded feature F from the cube projection branch cube ; (1.2) Project the feature map F of the cube cube F converted to equirectangular projection format by mapping c2e , F equi and F c2e Put them together to get F cat , allowing F c2e Get global information within the equirectangular projection; (1.3)F cat After convolution, F res , used to compensate F c2e Discontinuous features within (1.4) Introduce a mask, through F res The convolution and sigmoid function inside the mask are generated. The value of this mask is between 0 and 1. The purpose of the mask is to emphasize learning from the relevant parts of the cube projection as much as possible while minimizing the impact of discontinuous areas; (1.5)F′ c2e Multiply by Mask and add to F equi On, so that F equi Selectively learn F′ c2e The fused features are passed through SE to obtain the final result; (1.6) Repair the distorted equirectangular projection format feature map from the cubic projection feature.

Images