A panoramic depth estimation method combining spherical correction and semi-supervised learning

Abstract

The application discloses a panoramic image depth estimation method combining spherical correction and semi-supervised learning, innovatively designs a spherical correction (SR) mechanism based on spherical coordinate Fourier coding and feature linear modulation (FiLM), a lightweight plug-and-play PanoAdapter adaptive module, a teacher-student semi-supervised cross-domain knowledge transfer framework and a joint optimization strategy fusing pseudo-label distillation and consistency constraint, realizes apparent correction of equidistant cylindrical projection (ERP) nonlinear spherical distortion, alleviates the contradiction between panoramic labeling data scarcity and perspective model cross-domain migration difficulty, and improves the geometric consistency and generalization ability of panoramic depth estimation in diversified real scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a panoramic depth estimation method that combines spherical correction and semi-supervised learning. Background Technology

[0002] Panoramic images, with their 360°×180° full-coverage view, have become a crucial foundational data format for virtual reality (VR), autonomous navigation, robotic perception, and immersive media. Especially with the rapidly growing demand for real-time 3D reconstruction and environmental understanding, recovering high-precision depth information from single panoramic images has become a key industry technology. However, currently, panoramic images are commonly stored using equirectangular projection (ERP). While this projection method is easy to process, it introduces significant nonlinear spherical distortion, making traditional depth estimation models based on planar imaging assumptions difficult to use directly, particularly in high-latitude regions where severe structural distortion can occur. This places higher demands on the generalization ability and application stability of algorithms.

[0003] Although some specialized network structures for panoramas have emerged in recent years (such as OmniDepth and UniFuse), or methods attempting to transfer large-scale perspective depth models (such as MiDaS and Depth Anything) to the panoramas (such as PanDA), none have effectively resolved the contradiction between geometric distortion, data scarcity, and transfer difficulties. Specifically: methods like UniFuse rely excessively on scarce panoramas with annotation data, making them prone to overfitting and resulting in poor generalization under different scenes, lighting conditions, and geometric structures; the method used by PanDA cannot handle the nonlinear spherical distortion of equidistant cylindrical projections. Its fine-tuning-based approach only implicitly adjusts the internal statistics of the network without explicitly processing the spherical geometry, leading to problems such as structural collapse and depth discontinuities in the prediction.

[0004] Therefore, how to propose a panoramic depth estimation method that can both explicitly model and correct spherical geometric distortions and effectively utilize prior knowledge of perspective-based models to reduce annotation costs is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the aforementioned issues, this invention discloses a panoramic depth estimation method combining spherical correction and semi-supervised learning. It introduces a spherical correction (SR) mechanism based on spherical coordinate Fourier coding and characteristic linear modulation (FiLM), a lightweight plug-and-play PanoAdapter adaptation module, a teacher-student semi-supervised cross-domain knowledge transfer framework, and a joint optimization strategy integrating pseudo-label distillation and consistency constraints. This achieves explicit correction of nonlinear spherical distortion in equidistant cylindrical projection, alleviating the contradiction between the scarcity of panoramic annotation data and the difficulty of cross-domain transfer of perspective models, and improving the geometric consistency and generalization ability of panoramic depth estimation in diverse real-world scenarios.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A panoramic depth estimation method combining spherical correction and semi-supervised learning includes the following steps:

[0008] Step 1: Construct the teacher model and perform supervised fine-tuning. A pre-trained perspective depth baseline model is selected as the backbone network of the teacher model. This baseline model has learned general monocular depth estimation capabilities on large-scale perspective image data. To adapt the teacher model to the panoramic depth estimation task, low-rank adaptation (LoRA) is used on a labeled panoramic dataset. The teacher model underwent its first stage of fine-tuning. During this fine-tuning process, each labeled panoramic image was... Input the teacher model to obtain the predicted depth map. and compare it with the corresponding true depth map. Compare and calculate the supervision loss. This optimizes the teacher model parameters, enabling it to perform preliminary depth estimation for panoramic scenes. Step 2: Generate pseudo-depth labels using the teacher model. Using the teacher model trained in Step 1, generate pseudo-depth labels for the unlabeled panoramic dataset. Each unlabeled panoramic image in Perform forward inference to obtain the corresponding predicted depth map. Since the sky region of panoramic images lacks effective depth information, directly using it as a pseudo-label would introduce significant noise. Therefore, a semantic segmentation model is used to... Perform semantic parsing to generate a sky region mask. This mask is used to filter and zero out the depth values ​​of the sky region in the predicted depth map, ultimately generating high-quality pseudo-depth labels. The pseudo-depth label will be used in step 5 along with the labeled data for joint training of the student model. Step 3: Construct the student model. The backbone network structure of the student model is consistent with that of the teacher model, also based on the encoder-decoder architecture of the perspective depth base model. On this basis, the PanoAdapter module proposed in this invention is inserted between each Transformer layer of the encoder. The role of the PanoAdapter module is to introduce the spherical geometric prior information unique to panoramic images without changing the parameters of the original perspective base model, so that the feature representation can adapt to the isometric cylindrical projection distortion characteristics of panoramic images. The trainable parameters of the student model only include the parameters of the PanoAdapter module and the parameters of the decoder. Step 4: Adaptively correct panoramic image features through a spherical correction mechanism. The core of the PanoAdapter module is the spherical correction mechanism, which aims to map planar perspective features back to spherical geometric space. For the input panoramic image After being extracted by the encoder, it is in the first... The feature map output by the Transformer block is ,in These are the height and width of the feature map, respectively (in pixels). For feature channel dimensions.

[0009] First, the pixel coordinates of the feature map Mapped to azimuth in spherical coordinate system and polar angle To describe the cyclic continuity of the panoramic image in the horizontal direction, the origin of the coordinate system is defined at the center of the image, and its mapping relationship is shown in equation (1):

[0010] (1);

[0011] Subsequently, to map the spherical coordinates to a high-dimensional feature space ,in To embed the dimension, the azimuth angle and polar angle Encoding is performed using Fourier position coding to obtain the spherical embedding. :

[0012] (2);

[0013] Among them, the encoding function It consists of sine and cosine components of different frequencies. For the input variable... Its form is shown in equation (3): (3);

[0014] in, It is a frequency vector, and T denotes matrix transpose. , No. Each frequency component is defined as a geometric series as shown in equation (4):

[0015] (4);

[0016] Generated spherical embedding This will be used as a conditionally embedded input feature-wise linear modulation (FiLM) module to correct spherical distortion of the features. Specifically, for the ... Output features of layer Transformer blocks Using a lightweight MLP network to embed from a sphere Dynamic Prediction Modulation Parameters (Scaling factor) and (Translation coefficient), as shown in equation (5):

[0017] (5);

[0018] Subsequently, the modulation parameters were used to adjust the first... Original features of the layer Perform position-by-position affine transformation to obtain the corrected features. The calculation method is shown in equation (6):

[0019] (6);

[0020] in, This represents element-wise multiplication. This mechanism allows the model to maintain its original feature response in the low-distortion equatorial region, while adaptively adjusting the feature representation in the high-distortion polar region, thereby compensating for the spatial distortion caused by equidistant cylindrical projection. Step 5: During training, freeze the perspective base model (i.e., all parameters in the encoder except PanoAdapter), and only optimize and update the PanoAdapter module parameters and decoder parameters of the student model.

[0021] The training data consists of a combination of labeled and pseudo-labeled data, and the images in the joint training set are denoted as . For each one Apply color enhancement transformations respectively and rotation transformation The corresponding data-enhanced image is obtained. and .Will Input each student model separately to obtain their respective predicted depth maps. .

[0022] Total loss function It consists of a weighted average of the semi-supervised learning loss and the consistency loss, as shown in equation (7):

[0023] (7);

[0024] Here, For semi-supervised learning loss, This results in a loss of consistency.

[0025] The semi-supervised learning loss is shown in equation (8):

[0026] (8);

[0027] in, For supervised loss of labeled data, For pseudo-depth labels generated in step 2 Calculated pseudo-label distillation loss; and These are the weight coefficients corresponding to the labeled loss and pseudo-labeled loss, respectively, used to balance the contribution of the two types of data to model training. They are calculated in the same way, as shown in equation (9):

[0028] (9);

[0029] In the formula, Scale-and-shift invariant loss is used to eliminate the global scale and shift bias between the predicted depth and the true depth; Gradient matching loss is used to constrain the local structural consistency of the depth map.

[0030] The consistency loss is shown in equation (10):

[0031] (10);

[0032] Here, For color consistency loss, constrain the consistency of depth prediction between the original image and the color-enhanced image; The geometric isovariability loss constrains the consistency of depth prediction between the original image and the rotated transformed image; and These are the corresponding weighting coefficients. Specifically, their calculation methods are shown in equations (11) and (12):

[0033] (11);

[0034] (12);

[0035] in, Original image Predicted depth Enhance the color of the image Predicted depth For rotation transformation image The predicted depth. Equation (11) requires that after applying the same color enhancement transformation to the original predicted depth, the result should be consistent with the predicted depth of the color-enhanced image; Equation (12) requires that after applying the same rotation transformation to the original predicted depth, the result should be consistent with the predicted depth of the rotated image. This constraint makes the depth features learned by the model color invariant, rotation invariant and geometrically robust. Step 6: Inference stage. Input the panoramic image to be tested into the trained student model, extract features through the encoder, perform spherical correction through the PanoAdapter module, generate depth prediction through the decoder, and finally output a high-precision panoramic depth map.

[0036] The beneficial effects of this invention are as follows:

[0037] 1. Solved the geometric distortion problem: Addressing the shortcomings of existing technologies in handling ERP projection distortion, this invention innovatively introduces the SR mechanism. Through spherical position Fourier encoding and FiLM modulation, this mechanism can adaptively modulate image features according to the spatial physical coordinates of pixels, effectively eliminating nonlinear spherical distortion artifacts. Especially in high-latitude regions such as the polar regions, it can significantly reduce the root mean square error (RMSE) and absolute relative error (AbsRel) of depth prediction, completely solving the problem of panoramic depth structure collapse.

[0038] 2. Reduced training and computation costs: Compared to traditional full-parameter fine-tuning or de novo training of panoramic models, the student model of this invention only needs to train the inserted PanoAdapter module and decoder parameters, perfectly preserving the powerful general feature extraction capabilities of the perspective base model. This plug-in design significantly reduces the number of trainable parameters (e.g., only a very small percentage of the original model's parameters), greatly reducing dependence on computing power and GPU memory, and possessing strong engineering applicability.

[0039] 3. Enhanced generalization ability in real-world scenarios: Addressing the extreme scarcity of high-quality panoramic labeled data, this invention employs a pioneering semi-supervised cross-domain knowledge transfer framework. By generating and refining pseudo-label data through a teacher model, the student model is guided to learn structural consistency constraints, expanding the model's scene perception boundaries. This enables the invention to demonstrate significantly superior robustness across various depth evaluation metrics compared to similar methods in unseen real-world multi-class scenarios (zero-shot testing).

[0040] 4. High scalability and compatibility: The PanoAdapter module of this invention is designed independently of the specific backbone network and is not limited to a single pre-trained model. Instead, it can be used as a general plug-and-play component to adapt to any large vision model based on Transformer, enabling it to quickly acquire panoramic perception capabilities. Attached Figure Description

[0041] Figure 1 This is a flowchart of the present invention.

[0042] Figure 2 This is a diagram of the PanoAdapter training architecture described in this invention.

[0043] Figure 3 This is a schematic diagram of the PanoAdapter structure described in this invention.

[0044] Figure 4 This is a schematic diagram of the SR mechanism described in this invention. Detailed Implementation

[0045] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0046] like Figure 4 As shown, the panoramic depth estimation method combining spherical correction and semi-supervised learning described in this invention includes the following steps: Step 1: Construct a teacher model based on the perspective depth model, and perform first-stage fine-tuning on a labeled dataset using LoRA technology. Figure 1 As shown in (a), each labeled panoramic image Input the teacher model to obtain the predicted depth map. and compare it with the corresponding true depth map. Compare and calculate the supervision loss. This optimizes the teacher model parameters, enabling it to perform preliminary depth estimation for panoramic scenes. Step 2: (The rest of the text appears to be a separate, unrelated section.) Figure 1 As shown in (b), the trained teacher model is used on the unlabeled panoramic dataset. Each unlabeled panoramic image in Perform forward inference to obtain the corresponding predicted depth map. Since the sky region of panoramic images lacks effective depth information, directly using it as a pseudo-label would introduce significant noise. Therefore, a semantic segmentation model is used to... Perform semantic parsing to generate a sky region mask. This mask is used to filter and zero out the depth values ​​of the sky region in the predicted depth map, ultimately generating high-quality pseudo-depth labels. This pseudo-depth label will be used in step 5 along with the labeled data for joint training of the student model. Step 3: As... Figure 1 As shown in (c), a student model is constructed, whose backbone network is consistent with the teacher model, but a PanoAdapter module is inserted between the encoder layers; Step 4: The spherical coordinate information is transmitted through... Figure 2 The method shown is mapped to a spherical embedded input PanoAdapter module, and then through... Figure 3 The SR mechanism in the image corrects panoramic image features by combining them with spherical coordinate information;

[0047] The core of the PanoAdapter module is a spherical correction mechanism, designed to map planar perspective features back to spherical geometry. For example... Figure 3 As shown, for the input panoramic image After being extracted by the encoder, it is in the first... The feature map output by the Transformer block is ,in These are the height and width of the feature map, respectively (in pixels). For feature channel dimensions.

[0048] First, the pixel coordinates of the feature map Mapped to azimuth in spherical coordinate system and polar angle To describe the cyclic continuity of the panoramic image in the horizontal direction, the origin of the coordinate system is defined at the center of the image, and its mapping relationship is shown in equation (1):

[0049] (1);

[0050] Subsequently, to map the spherical coordinates to a high-dimensional feature space ,in To embed the dimension, the azimuth angle and polar angle Encoding is performed using Fourier position coding to obtain the spherical embedding. :

[0051] (2);

[0052] Among them, the encoding function It consists of sine and cosine components of different frequencies. For the input variable... Its form is shown in equation (3): (3);

[0053] in, It is a frequency vector, and T denotes matrix transpose. , No. Each frequency component is defined as a geometric series as shown in equation (4):

[0054] (4);

[0055] Generated spherical embedding This will be used as a conditionally embedded input feature-wise linear modulation (FiLM) module to correct spherical distortion of the features. Specifically, for the ... Output features of layer Transformer blocks Using a lightweight MLP network to embed from a sphere Dynamic Prediction Modulation Parameters (Scaling factor) and (Translation coefficient), as shown in equation (5):

[0056] (5);

[0057] Subsequently, the modulation parameters were used to adjust the first... Original features of the layer Perform position-by-position affine transformation to obtain the corrected features. The calculation method is shown in equation (6):

[0058] (6);

[0059] in, This represents element-wise multiplication. This mechanism allows the model to maintain its original feature response in the low-distortion equatorial region, while adaptively adjusting the feature representation in the high-distortion polar region, thereby compensating for the spatial distortion caused by equidistant cylindrical projection. Step 5: During training, freeze the perspective base model (i.e., all parameters in the encoder except for the PanoAdapter), and only optimize and update the PanoAdapter module parameters and decoder parameters of the student model, such as... Figure 1 As shown in (c) in the figure,

[0060] The training data consists of a combination of labeled and pseudo-labeled data, and the images in the joint training set are denoted as . For each one Apply color enhancement transformations respectively and rotation transformation The corresponding data-enhanced image is obtained. and .Will Input each student model separately to obtain their respective predicted depth maps. .

[0061] Total loss function It consists of a weighted average of the semi-supervised learning loss and the consistency loss, as shown in equation (7):

[0062] (7);

[0063] Here, For semi-supervised learning loss, This results in a loss of consistency.

[0064] The semi-supervised learning loss is shown in equation (8):

[0065] (8);

[0066] in, For supervised loss of labeled data, For pseudo-depth labels generated in step 2 Calculated pseudo-label distillation loss; and These are the weight coefficients corresponding to the labeled loss and pseudo-labeled loss, respectively, used to balance the contribution of the two types of data to model training. They are calculated in the same way, as shown in equation (9):

[0067] (9);

[0068] In the formula, Scale-and-shift invariant loss is used to eliminate the global scale and shift bias between the predicted depth and the true depth; Gradient matching loss is used to constrain the local structural consistency of the depth map.

[0069] The consistency loss is shown in equation (10):

[0070] (10);

[0071] Here, For color consistency loss, constrain the consistency of depth prediction between the original image and the color-enhanced image; The geometric isovariability loss constrains the consistency of depth prediction between the original image and the rotated transformed image; and These are the corresponding weighting coefficients. Specifically, their calculation methods are shown in equations (11) and (12):

[0072] (11);

[0073] (12);

[0074] in, Predict depth for the original image. To enhance the predicted depth of the image for color enhancement, This represents the predicted depth of the image after rotation transformation. This constraint ensures that the features learned by the model are rotation-invariant and geometrically robust. Step 6: Input the panoramic image to be tested into the trained student model, and output the final high-precision panoramic depth map.

[0075] Steps 2 and 5 of this invention employ a teacher-student paradigm. In the first stage, only the teacher model is trained, utilizing LoRA for rapid adaptation on synthetic data. In the second stage, the teacher model's pseudo-labels are frozen, and the student model is constructed. During training, the student model freezes all pre-trained parameters (ViT Encoder) except for the PanoAdapter module, training only the PanoAdapter and decoder modules. This design ensures efficient training and prevents overfitting of the model on small sample panoramic data. The specific training algorithm flowchart and framework are as follows... Figure 2 As shown.

[0076] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A panoramic depth estimation method combining spherical correction and semi-supervised learning, characterized in that: Includes the following steps: Step 1: Build a teacher model and perform supervised fine-tuning; Step 2: Generate pseudo-depth labels using the teacher model; Step 3: Build the student model; Step 4: Adaptively correct panoramic image features using a spherical correction mechanism; Step 5: During training, freeze the perspective base model and only optimize and update the PanoAdapter module parameters and decoder parameters of the student model; Step 6: Input the panoramic image to be tested into the trained student model. The encoder extracts features, the PanoAdapter module performs spherical correction, and the decoder generates depth prediction, finally outputting a high-precision panoramic depth map.

2. The panoramic depth estimation method combining spherical correction and semi-supervised learning according to claim 1, characterized in that: Step 1 is as follows: A pre-trained perspective depth baseline model was selected as the backbone network of the teacher model. To adapt the teacher model to the panoramic depth estimation task, a low-rank adaptive technique was used on a labeled panoramic dataset. The teacher model underwent its first phase of fine-tuning; during this fine-tuning process, each labeled panoramic image was... Input the teacher model to obtain the predicted depth map. and compare it with the corresponding true depth map. Compare and calculate the supervision loss. This optimizes the teacher model parameters, enabling it to perform preliminary depth estimation for panoramic scenes.

3. The panoramic depth estimation method combining spherical correction and semi-supervised learning according to claim 2, characterized in that: Step 2 is as follows: Using the teacher model trained in step 1, the unlabeled panopticon dataset was processed. Each unlabeled panoramic image in Perform forward inference to obtain the corresponding predicted depth map. Using semantic segmentation models to Perform semantic parsing to generate a sky region mask. This mask is used to filter and zero out the depth values ​​of the sky region in the predicted depth map, ultimately generating high-quality pseudo-depth labels. .

4. The panoramic depth estimation method combining spherical correction and semi-supervised learning according to claim 3, characterized in that: Step 3 is as follows: The backbone network structure of the student model is consistent with that of the teacher model, and is also based on the encoder-decoder architecture of the perspective depth basic model; on this basis, the PanoAdapter module proposed in this invention is inserted between each Transformer layer of the encoder; the trainable parameters of the student model only include the parameters of the PanoAdapter module and the parameters of the decoder.

5. The panoramic depth estimation method combining spherical correction and semi-supervised learning according to claim 4, characterized in that: Step 4 is as follows: For the input panoramic image After being extracted by the encoder, it is in the first... The feature map output by the Transformer block is ,in These represent the height and width of the feature map, respectively. For feature channel dimensions; First, the pixel coordinates of the feature map Mapped to azimuth in spherical coordinate system and polar angle To describe the cyclic continuity of the panoramic image in the horizontal direction, the origin of the coordinate system is defined at the center of the image, and its mapping relationship is shown in equation (1): (1)； Subsequently, to map the spherical coordinates to a high-dimensional feature space ,in To embed the dimension, the azimuth angle and polar angle Encoding is performed using Fourier position coding to obtain the spherical embedding. : (2)； Among them, the encoding function It consists of sine and cosine components of different frequencies; for the input variable Its form is shown in equation (3): (3); in, It is a frequency vector, and T denotes matrix transpose. , No. Each frequency component is defined as a geometric series as shown in equation (4): (4)； Generated spherical embedding This will be used as a conditional embedding of the input feature linear modulation module to achieve spherical distortion correction of the features; specifically, for the ... Output features of layer Transformer blocks Using a lightweight MLP network to embed from a sphere Dynamic Prediction Modulation Parameters and As shown in equation (5): (5)； in The scaling factor. The translation coefficient; Subsequently, the modulation parameters were used to adjust the first... Original features of the layer Perform position-by-position affine transformation to obtain the corrected features. The calculation method is shown in equation (6): (6)； in, This represents element-wise multiplication. .

6. The panoramic depth estimation method combining spherical correction and semi-supervised learning according to claim 5, characterized in that: Step 5 is as follows: The training data consists of a combination of labeled data and pseudo-labeled data. Let the images in the joint training set be denoted as . For each one Apply color enhancement transformations respectively and rotation transformation The corresponding data-enhanced image is obtained. and ;Will Input each student model separately to obtain their respective predicted depth maps. ; Total loss function It consists of a weighted average of the semi-supervised learning loss and the consistency loss, as shown in equation (7): (7)； Here, For semi-supervised learning loss, This results in a loss of consistency. The semi-supervised learning loss is shown in equation (8): (8)； in, For supervised loss of labeled data, For pseudo-depth labels generated in step 2 Calculated pseudo-label distillation loss; and These are the weight coefficients corresponding to the labeled loss and pseudo-label loss, respectively, used to balance the contribution of the two types of data to the model training; their calculation methods are the same, as shown in equation (9): (9)； In the formula, This is a scale-translation-invariant loss used to eliminate global scale and translation bias between the predicted depth and the true depth; Gradient matching loss is used to constrain the local structural consistency of the depth map; The consistency loss is shown in equation (10): (10)； Here, For color consistency loss, constrain the consistency of depth prediction between the original image and the color-enhanced image; The geometric isovariability loss constrains the consistency of depth prediction between the original image and the rotated transformed image; and These are the corresponding weighting coefficients; specifically, their calculation methods are shown in equations (11) and (12): (11)； (12)； in, Original image Predicted depth Enhance the color of the image Predicted depth For rotation transformation image Equation (11) requires that after applying the same color enhancement transformation to the original predicted depth, the result is consistent with the predicted depth of the color-enhanced image; Equation (12) requires that after applying the same rotation transformation to the original predicted depth, the result is consistent with the predicted depth of the rotated image.

Images