A panoramic depth estimation method integrated with geometry and semantic optimization and related devices

Abstract

The application discloses a panoramic depth estimation method integrated with geometry and semantic optimization and a related device. In the application, ICOSAP features are introduced as geometry complementary information, and dense semantic information provided by an EPR mode and non-distortion geometry information provided by an ICOSAP mode are combined. Semantic refinement processing is performed on the EPR mode and the ICOSAP mode to align the features of the multiple modes, so that multi-scale mode semantic refinement is realized. Cross-mode difference perception is performed on the EPR mode and the ICOSAP mode, so that the geometry relationship between the EPR mode and the ICOSAP mode is indirectly perceived through an agent as a bridge. The obtained geometry compensation features can better retain the geometry structure information in a scene, and further improve the panoramic depth estimation precision.

Description

Technical Field

[0001] This invention relates to the field of panoramic depth estimation technology, and in particular to a panoramic depth estimation method and related apparatus that integrates geometric and semantic optimization. Background Technology

[0002] Depth estimation technology aims to infer the distance (i.e., depth) from points in a scene to the camera from a single image. This technology is widely used in fields such as autonomous driving, robot navigation, and 3D reconstruction. For example, in autonomous driving systems, accurate real-time depth perception is crucial for environmental understanding, providing key spatial information for path planning and obstacle avoidance. In virtual reality applications, monocular depth reconstruction can significantly reduce 3D modeling costs and enhance the immersive experience.

[0003] However, traditional perspective cameras, limited by their narrow field of view (FOV), often fail to capture complete scene information. Occlusion outside the field of view and blurred local cues severely compromise the reliability of depth estimation in complex, large-scale environments. The widespread adoption of consumer-grade panoramic cameras offers new possibilities for overcoming these limitations. Panoramic imaging systems eliminate the blind spots of traditional FOVs and establish comprehensive contextual relationships by capturing 360° environmental information. This global perception capability not only effectively alleviates the inherent occlusion problem of perspective cameras but also provides richer geometric cues for depth inference. Spherical panoramic images with 360° attributes can be represented in various ways, with equirectangular projection (ERP) images being the mainstream storage format.

[0004] Perspective projection mapping (ERP) maps spherical pixels onto a rectangular plane by uniformly sampling the latitude and longitude of a sphere. However, this uniform sampling results in uneven pixel distribution along the latitudinal direction, causing severe geometric distortion in ERP images, especially in extreme regions such as ceilings and floors where they are severely stretched. Applying a deep neural network trained for perspective views directly to ERP images inevitably leads to suboptimal results. Most existing methods focus on fusing other projection images with less geometric distortion, such as cube projection, tangent projection, and icosahedral sampling projection (ICOSAP). However, existing depth estimation methods suffer from low accuracy in panoramic depth estimation (PDE). Summary of the Invention

[0005] This invention provides a panoramic depth estimation method and related apparatus that integrates geometric and semantic optimization, in order to solve the technical problem of low panoramic depth estimation accuracy in existing depth estimation methods.

[0006] This invention provides a panoramic depth estimation method integrating geometric and semantic optimization, the method comprising:

[0007] Acquire EPR and ICOSAP images of the target scene, and extract features from the EPR and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features;

[0008] The second-level scale EPR feature set in the multi-scale EPR features is subjected to cross-scale feature fusion processing to obtain multi-scale EPR fusion features; semantic refinement processing is performed on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features.

[0009] Cross-modal difference sensing processing is performed using the small-scale EPR features in the multi-scale EPR features and the small-scale ICOSAP point features in the multi-scale ICOSAP point features to obtain geometric compensation features.

[0010] The small-scale refinement features and the geometric compensation features in the semantic refinement features are dynamically aggregated to obtain the aggregated features;

[0011] A hierarchical decoding operation is performed on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature to generate a target panoramic depth map.

[0012] Optionally, the step of acquiring the EPR image and ICOSAP image of the target scene, and extracting features from the EPR image and the ICOSAP image respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features includes:

[0013] Acquire EPR and ICOSAP images of the target scene;

[0014] Multi-scale EPR features are obtained by extracting features from the EPR image through the backbone network.

[0015] The ICOSAP image is subjected to feature extraction using a point encoder to obtain multi-scale ICOSAP point features.

[0016] Optionally, the step of performing cross-modal difference sensing processing using the small-scale EPR features from the multi-scale EPR features and the small-scale ICOSAP point features from the multi-scale ICOSAP point features to obtain geometric compensation features includes:

[0017] A first key-value pair is generated based on the small-scale ICOSAP point features of the multi-scale ICOSAP point features; a second key-value pair is generated based on the small-scale EPR features in the multi-scale EPR features.

[0018] Initialize the agent and perform linear projection to generate an initial query; perform dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query; perform encoding optimization based on the optimized query to obtain an encoded query; use the encoded query as a new initial query and jump to execute the step of performing dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query, until a preset iteration condition is met to generate the target query;

[0019] The geometric compensation features are obtained by iteratively optimizing the decoding based on the target query and the second key-value pair using a difference decoder.

[0020] Optionally, the two-path interactive computation process in the t-th iteration is represented as follows:

[0021]

[0022] In the formula: This represents the semantic difference information between the ICOSAP modality and the agent in the t-th iteration. This represents the semantic difference information between the EPR modality and the agent in the t-th iteration; This represents the initial query vector in the t-th iteration; and These are the key vector and value vector in the first key-value pair, respectively. and These are the key vector and value vector in the second key-value pair, respectively. This represents the ICOSAP modal distance-aware attention weight in the t-th iteration. This represents the EPR modal distance-aware attention weight in the t-th iteration; This indicates the relative spatial location information between the agent and the ICOSAP modality. This indicates the relative spatial location information between the agent and the EPR modality; This represents the optimized query vector calculated in the t-th iteration; This indicates a normalization operation. This represents the convolution operation. This indicates a splicing operation.

[0023] Optionally, the step of performing semantic refinement processing based on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, and outputting semantically refined features, includes:

[0024] The coordinate coordinates of each point in the second-level scale ICOSAP point feature set of the multi-scale ICOSAP point features are projected to generate multiple initial sampling networks.

[0025] Based on the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, multiple dynamic offsets are predicted;

[0026] The initial sampling network is adjusted according to the dynamic offset to generate multiple target sampling networks;

[0027] Based on bilinear interpolation, the scale-correlated EPR fusion features of the small-scale EPR fusion feature set in the multi-scale EPR fusion features are sampled according to the target sampling network to obtain multiple geometrically aligned features.

[0028] The geometric alignment features and scale-correlated ICOSAP point features are subjected to dual cross-attention processing to obtain cross-modal semantic features;

[0029] Deformable attention processing is performed on the cross-modal semantic features to obtain semantic refinement features.

[0030] Optionally, the step of performing hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregated feature to generate a target panoramic depth map includes:

[0031] The aggregated features after upsampling are concatenated with the large-scale refined features in the semantic refined features by channel dimension to obtain the first-level concatenated features.

[0032] The first-level stitched feature, after upsampling, is stitched together with the large-scale EPR fusion feature in the multi-scale EPR fusion feature by channel dimension to obtain the second-level stitched feature.

[0033] The second-level spliced ​​features, after upsampling, are spliced ​​with the small-scale EPR features of the first-level EPR feature set in the multi-scale EPR features by channel dimension to obtain the third-level spliced ​​features.

[0034] The upsampled third-level stitched features are stitched together with the large-scale EPR features of the first-level scale EPR feature set in the multi-scale EPR features by channel dimension to obtain the target stitched features.

[0035] A panoramic depth map of the target is generated based on the target stitching features.

[0036] The present invention also provides a panoramic depth estimation system integrating geometric and semantic optimization, the system comprising:

[0037] The image feature acquisition unit is used to acquire EPR images and ICOSAP images of the target scene, and to extract features from the EPR images and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features.

[0038] The semantic refinement feature processing unit is used to perform cross-scale feature fusion processing on the second-level scale EPR feature set in the multi-scale EPR features to obtain multi-scale EPR fusion features; and to perform semantic refinement processing on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features.

[0039] The geometric compensation feature sensing unit is used to perform cross-modal difference sensing processing using the small-scale EPR features in the multi-scale EPR features and the small-scale ICOSAP point features in the multi-scale ICOSAP point features to obtain geometric compensation features.

[0040] The aggregation feature unit is used to dynamically aggregate the small-scale refinement features and the geometric compensation features in the semantic refinement features to obtain aggregated features;

[0041] The target panoramic depth map generation unit is used to perform hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature to generate a target panoramic depth map.

[0042] The present invention also provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the panoramic depth estimation methods described above.

[0043] The present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the panoramic depth estimation method as described above.

[0044] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the panoramic depth estimation methods described above.

[0045] As can be seen from the above technical solutions, the present invention has the following advantages:

[0046] This invention provides a panoramic depth estimation method and related apparatus integrating geometric and semantic optimization. The method includes: acquiring EPR images and ICOSAP images of a target scene, and extracting features from the EPR images and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features; performing cross-scale feature fusion processing on the second-level scale EPR feature set in the multi-scale EPR features to obtain multi-scale EPR fusion features; and performing cross-scale feature fusion processing on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point features in the multi-scale ICOSAP point features. The system performs semantic refinement processing on the set of features to output semantic refinement features; it then performs cross-modal difference perception processing on the small-scale EPR features and the small-scale ICOSAP point features of the multi-scale EPR features to obtain geometric compensation features; it dynamically aggregates the small-scale refinement features and the geometric compensation features to obtain aggregated features; and it performs hierarchical decoding operations on the first-level scale EPR feature set of the multi-scale EPR features, the large-scale EPR fusion features of the multi-scale EPR fusion features, the large-scale refinement features of the semantic refinement features, and the aggregated features to generate a target panoramic depth map.

[0047] In this invention, ICOSAP features are introduced as geometric supplementary information, combined with the dense semantic information provided by the EPR modality and the distortion-free geometric information provided by the ICOSAP modality. By performing semantic refinement processing on the EPR and ICOSAP modalities to align the features of the multimodalities, multi-scale modal semantic refinement is achieved. Cross-modal difference perception is performed on the EPR and ICOSAP modalities, so that the geometric relationship between the EPR and ICOSAP modalities is indirectly perceived through a proxy as a bridge. The resulting geometric compensation features can better preserve the geometric structure information of the scene, further improving the accuracy of panoramic depth estimation, thereby solving the technical problem of low panoramic depth estimation in existing depth estimation methods. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the steps of a panoramic depth estimation method integrating geometric and semantic optimization, provided in an embodiment of the present invention;

[0050] Figure 2 A flowchart illustrating the implementation of a panoramic depth estimation method integrating geometric and semantic optimization, provided in an embodiment of the present invention.

[0051] Figure 3 A flowchart of semantic refinement processing based on the CGRS module provided in this embodiment of the invention;

[0052] Figure 4 A flowchart of cross-modal difference sensing processing based on the DAPQ module provided in this embodiment of the invention;

[0053] Figure 5 A structural block diagram of a panoramic depth estimation system integrating geometric and semantic optimization provided in an embodiment of the present invention;

[0054] Figure 6 This is a flowchart illustrating the implementation of the panoramic depth estimation method based on ICOSAP projection fusion provided in this embodiment of the invention, combined with existing technologies. Detailed Implementation

[0055] This invention provides a panoramic depth estimation method and related apparatus that integrates geometric and semantic optimization, in order to solve the technical problem of low panoramic depth estimation accuracy in existing depth estimation methods.

[0056] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0057] Please see Figure 6 The following section provides further explanation of the visual performance of depth estimation methods that can be implemented using existing technologies.

[0058] Panoramic Depth Estimation Method Based on ICOSAP Projection Fusion

[0059] Please see Figure 6 First, an ICOSAP sphere with the most initial faces is introduced to represent the panoramic image, providing comprehensive and low-quality global perception. Then, to reduce computational cost, the ICOSAP sphere is discretized into a point set. On the one hand, the ERP image provides dense pixel values, while the discrete ICOSAP point set can prevent semantic redundancy while preserving spatial information and global perception. To better utilize these two projections, a dual-projection feature extraction branch, a dual-projection dual-attention fusion module (B2F), and a decoder are designed. Specifically, a pre-trained ResNet series network is used to extract features from the ERP image. A point encoder from the point cloud is introduced to extract point features from the discrete ICOSAP point set. To utilize the complementary information of these two modalities, a B2F module consisting of a distance-aware affinity module (DA) and a semantic-aware affinity module (SA) is proposed. Specifically, DA models distance information by performing subtraction operations between different modalities, while SA performs cross-fusion based on the semantic similarity of different modalities. Finally, the final output is obtained through its designed gated fusion module.

[0060] The panoramic depth estimation method based on ICOSAP projection fusion uses discrete point sets to represent ICOSAP projections instead of cutting the image into small pieces. This representation effectively preserves the original distortion-free spatial geometry. This not only maintains distortion-free geometric information but also makes it possible to achieve efficient and accurate panoramic depth estimation. At the same time, the method also introduces a dual attention fusion module, which aims to fuse ERP features and ICOSAP point features. Although this design effectively reduces the number of model parameters and computational complexity, its performance improvement is still limited by the fusion module itself: (1) Risk of information loss: Direct subtraction between the two modal features may lead to information loss, thus restricting further improvement of model accuracy. (2) Coarse feature alignment: Feature alignment strategies based entirely on semantic similarity can only achieve coarse alignment and are difficult to capture fine structures, ultimately resulting in blurred details and textures in the depth map.

[0061] This invention addresses the problem of poor panoramic depth estimation accuracy in existing depth estimation methods. It proposes a panoramic depth estimation method that integrates geometric and semantic optimizations to improve the accuracy of traditional depth estimation methods while maintaining low computational overhead.

[0062] Please see Figure 1 This invention provides a panoramic depth estimation method integrating geometric and semantic optimization, the method comprising:

[0063] Step 101: Obtain the EPR image and ICOSAP image of the target scene, and extract features from the EPR image and ICOSAP image respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features.

[0064] It is understood that this invention introduces ICOSAP features as geometric supplementary information, combining the dense semantic information provided by the EPR modality with the distortion-free geometric information provided by the ICOSAP modality; in order to ensure the integrity of feature information extraction for each modality, this embodiment adopts a dual-branch extraction structure to extract features from the EPR image and the ICOSAP image respectively.

[0065] This step specifically includes the following steps S11~S13:

[0066] S11. Obtain the EPR and ICOSAP images of the target scene.

[0067] In practical applications, a panoramic camera can be used to capture the target scene and generate an EPR image; then the EPR image can be converted into an ICOSAP image according to the ICOSAP projection format.

[0068] S12. Feature extraction is performed on the EPR image through the backbone network to obtain multi-scale EPR features.

[0069] It should be noted that the backbone network can be a pre-trained convolutional neural network, such as ResNet-34; the backbone network is used to extract features from the input ERP image to obtain multi-scale ERP features. .

[0070] S13. Extract features from the ICOSAP image using a point encoder to obtain multi-scale ICOSAP point features.

[0071] It should be noted that a point encoder can be used to extract features from ICOSAP images to obtain multi-scale ICOSAP point features. .

[0072] Step 102: Perform cross-scale feature fusion processing on the second-level scale EPR feature set in the multi-scale EPR features to obtain multi-scale EPR fusion features; perform semantic refinement processing on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features.

[0073] It should be noted that fine-grained semantic information can be achieved by accurately aligning heterogeneous data modalities. However, due to projection distortion and differences in representativeness between modalities, traditional methods often encounter the problem of feature misalignment. To address this issue, this invention proposes a Deformable Cross-Modal Refinement Transformer (DCRT) structure. This structure achieves multi-scale refinement through distortion-aware feature encoding and coordinate-guided semantic enhancement, thereby enabling more accurate feature alignment.

[0074] Specifically, the DCRT structure provided in this embodiment includes a Multi-Scale Deformable Encoder (MSDE) and a Coordinate-Guided Semantic Refiner (CGSR). MSDE aims to extract global and local contextual information from multi-scale image features while mitigating the impact of projection distortion on cross-modal alignment. This module operates on multi-scale features derived from the backbone network. For each scale, MSDE adaptively adjusts the receptive field through deformable attention, dynamically focusing on regions with similar semantic information. The features generated by MSDE share the same resolution as the features of the original input. MSDE captures cross-scale correlations from global semantics to local details, providing discriminative visual primitives for subsequent refinement stages.

[0075] In the DCRT structure, MSDE uses the second-level scale EPR feature set from the multi-scale EPR features generated by the backbone network. As input, multi-scale EPR fusion features are obtained. It is worth noting that this process does not change the resolution of the features, i.e. and They are of equal size. CGSR, on the other hand, focuses on multi-scale EPR fusion features. Small-scale EPR fusion feature set And the second-level scale ICOSAP point feature set in multi-scale ICOSAP point features To further refine the semantics.

[0076] Specifically, based on multi-scale EPR fusion features Small-scale EPR fusion feature set Second-level ICOSAP point feature set in multi-scale ICOSAP point features The steps for semantic refinement include steps S21 to S26:

[0077] S21. Project the coordinates of each point in the second-level scale ICOSAP point feature set of the multi-scale ICOSAP point feature set to generate multiple initial sampling networks.

[0078] S22. Based on the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, multiple dynamic migrations are predicted.

[0079] S23. Adjust the corresponding initial sampling network according to the dynamic offset to generate multiple target sampling networks.

[0080] S24. Based on bilinear interpolation, the scale-correlated EPR fusion features of the small-scale EPR fusion feature set in the multi-scale EPR fusion features are sampled according to the target sampling network to obtain multiple geometrically aligned features.

[0081] S25. Perform dual cross-attention processing on geometric alignment features and scale-correlated ICOSAP point features to obtain cross-modal semantic features.

[0082] S26. Obtain semantic refinement features by performing deformable attention processing on cross-modal semantic features.

[0083] Please see Figure 3 For ease of explanation, single-scale ICOSAP point features are used. For example; first, will Corresponding point coordinates Perform coordinate projection, that is, perform point coordinate projection. Mapping onto the pixel plane of the ERP image generates an initial sampling mesh that reflects the ideal point cloud projection on the ERP image. .

[0084] To address the offset issue between different projections, based on ICOSAP point features... Perform linear projection to encode the local geometric context and predict the dynamic offset. Then, through dynamic offset To adjust the sampling grid, i.e.:

[0085]

[0086] in, The target sampling network is used; the above adjustment operations are intended to correct spatial misalignment caused by projection distortion.

[0087] Next, based on bilinear interpolation... The specified pixel position is from Sampling is performed to obtain geometric alignment features. Subsequently, a dual cross-attention mechanism is used to couple the original point features and ERP features through semantic similarity. Finally, in the semantic enhancement stage, deformable self-attention is adopted instead of standard multi-head self-attention to balance computational efficiency and geometric sensitivity. This design achieves a strong trade-off between efficiency and accuracy, resulting in semantically refined features that couple geometric guidance and semantic similarity. .

[0088] More specifically, in combination Figure 3 In the first layer of dual cross-attention mechanism, using ICOSAP point features For the query, both the key and value are geometrically aligned features. In the second-level dual-cross-attention mechanism, the cross-attention output of the first-level dual-cross-attention mechanism is used as the key and value, and the EPR fusion feature is used. For querying; at the same time, after each attention cross-processing, the data also needs to be processed by layer normalization (Add & Norm) and forward propagation network (FFN) to stabilize the transformation process of features.

[0089] In this invention, the DCRT module can perform feature alignment between different modalities and achieve multi-scale semantic refinement through a coordinate-guided strategy; the CGSR module applies the coordinate-guided strategy to features at multiple scales, achieving semantic alignment and refinement through explicit coordinate guidance and multi-stage attention interaction. CGSR converts the multi-scale features output by MSDE... and ICOSAP point features Perform alignment and fusion to generate aligned semantic refinement features. This enables more precise semantic alignment.

[0090] Step 103: Use the small-scale EPR features from the multi-scale EPR features and the small-scale ICOSAP point features from the multi-scale ICOSAP point features to perform cross-modal difference sensing processing to obtain geometric compensation features.

[0091] This invention provides a Difference-Aware Proxy Query (DAPQ) module. The DAPQ module introduces a learnable proxy as a cross-modal interaction mediator, enabling the geometric relationship between ERP and ICOSAP point features to be indirectly perceived through the proxy as a bridge. The proxy is set as a reference coordinate distance instead of the absolute coordinate distance between the two in the original method, thereby better preserving the geometric structure information of the scene. At the same time, a cascaded encoder-decoder stacked architecture is used to replace channel summation, achieving smoother multimodal fusion.

[0092] Please see Figure 4The differential encoder generates optimized cross-modal tags through a two-path interaction mechanism with the agent and improves them through M iterations. The differential decoder uses these tags as query values ​​and models them with the original ERP features through L feature interactions. In this way, semantic differences between modalities can be modeled through the agent, and spatial coordinate information is used as positional encoding, gradually fusing geometric priors and visual semantics, thereby preserving the geometric integrity of the original modality.

[0093] This step specifically includes steps S31 to S33:

[0094] S31. Generate the first key-value pair based on the small-scale ICOSAP point features of the multi-scale ICOSAP point features; generate the second key-value pair based on the small-scale EPR features in the multi-scale EPR features.

[0095] S32. Initialize the agent and perform linear projection to generate an initial query; perform two-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query; perform encoding optimization based on the optimized query to obtain an encoded query; use the encoded query as a new initial query, and jump to execute the steps of performing two-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query, until the preset iteration conditions are met, generating the target query. Here, the preset iteration conditions refer to the set number of iterations.

[0096] S33. The geometric compensation features are obtained by iteratively optimizing the decoding based on the target query and the second key-value pair using the difference decoder.

[0097] Specifically, firstly, based on multi-scale ICOSAP point features Small-scale ICOSAP point features and multi-scale EPR features Small-scale EPR features Generate two pairs of key-value pairs respectively. and , and Then, the randomly initialized agent P generates an initial query through linear projection. The two key-value pairs and the initial query interact through a designed two-path process. Therefore, the calculation process for this two-path interaction in the t-th iteration is represented as follows:

[0098]

[0099]

[0100]

[0101] In the formula: This represents the semantic difference information between the ICOSAP modality and the agent in the t-th iteration. This represents the semantic difference information between the EPR modality and the agent in the t-th iteration; This represents the initial query vector in the t-th iteration; and These are the key vector and value vector in the first key-value pair, respectively. and These are the key vector and value vector in the second key-value pair, respectively. This represents the ICOSAP modal distance-aware attention weight in the t-th iteration. This represents the EPR modal distance-aware attention weight in the t-th iteration; This indicates the relative spatial location information between the agent and the ICOSAP modality. This indicates the relative spatial location information between the agent and the EPR modality; This represents the optimized query vector calculated in the t-th iteration; This indicates a normalization operation. This represents the convolution operation. This represents a concatenation operation. Here, t is a positive integer.

[0102] In this embodiment, the relative spatial position information between the agent P and the ICOSAP mode can be calculated by projection subtraction. The relative spatial position information between agent P and EPR modes is obtained by broadcast subtraction calculation. Generate distance-aware attention weights using semantic difference information and spatial location information. and After employing a heterogeneous interaction mechanism, connections are made at the channel dimension to obtain cross-modal tags with dual-path optimization, thus optimizing the query. .

[0103] Then, multi-head self-attention is used to model global dependencies within cross-modal labels, and this is enhanced by a standard forward propagation network (FFN). This process can be represented as:

[0104]

[0105]

[0106]

[0107] In the formula: This represents the first-level attention query vector in the t-th iteration. This represents the second-level attention query vector in the t-th iteration. This represents the encoded query vector in the t-th iteration; This indicates multi-head self-attention processing. Presentation layer normalization processing, This indicates the enhanced processing of the standard forward propagation network.

[0108] Therefore, after M iterations of optimization, the optimized query from the Mth iteration is used as the target query in the encoding stage to achieve the geometrically optimized cross-modal representation. In this embodiment, the number of iterations M is set to 3 based on experimental experience.

[0109] During the decoding phase, the difference decoder uses the target query from the difference encoder as the query, iteratively combining them with the features from the original ERP across L phases. The exported key-value pairs are interacted. This operation further leverages the semantic information dense in the ERP features, ensuring geometric integrity while achieving semantic compensation. Each interaction layer applies an FFN-based nonlinear transformation to enhance discriminative features, accompanied by residual connections and layer normalization for training stability. After L iterations, the final geometrically compensated features are generated. .

[0110] In this step, the DAPQ module uses the difference perception principle of dual-path proxy to model the difference relationship between different modalities. It can dynamically adapt to changes in the geometric environment and maintain geometric integrity, better preserve the geometric information of the panoramic image, and thus better reconstruct the geometric structure of the final depth map.

[0111] Step 104: Dynamically aggregate the small-scale refinement features and geometric compensation features in the semantic refinement features to obtain the aggregated features.

[0112] In this step, a gating mechanism is used to refine small-scale features that incorporate rich geometric information. and geometric compensation features after semantic refinement Perform dynamic aggregation to obtain aggregation features .

[0113] Step 105: Perform hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature of the multi-scale EPR feature to generate a target panoramic depth map.

[0114] Understandably, this step uses a hierarchical decoder to process these features. Perform hierarchical decoding to aggregate features After the upsampling module and Connect along the channel dimension, repeating this process until you connect to the large-scale EPR features in the first-level scale EPR feature set. Then, after upsampling, a depth map with the same resolution as the input is finally generated.

[0115] Specifically, this process includes steps S41 to S44:

[0116] S41. The aggregated features after upsampling processing Large-scale refinement features in semantic refinement Channel-dimensional splicing is performed to obtain the first-level splicing feature.

[0117] S42. Combine the upsampled first-level stitched features with the large-scale EPR fusion features from the multi-scale EPR fusion features. Channel-dimensional concatenation is performed to obtain secondary concatenated features. The semantic information of the EPR modality is incorporated to supplement the shortcomings of the geometric-semantic fusion features of the primary concatenated features.

[0118] S43. The upsampled secondary concatenated features are concatenated with the small-scale EPR features from the primary-scale EPR feature set in the multi-scale EPR feature set along the channel dimension to obtain the tertiary concatenated features. Incorporating local detail information can improve the accuracy of the features.

[0119] S44. The upsampled three-level stitched features are stitched together with the large-scale EPR features of the first-level EPR feature set in the multi-scale EPR feature set by channel dimension to obtain the target stitched features.

[0120] This embodiment completes full-scale feature fusion from high to low scale, ensuring that the resolution of the depth map matches the input.

[0121] S45. Generate a panoramic depth map of the target based on the target stitching features.

[0122] Understandably, the target stitching features are fused with multi-scale features, transforming them into a panoramic depth map of the target, thus meeting the requirements for panoramic depth estimation. During the testing phase, this invention demonstrated significant advantages on multiple datasets and exhibited strong adaptability in cross-dataset generalization tests.

[0123] This invention provides a panoramic depth estimation method that integrates geometric and semantic optimization. By combining dense semantic information provided by ERP and distortion-free geometric information provided by ICOSAP, the accuracy of panoramic depth estimation is further improved, and it has the following advantages:

[0124] 1. Existing panoramic depth estimation methods based on ICOSAP projection fusion fuse multimodal features by directly subtracting ICOSAP point features from ERP features to capture semantic differences, and then perform channel-wise summation for aggregation. However, this subtraction may lead to the loss of geometric information, resulting in the loss of large areas of object geometric structure. To address this, the DAPQ module provided in this invention utilizes a learnable surrogate as an intermediary to provide reference coordinates. Through this indirect difference modeling, the geometric information of the panoramic image can be better preserved.

[0125] 2. The panoramic depth estimation method based on ICOSAP projection fusion achieves feature alignment based on semantic similarity. While this method captures the semantic correlation between different features, it ignores spatial misalignment caused by geometric differences, and it is a coarse alignment, limiting the alignment accuracy across modalities. This invention, however, uses the CGSR module to guide feature alignment with explicit spatial coordinates and corrects spatial misalignment by superimposing offsets, thereby achieving more accurate semantic alignment.

[0126] Please see Figure 5 The present invention also provides a panoramic depth estimation system integrating geometric and semantic optimization, the system comprising:

[0127] The image feature acquisition unit 201 is used to acquire the EPR image and ICOSAP image of the target scene, and to extract features from the EPR image and ICOSAP image respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features.

[0128] The semantic refinement feature processing unit 202 is used to perform cross-scale feature fusion processing on the second-level scale EPR feature set in the multi-scale EPR features to obtain multi-scale EPR fusion features; and to perform semantic refinement processing on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features.

[0129] The geometric compensation feature sensing unit 203 is used to perform cross-modal difference sensing processing using small-scale EPR features in multi-scale EPR features and small-scale ICOSAP point features in multi-scale ICOSAP point features to obtain geometric compensation features.

[0130] Aggregation feature aggregation unit 204 is used to dynamically aggregate small-scale refinement features and geometric compensation features in semantic refinement features to obtain aggregated features;

[0131] The target panoramic depth map generation unit 205 is used to perform hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature in the multi-scale EPR feature set to generate the target panoramic depth map.

[0132] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the panoramic depth estimation methods described above.

[0133] The present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of any of the panoramic depth estimation methods described above.

[0134] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the panoramic depth estimation methods described above.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panoramic depth estimation method integrating geometric and semantic optimization, characterized in that, The method includes: Acquire EPR and ICOSAP images of the target scene, and extract features from the EPR and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features; The second-level scale EPR feature set in the multi-scale EPR features is subjected to cross-scale feature fusion processing to obtain multi-scale EPR fusion features; semantic refinement processing is performed on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features. Cross-modal difference sensing processing is performed using the small-scale EPR features in the multi-scale EPR features and the small-scale ICOSAP point features in the multi-scale ICOSAP point features to obtain geometric compensation features. The small-scale refinement features and the geometric compensation features in the semantic refinement features are dynamically aggregated to obtain the aggregated features; A hierarchical decoding operation is performed on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature to generate a target panoramic depth map. The step of performing semantic refinement processing based on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, and outputting semantically refined features, includes: The coordinate coordinates of each point in the second-level scale ICOSAP point feature set of the multi-scale ICOSAP point features are projected to generate multiple initial sampling networks. Based on the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, multiple dynamic offsets are predicted; The initial sampling network is adjusted according to the dynamic offset to generate multiple target sampling networks; Based on bilinear interpolation, the scale-correlated EPR fusion features of the small-scale EPR fusion feature set in the multi-scale EPR fusion features are sampled according to the target sampling network to obtain multiple geometrically aligned features. The geometric alignment features and scale-correlated ICOSAP point features are subjected to dual cross-attention processing to obtain cross-modal semantic features; Deformable attention processing is performed on the cross-modal semantic features to obtain semantic refinement features; The step of performing cross-modal difference sensing processing using the small-scale EPR features from the multi-scale EPR features and the small-scale ICOSAP point features from the multi-scale ICOSAP point features to obtain geometric compensation features includes: A first key-value pair is generated based on the small-scale ICOSAP point features of the multi-scale ICOSAP point features; a second key-value pair is generated based on the small-scale EPR features in the multi-scale EPR features. Initialize the agent and perform linear projection to generate an initial query; perform dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query; perform encoding optimization based on the optimized query to obtain an encoded query; use the encoded query as a new initial query and jump to execute the step of performing dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query, until a preset iteration condition is met to generate the target query; The geometric compensation features are obtained by iteratively optimizing the decoding based on the target query and the second key-value pair using a difference decoder. The step of performing hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature to generate a target panoramic depth map includes: The aggregated features after upsampling are concatenated with the large-scale refined features in the semantic refined features by channel dimension to obtain the first-level concatenated features. The first-level stitched feature, after upsampling, is stitched together with the large-scale EPR fusion feature in the multi-scale EPR fusion feature by channel dimension to obtain the second-level stitched feature. The second-level spliced ​​features, after upsampling, are spliced ​​with the small-scale EPR features of the first-level EPR feature set in the multi-scale EPR features by channel dimension to obtain the third-level spliced ​​features. The upsampled third-level stitched features are stitched together with the large-scale EPR features of the first-level scale EPR feature set in the multi-scale EPR features by channel dimension to obtain the target stitched features. A panoramic depth map of the target is generated based on the target stitching features.

2. The panoramic depth estimation method according to claim 1, characterized in that, The steps of acquiring EPR and ICOSAP images of the target scene, and extracting features from the EPR and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features include: Acquire EPR and ICOSAP images of the target scene; Multi-scale EPR features are obtained by extracting features from the EPR image through the backbone network. The ICOSAP image is subjected to feature extraction using a point encoder to obtain multi-scale ICOSAP point features.

3. The panoramic depth estimation method according to claim 1, characterized in that, The process of two-path interactive computation in the t-th iteration is represented as follows: In the formula: This represents the semantic difference information between the ICOSAP modality and the agent in the t-th iteration. This represents the semantic difference information between the EPR modality and the agent in the t-th iteration; This represents the initial query vector in the t-th iteration; and These are the key vector and value vector in the first key-value pair, respectively. and These are the key vector and value vector in the second key-value pair, respectively. This represents the ICOSAP modal distance-aware attention weight in the t-th iteration. This represents the EPR modal distance-aware attention weight in the t-th iteration; This indicates the relative spatial location information between the agent and the ICOSAP modality. This indicates the relative spatial location information between the agent and the EPR modality; This represents the optimized query vector calculated in the t-th iteration; This indicates a normalization operation. This represents the convolution operation. This indicates a splicing operation.

4. A panoramic depth estimation system integrating geometric and semantic optimization, characterized in that, The system includes: The image feature acquisition unit is used to acquire EPR images and ICOSAP images of the target scene, and to extract features from the EPR images and ICOSAP images respectively to obtain multi-scale EPR features and multi-scale ICOSAP point features. The semantic refinement feature processing unit is used to perform cross-scale feature fusion processing on the second-level scale EPR feature set in the multi-scale EPR features to obtain multi-scale EPR fusion features; and to perform semantic refinement processing on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features to output semantic refinement features. The geometric compensation feature sensing unit is used to perform cross-modal difference sensing processing using the small-scale EPR features in the multi-scale EPR features and the small-scale ICOSAP point features in the multi-scale ICOSAP point features to obtain geometric compensation features. The aggregation feature unit is used to dynamically aggregate the small-scale refinement features and the geometric compensation features in the semantic refinement features to obtain aggregated features; The target panoramic depth map generation unit is used to perform hierarchical decoding operations on the first-level scale EPR feature set, the large-scale EPR fusion feature of the multi-scale EPR fusion feature, the large-scale refinement feature of the semantic refinement feature, and the aggregation feature in the multi-scale EPR feature to generate a target panoramic depth map. The semantic refinement feature processing unit performs semantic refinement processing based on the small-scale EPR fusion feature set in the multi-scale EPR fusion features and the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, and outputs semantic refinement features, including: The coordinate coordinates of each point in the second-level scale ICOSAP point feature set of the multi-scale ICOSAP point features are projected to generate multiple initial sampling networks. Based on the second-level scale ICOSAP point feature set in the multi-scale ICOSAP point features, multiple dynamic offsets are predicted; The initial sampling network is adjusted according to the dynamic offset to generate multiple target sampling networks; Based on bilinear interpolation, the scale-correlated EPR fusion features of the small-scale EPR fusion feature set in the multi-scale EPR fusion features are sampled according to the target sampling network to obtain multiple geometrically aligned features. The geometric alignment features and scale-correlated ICOSAP point features are subjected to dual cross-attention processing to obtain cross-modal semantic features; The geometric compensation feature sensing unit is specifically used for... A first key-value pair is generated based on the small-scale ICOSAP point features of the multi-scale ICOSAP point features; a second key-value pair is generated based on the small-scale EPR features in the multi-scale EPR features. Initialize the agent and perform linear projection to generate an initial query; perform dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query; perform encoding optimization based on the optimized query to obtain an encoded query; use the encoded query as a new initial query and jump to execute the step of performing dual-path interactive calculations using the first key-value pair, the second key-value pair, and the initial query to obtain an optimized query, until a preset iteration condition is met to generate the target query; The geometric compensation features are obtained by iteratively optimizing the decoding based on the target query and the second key-value pair using a difference decoder. Deformable attention processing is performed on the cross-modal semantic features to obtain semantic refinement features; The target panoramic depth map generation unit is specifically used for, The aggregated features after upsampling are concatenated with the large-scale refined features in the semantic refined features by channel dimension to obtain the first-level concatenated features. The first-level stitched feature, after upsampling, is stitched together with the large-scale EPR fusion feature in the multi-scale EPR fusion feature by channel dimension to obtain the second-level stitched feature. The second-level spliced ​​features, after upsampling, are spliced ​​with the small-scale EPR features of the first-level EPR feature set in the multi-scale EPR features by channel dimension to obtain the third-level spliced ​​features. The upsampled third-level stitched features are stitched together with the large-scale EPR features of the first-level scale EPR feature set in the multi-scale EPR features by channel dimension to obtain the target stitched features. A panoramic depth map of the target is generated based on the target stitching features.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the panoramic depth estimation method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the panoramic depth estimation method as described in any one of claims 1-3.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the panoramic depth estimation method as described in any one of claims 1-3.

Images