A panoramic image dual perception enhancement method and device fusing deformable convolution and spherical attention

Abstract

The application provides a panoramic image dual perception enhancement method and device fusing deformable convolution and spherical attention. The method comprises: mapping an ERP image into a polyhedral mesh vertex, inputting image features of all mesh vertices into a U-shaped coding and decoding network to obtain an image perception result; the U-shaped coding and decoding network comprises a D-level encoder and a D-level decoder, each level of the encoder comprises M stacked dual perception enhancement modules and a down-sampling layer, each level of the decoder comprises M stacked dual perception enhancement modules and an up-sampling layer, the encoder and the decoder are connected through a jump connection to transmit features between corresponding levels, and the dual perception enhancement module comprises a deformable spherical convolution branch, a deformable spherical local self-attention branch and an adaptive gating fusion block. Based on the dual perception enhancement module, the application can realize deep collaboration of spherical discrete mesh geometry adaptive capability and global semantic modeling capability, and improve the perception performance of the ERP image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and apparatus for enhancing dual perception of panoramic images by fusing deformable convolution and spherical attention. Background Technology

[0002] Panoramic images based on equirectangular projection (ERP) have become the mainstream data format in fields such as virtual reality, autonomous driving, and robot navigation because they can completely record 360° field of view information. However, ERP projection suffers from severe geometric stretching and shape distortion in the polar regions, making it difficult for traditional two-dimensional convolutional neural networks (CNNs) to maintain geometric consistency when processing ERP images, which severely limits the accuracy of downstream perception tasks.

[0003] To overcome the inherent limitations of ERP projection, researchers have proposed a spherical discretization representation method based on icosahedral meshes. By recursively subdividing the sphere into approximately uniform triangular meshes, convolution operators can be directly defined on the spherical manifold, fundamentally eliminating polar distortion. The SphDeformConv (Spherical DeformableConvolution) scheme further introduces a deformable convolution mechanism, generating learnable offsets for each mesh vertex and dynamically adjusting the sampling position of the convolution kernel, enhancing the model's adaptability to local geometric deformations. However, pure convolution methods are limited by a fixed receptive field, making it difficult to capture long-range global contextual dependencies in images, thus limiting their performance in semantic understanding tasks.

[0004] Meanwhile, the Transformer architecture based on self-attention has achieved breakthroughs in vision tasks due to its global modeling capabilities. SphereUFormer was the first to introduce a U-shaped Transformer architecture into the spherical domain. By defining spherical local self-attention on an icosahedral mesh and introducing a relative position bias based on spherical coordinate differences, it achieved effective global modeling of panoramic images. However, pure Transformer methods lack the local inductive bias of convolution, are insufficiently sensitive to edge details and geometric deformations of small target regions, and the secondary computational complexity of self-attention limits its application on high-resolution feature maps.

[0005] To integrate the complementary advantages of convolution and attention, various hybrid architectures have been proposed. For example, CTRL-F uses a dual-branch structure to extract local features from CNNs and global features from Transformers in parallel, and integrates them through an adaptive knowledge fusion strategy; WiTUnet introduces windowed self-attention and local perception enhancement modules in its U-shaped encoder and decoder; and Translation unifies self-attention and convolution at the operator level, assigning an independent parameter matrix to each relative displacement. However, all of the above methods are designed for planar images, and no method has yet achieved a deep fusion of the geometric adaptability of deformable convolution and the global modeling capability of self-attention on spherical manifolds, nor is there a dedicated fusion mechanism for the topological characteristics of spherical meshes. Summary of the Invention

[0006] In view of this, this application proposes a method and apparatus for dual-perception enhancement of panoramic images that integrates deformable convolution and spherical attention.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] According to a first aspect of the embodiments of this specification, a method for dual-perception enhancement of panoramic images that integrates deformable convolution and spherical attention is provided, comprising the following steps:

[0009] Step S1: Map the original panoramic image in equidistant cylindrical projection ERP format to polyhedral mesh vertices to obtain the image features of each mesh vertex;

[0010] Step S2: Input the image features of all grid vertices into the U-shaped encoder-decoder network, and obtain the image perception result based on the output of the U-shaped encoder-decoder network;

[0011] The U-shaped encoder-decoder network includes a D-level encoder and a D-level decoder. Each encoder level contains M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level contains M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and decoder via skip connections. M and D are both positive integers greater than 1.

[0012] The dual-sensory enhancement module includes a deformable spherical convolution branch, a deformable spherical local self-attention branch, and an adaptive gating fusion block.

[0013] The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices.

[0014] The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex.

[0015] The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

[0016] According to a second aspect of the embodiments of this specification, a panoramic image dual-perception enhancement device that integrates deformable convolution and spherical attention is provided, comprising:

[0017] The image conversion unit is used to map the original panoramic image in equidistant cylindrical projection ERP format into polyhedral grid vertices to obtain the image features of each grid vertex;

[0018] An image sensing unit is used to input the image features of all grid vertices into a U-shaped encoder-decoder network and obtain the image sensing result based on the output of the U-shaped encoder-decoder network. The U-shaped encoder-decoder network includes a D-level encoder and a D-level decoder. Each encoder level contains M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level contains M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and decoder via skip connections. M and D are both positive integers greater than 1.

[0019] The dual-sensory enhancement module includes a deformable spherical convolution branch, a deformable spherical local self-attention branch, and an adaptive gating fusion block.

[0020] The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices.

[0021] The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex.

[0022] The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

[0023] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0024] The embodiments of this application have at least the following technical effects:

[0025] First, the embodiments of this application convert the original panoramic image in ERP format into a polyhedral mesh image and design a dual-branch parallel feature extraction architecture for it. This architecture deeply integrates the geometric adaptive capability of deformable convolution with the global modeling capability of self-attention, overcoming the design limitations of related schemes that are serially stacked or simply connected in parallel. This enables attention to have geometric adaptive capability on the spherical manifold, breaking through the limitation of fixed sampling of traditional spherical attention.

[0026] Second, in the embodiments of this application, a deformable offset is introduced into the local self-attention branch of the deformable sphere, so that the sampling points in the attention neighborhood can be dynamically adjusted according to the feature content, thereby enhancing the model's adaptability to irregular geometric deformations.

[0027] Third, the adaptive gating fusion block in this application learns fusion weights for each mesh vertex and each channel in a data-driven manner, dynamically balancing the contributions of local geometric features and global context features, and avoiding the feature annihilation problem caused by fixed weight fusion;

[0028] Fourth, the U-shaped encoder-decoder network in this application embodiment can be trained using multi-task joint loss based on geometry and semantics to achieve computational sharing and feature mutual calibration among operators, thereby improving computational efficiency. Attached Figure Description

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0030] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of a panoramic image dual-perception enhancement method that integrates deformable convolution and spherical attention;

[0031] Figure 2 This is a schematic diagram illustrating the framework of a U-shaped codec network according to an exemplary embodiment of this application;

[0032] Figure 3 This is a schematic diagram illustrating the mesh topology and vertex density distribution at different levels when a sphere is discretized into an icosahedral mesh, as shown in an exemplary embodiment of this application.

[0033] Figure 4 This is a schematic diagram of the framework of a deformable spherical convolution branch shown in an exemplary embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the framework of a deformable spherical local self-attention branch, as illustrated in an exemplary embodiment of this application.

[0035] Figure 6 This is a schematic diagram of the framework of an adaptive gating fusion block shown in an exemplary embodiment of this application;

[0036] Figure 7 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application;

[0037] Figure 8 This is a structural block diagram of a panoramic image dual-sensory enhancement device that integrates deformable convolution and spherical attention, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0039] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0040] This application provides a panoramic image dual-perception enhancement method that integrates deformable convolution and spherical attention. By deploying geometrically adaptive convolution branches and global contextual attention branches in parallel on a polyhedral mesh, it achieves the geometric adaptability of deformable convolution and the global modeling capability of self-attention. Through deep synergy between the geometric adaptability of spherical discrete mesh and global semantic modeling capability in four dimensions: operator, computation, fusion, and cross-scale, it solves the technical problems of operator mismatch, computational independence, lack of spherical specificity in fusion, and easy topological breakage across scales in related schemes.

[0041] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of a panoramic image dual-perception enhancement method that integrates deformable convolution and spherical attention. Figure 1 As shown, the panoramic image dual-sensory enhancement method includes at least the following steps:

[0042] Step S1: Map the original panoramic image in equidistant cylindrical projection ERP format to polyhedral mesh vertices to obtain the image features of each mesh vertex.

[0043] In some embodiments, the polyhedral mesh vertices are icosahedral mesh vertices. It is understood that in other embodiments, the original panoramic image in ERP format may also be mapped to other polyhedral mesh vertices.

[0044] Step S2: Input the image features of all grid vertices into the U-shaped encoder-decoder network, and obtain the image perception result based on the output of the U-shaped encoder-decoder network.

[0045] like Figure 2 As shown, the U-shaped encoder-decoder network in this embodiment includes a D-level encoder and a D-level decoder. Each encoder level includes M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level includes M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and the decoder through skip connections. M and D are both positive integers greater than 1.

[0046] The Dual-Perception Enhancement Block (DPEB) module includes a Deformable Spherical Convolution (SDC) branch, a Deformable Spherical Self-Attention (DSSA) branch, and a Gated Fusion Unit (GFU).

[0047] The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices.

[0048] The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex.

[0049] The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

[0050] To address the problems of icosahedral meshes, such as polar geometric distortion, strong regional heterogeneity, and easy topological breakage across scales, this embodiment designs a four-dimensional systematic fusion architecture. It extracts local geometric deformation features and global context features in parallel through deformable spherical convolutional branches and deformable spherical local self-attention branches. It achieves deep feature integration by combining the adaptive gating fusion block of the icosahedral mesh, and ensures the topological consistency of cross-scale transmission of the two branches through a geometrically aware upsampling module. Finally, multi-scale feature extraction and task decoding are completed in a U-shaped encoding and decoding network.

[0051] In some embodiments, step S1 obtains the image features of each grid vertex through the following steps:

[0052] Calculate the spherical coordinates of each grid vertex and map the spherical coordinates to ERP image coordinates;

[0053] In the ERP image coordinate system, the RGB values ​​of the original panoramic image are sampled using a preset interpolation algorithm to obtain the features of each grid vertex;

[0054] By projecting the features of each grid vertex onto an embedding space of a preset dimension using linear projection, the image features of each grid vertex are obtained.

[0055] For example, suppose an ERP panoramic image By using spherical coordinate mapping and differentiable resampling, it is converted into a vertex representation of a regular icosahedral mesh. ,in The number of grid vertices, This represents the initial number of feature channels.

[0056] The icosahedral mesh is generated through recursive subdivision, with subdivision levels... Corresponding number of vertices ,refer to Figure 3 The diagram shows the discretization at L=0, L=1, L=2, L=3, L=4, and L=5. When L=0, the corresponding icosahedral mesh has 12 vertices and 20 faces; when L=1, it has 42 vertices and 80 faces; when L=2, it has 162 vertices and 320 faces; when L=3, it has 642 vertices and 1280 faces; when L=4, it has 2562 vertices and 5120 faces; and when L=5, it has 10242 vertices and 20480 faces. Considering the polar geometric distortion, strong regional heterogeneity, and cross-scale topological breakage inherent in icosahedral meshes, the subdivision level of the icosahedral mesh in this embodiment is... Among them, grid vertices Satisfying the unit spherical constraint Its neighborhood structure is uniquely determined by the grid topology.

[0057] like Figure 2 As shown, the U-shaped codec network in this embodiment uses spherical features. For input, a U-shaped encoder-decoder is constructed, comprising D-level downsampling and D-level upsampling, exemplarily where D=4. Each encoder level consists of M stacked dual-sensory enhancement modules and one downsampling layer; each decoder level consists of M stacked DPEB modules and one upsampling layer; features are transferred between corresponding levels of the encoder and decoder via skip connections, and the skip connection features are calibrated for icosahedral mesh vertex alignment before fusion to ensure topological consistency of the dual-branch features.

[0058] Meanwhile, based on the characteristics of icosahedral meshes with polar region emphasis on geometry and deep layer emphasis on semantics, a resolution adaptive operator allocation strategy is designed: on shallow high-resolution feature maps (corresponding to fine polar region geometric features), deformable spherical convolutional branches are deployed, and on deep low-resolution feature maps (corresponding to global semantic features), deformable spherical local self-attention branches are deployed, so as to achieve efficient allocation of computing resources and accurate capture of spherical features.

[0059] In this embodiment, the DPEB module adopts a dual-branch parallel architecture with heterogeneous operator adaptation and collaborative computing sharing. It also includes a deformable spherical convolution branch and a deformable spherical local self-attention branch, and achieves feature integration through an adaptive gating fusion block dedicated to the icosahedral mesh.

[0060] In some embodiments, the deformable spherical convolution branch is configured to obtain local geometric features through the following steps:

[0061] Obtain the K-order neighbor vertex set of the current grid vertex; predict the deformable offset of each neighbor vertex in the K-order neighbor vertex set using a convolutional subnetwork; adjust the position of each neighbor vertex according to its deformable offset to obtain the adjusted sampling position; obtain the interpolation feature of the nearest neighbor vertex of the sampling position by weighting the features of the nearest neighbor grid vertex; and aggregate the interpolation features of all neighbor vertices of the current grid vertex by weighting and a shared projection matrix using learnable neighborhood weights to obtain the local geometric features corresponding to the current grid vertex.

[0062] like Figure 4 The deformable spherical convolution branch shown is used for input features. Where N is the number of vertices and C is the number of channels. For each grid vertex... Define the K-order neighborhood vertex set , .

[0063] For the k-th original neighbor vertex of the current grid vertex i The deformable offset is predicted by the offset prediction network. Optionally, the deformable offset can be normalized to a unit sphere using the following formula to preserve topological constraints:

[0064] (1)

[0065] In formula (1), This is the sampling position of the kth neighboring vertex of the current grid vertex i.

[0066] The image features of the original sphere's mesh vertices exist only on fixed mesh vertices, due to the deformable offset. The mesh vertices of the deformable offsets do not usually coincide with those of the original sphere, therefore, there are no directly corresponding image features for each deformable offset.

[0067] Based on this, this embodiment employs a spherical bilinear interpolation algorithm to... The four nearest original grid vertices on the sphere are taken as nearest neighbor grid vertices. Weights are assigned according to the arc length of the sphere, and the image features of these four nearest neighbor grid vertices are weighted and summed to extract interpolation features from the input feature X.

[0068] (2)

[0069] In formula (2), It is the interpolation feature of the k-th neighborhood vertex of the current grid vertex i.

[0070] Learnable neighborhood weights With shared projection matrix We perform weighted aggregation of the interpolation features of all neighborhood vertices of the current mesh vertex i to obtain the local geometric features of the target vertex i:

[0071] (3)

[0072] In formula (3), For learnable neighborhood weights, This represents the local geometric features extracted from the current mesh vertex i through deformable spherical convolution.

[0073] In some embodiments, the deformable spherical local self-attention branch is configured to obtain global context features through the following steps:

[0074] The image features of the current grid vertex and the interpolated features of its neighboring vertices are projected using a learnable linear projection matrix to obtain the query vector corresponding to the current grid vertex and the key and value vectors corresponding to its neighboring vertices. The deformable offsets of each neighboring vertex are then corrected to obtain the relative position offsets of each neighboring vertex. Based on the query vector corresponding to the current grid vertex, the key vectors corresponding to each neighboring vertex, and the relative position offsets of each neighboring vertex, the attention weights corresponding to each neighboring vertex are obtained. Based on the attention weights, the value vectors of all neighboring vertices of the current grid vertex are weighted and aggregated to obtain the global context features of the current grid vertex.

[0075] The relative position offsets of each neighboring vertex are obtained through the following steps:

[0076] The adjusted sampling position is calculated based on the deformable offset of each neighboring vertex, as well as the longitude and latitude difference of the sampling position relative to the current grid vertex; the relative position offset of each neighboring vertex is obtained by bilinear interpolation of the longitude and latitude difference.

[0077] like Figure 5 The deformable spherical local self-attention branch shown applies to the same input features. For each grid vertex Define the same K-order neighborhood A lightweight offset prediction network pre-trained with deformable spherical convolution branches is used to extract the sampling position of the k-th neighborhood vertex of the current grid vertex i. and its interpolation characteristics It then performs dimensional transformation using a learnable linear projection matrix to generate a query vector. key vector Sum value vector :

[0078] (4)

[0079] In formula (4), d is the shared learnable projection matrix, where d is the attention head dimension.

[0080] This indicates that the query vector is derived from the image features of the current grid vertex i itself. The generation corresponds to the currently unprocessed grid vertex i, representing the information query requirement of grid vertex i, and is independent of the neighborhood sampling position; while K and This indicates that the key and value vector is derived from the interpolation features of the k-th neighborhood vertex of grid vertex i. Generate, where K and In the index i, the grid vertex i is represented, and the index k represents the sampling position of the kth neighborhood of the grid vertex i. The index k represents the information label (K) and actual information (V) of the kth sampling position.

[0081] This embodiment calculates the sampling position based on the spherical topological properties of a regular icosahedral mesh. Relative to the current grid vertex Longitude difference Latitude difference and normalized to After the interval, the learnable 7×7 grid parameters Perform bilinear interpolation to obtain the relative position offset. The relative position offset is used to perform position correction on feature similarity to enhance the spherical spatial position perception capability of the attention mechanism.

[0082] Combined with the current mesh vertices The query vector, the grid vertex The original attention score is calculated using the key vector corresponding to the k-th domain vertex and its relative position bias. This score is then normalized using the Softmax function to obtain the attention weight. The attention weight is expressed by the following formula:

[0083] (5)

[0084] In formula (5), , The unnormalized raw attention score, The normalized attention weights represent the sampling positions. Interpolation features for mesh vertices The contribution of feature modeling.

[0085] Based on the obtained attention weights, the value vectors corresponding to all neighborhood vertices are weighted and summed to obtain the current grid vertex. The global context features are represented by the following formula:

[0086] (6)

[0087] Current grid vertex The global context features not only preserve the global semantic association information of the spherical panoramic image, but also adapt to the geometric deformation characteristics of the local region of the spherical surface, providing a high-quality global semantic feature foundation for subsequent dual-branch feature fusion.

[0088] In some embodiments, the adaptive gating fusion block of the U-shaped codec network is as follows: Figure 6 As shown,

[0089] To effectively integrate the local geometric features of the deformable spherical convolutional branch with the global context features of the deformable spherical local self-attention branch, and considering the regional heterogeneity of the icosahedral mesh—heavy geometry in the polar region, heavy semantics in the equatorial sky, heavy geometry with large deformation, and heavy semantics with small deformation—this embodiment designs an adaptive gating fusion block driven by three factors: spherical position, mesh topology, and degree of deformation. The fusion weight is dynamically determined by the inherent characteristics of the icosahedral mesh and the spherical geometric deformation state.

[0090] Local geometric features based on the output of deformable spherical convolution branches Global context features of the output of the local self-attention branch of the deformable sphere Construct joint features :

[0091] (7)

[0092] And generate fusion weights through a gating mechanism. :

[0093] (8)

[0094] In formula (8), For learnable parameters, The learning process of the fusion weight G is bound to the spherical position, neighborhood topology type, and degree of geometric deformation of the icosahedral mesh, using the Sigmoid activation function.

[0095] The adaptive gating fusion block ultimately outputs dual-sensor enhanced features. :

[0096] (9)

[0097] In formula (9), This represents the element-wise Hadamard product.

[0098] In some embodiments, the U-shaped codec network employs a geometry-aware upsampling and downsampling strategy to ensure geometric fidelity during feature transfer.

[0099] Continue to refer to Figure 2 During the downsampling process, let the first... The number of vertices in the level mesh is After downsampling, the first The number of vertices in the level mesh is Each parent vertex Corresponding to a set of child vertices The downsampling operation aggregates sub-vertex features:

[0100] (10)

[0101] The Aggregate function can be selected from max pooling, average pooling or deformable pooling. Optionally, max pooling is preferred to preserve salient features.

[0102] During upsampling, the features of the parent vertex are propagated to its child vertices. its child vertices The characteristics are:

[0103] (11)

[0104] This embodiment can use nearest neighbor interpolation and also supports advanced strategies such as bilinear interpolation.

[0105] In some embodiments, the U-shaped codec network further includes an output task head, which is configured to perform task processing on the final output features of the decoder to obtain the image perception result corresponding to the original panoramic image.

[0106] The output task header includes at least one of the following task headers:

[0107] The semantic segmentation head is configured to linearly map the final output features of the decoder and process them through the Softmax function to obtain the semantic category profile;

[0108] Depth estimation head: configured to linearly map the final output features of the decoder and process them with the Sigmoid function to obtain normalized depth values.

[0109] In practical applications, in building such Figure 2 Following the U-shaped encoder-decoder network shown, end-to-end joint optimization can be performed using a multi-task loss function. For semantic segmentation tasks, cross-entropy loss is used; for depth estimation tasks, BerHu loss or L1 loss is used; and for joint tasks, weighted sum loss is used.

[0110] Next, the implementation effects of the embodiments of this application will be explained with reference to experimental data.

[0111] First, perform data preparation and preprocessing.

[0112] This embodiment uses the Stanford 2D-3D dataset and the CVRG-Pano dataset to verify the effectiveness of the embodiments of this application on the panoramic image semantic segmentation task.

[0113] The Stanford 2D-3D dataset contains 1413 high-resolution indoor panoramic images with an original resolution of 2048×4096, providing pixel-level semantic annotations for 13 categories. The CVRG-Pano dataset contains 600 outdoor panoramic images with an original resolution of 1664×832, merging the original 20 categories into 7 main categories: ground, buildings, objects, nature, sky, people, and vehicles.

[0114] This embodiment performs the following preprocessing on the original dataset:

[0115] The panoramic images of the ERP were uniformly scaled to a resolution of 512×1024 while maintaining a 2:1 aspect ratio.

[0116] The pixel values ​​of the ERP panoramic image are normalized to the [0,1] range.

[0117] It is worth noting that this embodiment does not perform any data augmentation to ensure a fair comparison with the baseline method.

[0118] Then, a spherical discretization representation is constructed.

[0119] An icosahedral mesh was used as the spherical discretization basis, with a subdivision level of L=6, corresponding to 40,962 vertices and 81,920 faces. The ERP panoramic image was mapped to the mesh vertices via spherical inverse projection.

[0120] For grid vertices Its spherical coordinates are as follows:

[0121] (12)

[0122] in, .

[0123] grid vertices Mapping the spherical coordinates to the ERP image coordinates, we have:

[0124] (13)

[0125] Vertex features are obtained by sampling RGB values ​​from the ERP image using bilinear interpolation. .

[0126] Extending 3D RGB features to a C=64-dimensional embedding space using linear projection:

[0127] (14)

[0128] Next, configure the U-shaped codec network.

[0129] This embodiment adopts a D=4 level encoding / decoding architecture, and the feature dimensions and resolution configurations for each level are shown in Table 1:

[0130]

[0131] Table 1

[0132] In Table 1, α represents the initial weight tendency of the deformable spherical convolution branch in the adaptive gated fusion block, which is adaptively adjusted through learnable parameters.

[0133] The DPEB module parameters are configured as follows:

[0134] 1) The configuration of the deformable spherical convolution branch is as follows:

[0135] Neighborhood order K=2, neighborhood size ;

[0136] Offset prediction network: two-layer MLP, hidden layer dimension 32, output dimension 19×3;

[0137] Neighborhood weight Learnable scalar, initialized to 1 / 19;

[0138] Shared projection matrix: .

[0139] 2) The configuration of the local self-attention branch on the deformable sphere is as follows:

[0140] The neighborhood order K=2, and the neighborhood size is consistent with the deformable spherical convolution branch;

[0141] Attention head count H=4, head dimension d=C / H

[0142] Relative position offset grid 7×7, learnable parameters, uniform initialization;

[0143] Q / K / V projection matrix: .

[0144] 3) The adaptive gating fusion block is configured as follows:

[0145] Gated projection matrix: ;

[0146] Sigmoid activation outputs fusion weights;

[0147] Residual connection: Output .

[0148] 4) The upsampling configuration is as follows:

[0149] Max pooling is used to aggregate sub-vertex features, reducing the number of vertices from [previous level] to [new level] across different levels of the Encoder. Down to The number of channels is doubled.

[0150] 5) The downsampling configuration is as follows:

[0151] Nearest neighbor interpolation is used to directly copy the features of the parent vertex to the four child vertices. Between each level of the decoder, the number of vertices is reduced from... Restore to The number of channels is halved.

[0152] 6) The output header and loss function are configured as follows:

[0153] Semantic segmentation head: Single-layer linear classifier K=13 (Stanford 2D-3D) or 7 (CVRG-Pano).

[0154] The loss function is cross-entropy loss, ignoring the background class.

[0155] 7) The training configuration is as follows:

[0156] Optimizer: AdamW, initial learning rate 1e-4, weight decay 0.05;

[0157] Learning rate scheduling: cosine annealing, minimum learning rate 1e-5, linear warm-up for 5 epochs;

[0158] Training epochs: 200;

[0159] Batch size: 8 (distributed data parallelism, 4×NVIDIA RTX 4090);

[0160] Gradient clipping: Global norm clipped to 1.0;

[0161] Random seed: 42.

[0162] 8) The evaluation indicators are configured as follows:

[0163] The main indicator is the mean Intersection over Union (mIoU).

[0164] A secondary metric is pixel accuracy.

[0165] Specialized indicators include latitude-segmented mIoU (each interval is 30°) and category IoU (focusing on smaller targets).

[0166] To verify the effectiveness of each core module of the present invention, a systematic ablation experiment was conducted on the Stanford 2D-3D dataset in response to the above configuration.

[0167] First, the necessity of the two-branch path was verified, and the verification results are shown in Table 2:

[0168]

[0169] Table 2

[0170] Based on the experimental results shown in Table 2, it can be seen that using either branch alone cannot achieve optimal performance. The dual-branch parallel architecture brings a significant gain (+4.9% mIoU), which verifies the necessity of the dual-branch design in the embodiments of this application.

[0171] Next, we performed ablation verification using a deformable mechanism. The verification results are shown in Table 3.

[0172]

[0173] Table 3

[0174] Based on the experimental results shown in Table 3, it can be seen that the adaptive gating fusion strategy is significantly better than fixed weight fusion and simple summation, which verifies the effectiveness of the adaptive gating fusion block in the embodiments of this application.

[0175] Furthermore, the ablation verification of the neighborhood order K was performed, and the verification results are shown in Table 4:

[0176]

[0177] Table 4

[0178] Based on the experimental results shown in Table 4, it can be seen that K=2 achieves a balance between accuracy and efficiency, while K=3 provides only a slight performance gain but a significant increase in computational load. Therefore, K=2 is an option.

[0179] Then, mesh resolution ablation verification was performed, and the verification results are shown in Table 5:

[0180]

[0181] Table 5

[0182] Based on the experimental results shown in Table 5, L=7 is the optimal resolution level, but the computational load increases by 4 times. Therefore, L=6 can be optionally set.

[0183] In addition, this embodiment also conducted comparative experiments with related technologies, referring to Table 6 for semantic segmentation performance comparison on the Stanford2D-3D dataset, Table 7 for semantic segmentation performance comparison on the CVRG-Pano dataset, and Table 8 for rotation robustness test comparison on the Stanford2D-3D dataset:

[0184]

[0185] Table 6

[0186] It is worth noting that the MOAT-0 report results are based on ImageNet pre-training, and all methods in Table 6 are trained from scratch.

[0187]

[0188] Table 7

[0189] This embodiment also verifies the effectiveness of the technical solution in this application on the depth estimation task on the Stanford2D-3D dataset.

[0190] The semantic segmentation head of the U-shaped encoder-decoder network in this application embodiment is replaced with a depth regression head, with single-channel output. Sigmoid activation normalizes the depth values ​​to the [0,1] interval. The BerHu loss function is used, with a threshold c=0.2. Evaluation metrics include mean absolute error (MAE), mean relative error (MRE), root mean square error (RMSE), and δ1 accuracy. Experimental results are shown in Table 8:

[0191]

[0192] Table 8

[0193] Based on the experimental results shown in Table 8, it can be seen that the embodiments of this application also achieve the best performance in the depth estimation task, with MAE reduced by 10.9% and δ1 accuracy improved by 2.4 percentage points compared to SphereUFormer.

[0194] This embodiment also compares the inference speed and memory usage of each method on an NVIDIA RTX 4090 GPU. The comparison results are shown in Table 9:

[0195]

[0196] Table 9

[0197] Based on the experimental results shown in Table 9, it can be seen that, with a slight increase in the number of parameters, the embodiments of this application control FLOPs to 4.1G through the windowed attention mechanism, which is 58.6% lower than the pure Transformer method; the inference speed reaches 7.5ms / frame, which is 33.0% faster than SphereUFormer, meeting the requirements for real-time processing.

[0198] Furthermore, to verify the generality of the DPEB module in this embodiment, it was inserted as an independent operator into the SphereUFormer backbone network. The overall U-shaped architecture and upsampling / downsampling strategy of SphereUFormer were retained, the original self-attention module was replaced with the DPEB module, and other configurations remained unchanged. The network was trained from scratch on the Stanford 2D-3D dataset. The experimental results are shown in Table 10.

[0199]

[0200] Table 10

[0201] Based on the experimental results shown in Table 10, it can be seen that the DPEB module in this embodiment can directly replace the self-attention module in the existing spherical Transformer architecture, and can bring significant performance gains without any architectural adjustments, verifying its plug-and-play capability and good generalization.

[0202] Based on the above embodiments of this application, the panoramic image dual-sensory enhancement method of this application has at least the following beneficial effects:

[0203] (1) Segmentation accuracy is significantly improved;

[0204] On the Stanford 2D-3D panoramic semantic segmentation dataset, the mIoU of the embodiments in this application reaches 74.3%, which is 8.5 percentage points higher than SphDeformConv (65.8%), 4.8 percentage points higher than SphereUFormer (69.5%), and 9.8 percentage points higher than CTRL-F (65.0%).

[0205] On the CVRG-Pano outdoor panoramic dataset, the mIoU of the embodiments in this application reaches 68.3%, which is more than 2.5 percentage points higher than the existing best methods.

[0206] (2) Excellent ability to segment small targets;

[0207] On the pedestrian category (typical small target) of the CVRG-Pano dataset, the IoU of the embodiments in this application reaches 27.8%, which is 16.7 percentage points higher than SphDeformConv (11.1%), 13.6 percentage points higher than SphereUFormer (14.2%), and 7.3 percentage points higher than CTRL-F (20.5%).

[0208] Experiments have shown that the embodiments of this application can effectively maintain the edge details and geometric integrity of small targets.

[0209] (3) Good rotational isomorphism;

[0210] In the horizontal rotation test (Z-axis rotation), the mIoU fluctuation of the embodiment of this application is very small, on par with SphereUFormer, and significantly better than SphDeformConv (fluctuation of 1.2%).

[0211] In the vertical / horizontal flip test (X / Y axis rotation), the mIoU of the embodiments of this application decreased only slightly, while SphDeformConv decreased by 41.2% and SphereUFormer decreased by 38.5%.

[0212] Experiments have shown that the embodiments of this application inherit the rotational equivariance advantages of spherical representation.

[0213] (4) Excellent computational efficiency;

[0214] Under the same parameter scale (approximately 25M), the FLOPs of the embodiments in this application are 4.1G, which is 58.6% lower than SphereUFormer (9.9G) and 24.2% higher than CTRL-F (3.3G), but the performance gain is significant.

[0215] In terms of inference speed, the embodiment of this application takes 7.5ms to process a single 512×1024 panoramic image on an NVIDIA RTX 4090 GPU, which is 33.0% faster than SphereUFormer (11.2ms).

[0216] Experiments have shown that the embodiments of this application achieve the optimal balance between accuracy and efficiency through a windowed attention and resolution adaptive strategy.

[0217] (5) Plug and play and generalization capabilities;

[0218] The DPEB module in this embodiment can be embedded as an independent operator into any spherical network architecture. After directly replacing the original self-attention module with the DPEB module on the SphereUFormer backbone network, a large gain can be obtained on the Stanford2D-3D dataset without retraining.

[0219] Experiments have shown that the DPEB module of this application has good versatility and portability.

[0220] (6) The embodiments of this application have broad industrial application prospects, for example:

[0221] Autonomous driving surround view perception: Autonomous vehicles are generally equipped with multi-view surround view cameras to generate a 360° panoramic view. The embodiments of this application can be directly applied to the segmentation and depth estimation of surround view fisheye images, providing vehicles with omnidirectional perception capabilities, significantly improving the detection accuracy of small targets such as pedestrians and non-motorized vehicles, and enhancing the safety of autonomous driving systems in complex urban environments.

[0222] Virtual reality and augmented reality (VR / AR) devices require real-time three-dimensional understanding of the surrounding environment to achieve functions such as spatial positioning and object interaction. The embodiments of this application achieve high-precision panoramic segmentation and depth estimation while maintaining low computational latency, providing a lightweight and high-precision environmental perception solution for head-mounted display devices.

[0223] Robot navigation and obstacle avoidance: Mobile platforms such as service robots and sweeping robots need to perceive obstacles and passable areas in real time in complex indoor environments. The embodiments of this application are based on the rotational isotropic properties of spherical manifolds, which enable robots to stably identify environmental structures without relying on absolute orientation, significantly improving navigation robustness.

[0224] In smart security and video surveillance, panoramic surveillance cameras need to detect and track people, vehicles, and abnormal events in real time within a 360° field of view. The small target segmentation advantage of this application embodiment can effectively improve the detection rate of people at a distance, providing a more reliable intelligent analysis capability for security systems.

[0225] In medical image processing, the dual-branch parallel fusion architecture proposed in this application is universal and can be transferred to medical image segmentation tasks. By replacing the icosahedral mesh with a medical image-specific mesh structure (such as a spherical cranial atlas), this application can be used for medical imaging applications such as cortical segmentation and retinal surface analysis.

[0226] Figure 7 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 708, a hardware acceleration device 710, and non-volatile memory 712, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 702 reads the corresponding computer program from the non-volatile memory 712 into the memory 708 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0227] Figure 8 This is a structural block diagram illustrating an exemplary embodiment of this application of a panoramic image dual-sensory enhancement device that integrates deformable convolution and spherical attention. The panoramic image dual-sensory enhancement device can be applied to, for example... Figure 7 The electronic device shown implements the technical solution of this application. The panoramic image dual-sensing enhancement device includes: an image conversion unit 810 and an image sensing unit 820, wherein:

[0228] Image conversion unit 810 is used to map the original panoramic image in equidistant cylindrical projection ERP format into polyhedral grid vertices to obtain the image features of each grid vertex;

[0229] The image sensing unit 820 is used to input the image features of all grid vertices into a U-shaped encoder-decoder network and obtain the image sensing result based on the output of the U-shaped encoder-decoder network. The U-shaped encoder-decoder network includes a D-level encoder and a D-level decoder. Each encoder level contains M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level contains M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and decoder via skip connections. M and D are both positive integers greater than 1.

[0230] The dual-sensory enhancement module includes a deformable spherical convolution branch, a deformable spherical local self-attention branch, and an adaptive gating fusion block.

[0231] The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices.

[0232] The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex.

[0233] The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

[0234] In some embodiments, the image conversion unit 810 is used to calculate the spherical coordinates of each grid vertex and map the spherical coordinates to ERP image coordinates; in the ERP image coordinate system, the RGB values ​​of the original panoramic image are sampled using a preset interpolation algorithm to obtain the features of each grid vertex; and the features of each grid vertex are projected to an embedding space of a preset dimension through linear projection to obtain the image features of each grid vertex.

[0235] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0236] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0237] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0238] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0239] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0240] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0241] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0242] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0243] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0244] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0245] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0246] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A panoramic image dual-perception enhancement method integrating deformable convolution and spherical attention, characterized in that, Includes the following steps: Step S1: Map the original panoramic image in equidistant cylindrical projection ERP format to polyhedral mesh vertices to obtain the image features of each mesh vertex; Step S2: Input the image features of all grid vertices into a U-shaped encoder-decoder network, and obtain the image perception result based on the output of the U-shaped encoder-decoder network. The U-shaped encoder-decoder network includes a D-level encoder and a D-level decoder. Each encoder level contains M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level contains M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and the decoder through skip connections. M and D are both positive integers greater than 1. Wherein: The dual-sensory enhancement module includes a deformable spherical convolution branch, a deformable spherical local self-attention branch, and an adaptive gating fusion block. The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices. The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex. The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

2. The method according to claim 1, characterized in that, The deformable spherical convolution branch is configured to obtain local geometric features through the following steps: Get the set of K-order neighborhood vertices of the current grid vertex; Predict the deformable offset of each neighboring vertex in the K-order neighborhood vertex set using a convolutional subnetwork; The position of each neighboring vertex is adjusted based on its deformable offset to obtain the adjusted sampling position. The interpolated features of the nearest neighbor grid vertex at the sampling location are obtained by weighting the features of that neighboring vertex. By weighting and aggregating the interpolated features of all neighboring vertices of the current grid vertex using learnable neighborhood weights and a shared projection matrix, the local geometric features corresponding to the current grid vertex are obtained.

3. The method according to claim 1, characterized in that, The deformable spherical local self-attention branch is configured to obtain global context features through the following steps: By using a learnable linear projection matrix, the image features of the current grid vertex and the interpolation features of each neighboring vertex of the current grid vertex are projected and transformed to obtain the query vector corresponding to the current grid vertex and the key vector and value vector corresponding to each neighboring vertex. The deformable offsets of each neighboring vertex are corrected to obtain the relative position offsets of each neighboring vertex. Based on the query vector corresponding to the current grid vertex, the key vector corresponding to each neighboring vertex, and the relative position bias corresponding to each neighboring vertex, the attention weight corresponding to each neighboring vertex is obtained. Based on the attention weights, the value vectors of all neighboring vertices of the current grid vertex are weighted and aggregated to obtain the global context features of the current grid vertex.

4. The method according to claim 3, characterized in that, The deformable spherical local self-attention branch is configured to obtain the relative position bias of each neighboring vertex through the following steps: The adjusted sampling position is calculated based on the deformable offset of each neighboring vertex, as well as the longitude and latitude difference of the sampling position relative to the current grid vertex. The relative position offset of each neighboring vertex is obtained by bilinear interpolation of the longitude difference and latitude difference.

5. The method according to claim 1, characterized in that, The dual-sensory enhancement feature is determined by the following formula: in, The fusion weights are... For local geometric features, For global context features, The element-wise Hadamard product operator, This is a feature that enhances dual perception.

6. The method according to claim 1, characterized in that, The U-shaped encoding / decoding network also includes an output task head, which is configured to perform task processing on the final output features of the decoder to obtain the image perception result corresponding to the original panoramic image.

7. The method according to claim 6, characterized in that, The output task header includes at least one of the following task headers: The semantic segmentation head is configured to linearly map the final output features of the decoder and process them through the Softmax function to obtain the semantic category probabilities; Depth estimation head: configured to linearly map the final output features of the decoder and process them with the Sigmoid function to obtain normalized depth values.

8. The method according to claim 1, characterized in that, Step S1 obtains the image features of each grid vertex through the following steps: Calculate the spherical coordinates of each grid vertex and map the spherical coordinates to ERP image coordinates; In the ERP image coordinate system, the RGB values ​​of the original panoramic image are sampled using a preset interpolation algorithm to obtain the features of each grid vertex; By projecting the features of each grid vertex onto an embedding space of a preset dimension using linear projection, the image features of each grid vertex are obtained.

9. A panoramic image dual-perception enhancement device integrating deformable convolution and spherical attention, characterized in that, include: The image conversion unit is used to map the original panoramic image in equidistant cylindrical projection ERP format into polyhedral grid vertices to obtain the image features of each grid vertex; An image sensing unit is used to input the image features of all grid vertices into a U-shaped encoder-decoder network and obtain the image sensing result based on the output of the U-shaped encoder-decoder network. The U-shaped encoder-decoder network includes a D-level encoder and a D-level decoder. Each encoder level contains M stacked dual-sensory enhancement modules and a downsampling layer, and each decoder level contains M stacked dual-sensory enhancement modules and an upsampling layer. Features are transferred between corresponding levels of the encoder and decoder via skip connections. M and D are both positive integers greater than 1. The dual-sensory enhancement module includes a deformable spherical convolution branch, a deformable spherical local self-attention branch, and an adaptive gating fusion block. The deformable spherical convolution branch is configured to predict the deformable offset of its neighboring vertices through convolution for each grid vertex of the current input, and determine the interpolation features of each neighboring vertex based on the deformable offset, so as to obtain the local geometric features of each grid vertex based on the aggregation of the interpolation features of all neighboring vertices. The deformable spherical local self-attention branch is configured to calculate a query vector, a key vector, and a value vector for each grid vertex of the current input based on the image features of the current grid vertex and the interpolation features of its neighboring vertices. Based on the query vector, the deformable offset corresponding to each neighboring vertex, and the key vector, the attention weight of the neighboring vertex is generated. The attention weight is used to perform weighted aggregation on the value vector of the corresponding neighboring vertex to obtain the global context features of the current grid vertex. The adaptive gating fusion block is configured to concatenate the local geometric features and the global context features and generate fusion weights through a gating mechanism, so as to perform weighted fusion of the local geometric features and the global context features based on the fusion weights to obtain dual-perception enhancement features.

10. An electronic device, characterized in that, include: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 8.

Images