Panoramic image restoration method based on adaptive feature fusion and self-attention mechanism
Through the panoramic image repair method that integrates adaptive features into Transformer and content-focused self-attention mechanism, the problems of spherical projection distortion and high-resolution characteristics in panoramic image repair are solved, and better repair results are achieved.
Patent Information
- Application Number
- CN202510429806.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-08-15
AI Technical Summary
The prior art is difficult to effectively handle the spherical projection distortion and high resolution characteristics of panoramic images, resulting in poor repair effects and prone to artifacts and blurring.
A panoramic image repair model that combines the Transformer module and the content-focused self-attention module is adopted, and the training and optimization is combined with the joint loss function, including adaptive feature extraction, dual-channel self-attention calculation and the design of multiple loss functions.
In panoramic image repair, global feature modeling and local details are taken into account, adapting to high-resolution characteristics, improving the repair effect and reducing artifacts and blur.
Smart Images
Figure CN120495131A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image restoration, and in particular to a panoramic image restoration method based on adaptive feature fusion and self-attention mechanism. Background Art
[0002] Panoramic images not only provide an immersive visual experience but also offer more comprehensive data support for tasks such as scene perception and environmental reconstruction. They have been a research hotspot in recent years, but their application also presents numerous challenges. Panoramic images differ significantly from traditional planar images in their data structure. The spherical projection characteristics of panoramic images lead to significant deformation and distortion when converted to a planar form. Furthermore, their ultra-high resolution and complex scene information make restoration tasks even more challenging.
[0003] Traditional planar image restoration methods, such as those based on convolutional neural networks, often exhibit certain limitations when processing panoramic images. On the one hand, CNNs are unable to fully capture the global information of panoramic images under a fixed receptive field and cannot fully understand the semantic information of panoramic images. On the other hand, existing methods have weak deformation processing capabilities for specific areas in panoramic images and cannot effectively deal with spherical projection distortion problems, which easily leads to poor image restoration effects, restoration results containing a large number of artifacts and blurs, or even restoration failures. Therefore, designing a restoration method that can take into account both global feature modeling and local details while adapting to the high-resolution characteristics of panoramic images is a hot research direction.
[0004] The Transformer model has made breakthrough progress in the field of computer vision in recent years. Its multi-head self-attention mechanism can efficiently capture long-range dependency information and is good at extracting global context information in images. However, it consumes a lot of computing resources when calculating global attention, and there are certain modeling conflicts when processing global features and local details. Summary of the Invention
[0005] In order to solve the above technical problems existing in the prior art, the present invention proposes a panoramic image restoration method based on adaptive feature fusion and self-attention mechanism. The specific technical solution is as follows:
[0006] A panoramic image restoration method based on adaptive feature fusion and self-attention mechanism, including:
[0007] A panoramic image restoration model is constructed, which includes an adaptive feature fusion Transformer module and a content-focused self-attention module. The original panoramic image is superimposed with a mask and then input into the panoramic image restoration model. The adaptive feature fusion Transformer module is used to extract and fuse features to obtain a fused feature map.
[0008] The fusion feature map input content is focused on the self-attention module to perform dual-channel self-attention calculation, and then added to obtain the output feature representation;
[0009] A joint loss function is used to train and optimize the panoramic image restoration model.
[0010] Furthermore, the mask image is used to divide the original panoramic image into visible area features F e and the feature F of the area to be repaired g Two independent parts, expressed as follows:
[0011] F e =F⊙φ(M)
[0012] F g =F⊙φ(1-M)
[0013] F represents the panoramic image feature map, M represents the mask map, ⊙ represents element-by-element matrix multiplication, and φ represents the closest downscaling.
[0014] Furthermore, the adaptive feature fusion Transformer module includes a context module, a potential module and an adaptive feature extraction module.
[0015] Furthermore, the adaptive feature extraction module inputs the visible area feature F e and the feature F of the area to be repaired g , after passing through the 1*1 convolution layer and the sigmoid layer, we get q, k, and v. We add q and k to v after matrix product, and then output feature A through the fully connected layer and the ReLu layer. p2p , the expression is as follows:
[0016]
[0017] Furthermore, the context module inputs the visible area feature F e and the feature F of the area to be repaired g After passing through the 1*1 convolution layer and the sigmoid layer, we get q, k, and v. We multiply the q and k matrices and pass them through the softmax layer and then multiply them with the v matrix to output the feature R. c .
[0018] Furthermore, the output feature R of the context module c And the output feature A of the adaptive feature extraction module p2p Multiply to get the feature map F e ', the expression is as follows:
[0019] F e '=A p2p ⊙R c ,
[0020] The latent module inputs the feature map F e ', and then pass through layer normalization, convolution layer, ReLu layer, convolution layer, ReLu, fully connected layer, convolution layer, ReLu, and convolution layer to output feature R l ;
[0021] The output feature R of the latent module l And the output feature A of the adaptive feature extraction module p2p Multiply to get R l * , the expression is as follows:
[0022] R l * =(1-A p2p )⊙R l ,
[0023] Finally, the original panoramic image feature F is combined with the output of the adaptive feature fusion Transformer module to obtain the fusion feature F * , the expression is as follows:
[0024] F * =A p2p ⊙R c +(1-A p2p )⊙R l +F.
[0025] Furthermore, in the panoramic image restoration model, the content-focused self-attention module is added between each adaptive feature fusion Transformer module, where the first content-focused self-attention module is jump-connected to the last content-focused self-attention module, the second content-focused self-attention module is jump-connected to the penultimate content-focused self-attention module, and so on.
[0026] Furthermore, the content-focused self-attention module divides the input fusion feature map into four windows, calculates Q, K and V through 1*1 convolution and 3*3 convolution in sequence, and divides the feature map into upper and lower channels for self-attention calculation, and finally adds the calculation results to obtain the output feature representation.
[0027] Furthermore, the upper channel of the upper and lower channels calculates the channel self-attention in each window for the input multiple feature maps to obtain the feature representation F top , the lower channel calculates the channel self-attention on multiple feature maps of the input without dividing the window, and obtains F bottom ;
[0028] Add the output results of the upper and lower channels to get the output feature representation F out =w top*F top +w bottom *F bottom , w top , w bottom represents a learnable parameter.
[0029] Furthermore, the joint loss function includes L1 loss, adversarial loss, feature matching loss and high receptive field perception loss.
[0030] The L1 loss: represent the real image and the repaired image respectively, ||·||1 represents the L1 norm;
[0031] The adversarial loss is composed of the discriminator loss L D and the generator loss L G Composition, only the features from the masked area are considered as L D For fake samples in , the adversarial loss is expressed as:
[0032]
[0033] L adv =L D +L G +λ GP L GP ,
[0034] L GP represents the gradient penalty, taking λ GP =1e-3, E[] represents the expected function, D() represents the feature map of the discriminator, represents the gradient;
[0035] The feature matching loss L FM Based on the L1 loss between the discriminator features of real and fake samples, L FM For stable training, the expression is as follows:
[0036]
[0037] N represents the number of layers of the discriminator, D (i) () represents the feature map of the i-th layer of the discriminator, M i It is the product of the number of channels, height and width of the feature map of the i-th layer, that is, M i =C i *H i *W i ;
[0038] The high receptive field perception loss L hrf The expression is as follows:
[0039]
[0040] where φ hrf Use the pre-trained segmentation network ResNet50 with dilated convolutions;
[0041] Combining the above loss functions, we get the final loss function, which is expressed as follows:
[0042] L final =λ L1 L L1 +λ adv L adv +λ fm L fm +L hrf L hrf ;
[0043] The weights of different loss items are determined by experimental tuning, λ L1 =10,λ adv =10,λ FM =100,λ hrf =30.
[0044] Beneficial effects: The present invention utilizes the adaptive feature fusion Transformer and the content-focused self-attention mechanism, which can take into account both global feature modeling and local details during image restoration while adapting to the high-resolution characteristics of panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 This is a main flow chart of a panoramic image restoration method based on adaptive feature fusion and self-attention mechanism according to an embodiment of the present invention;
[0046] Figure 2 1 is a schematic diagram of a model network of a panoramic image restoration method based on adaptive feature fusion and self-attention mechanism according to an embodiment of the present invention;
[0047] Figure 3 is a network diagram of an adaptive feature extraction module according to an embodiment of the present invention;
[0048] Figure 4 is a network diagram of a context module according to an embodiment of the present invention;
[0049] Figure 5 is a network diagram of potential modules of an embodiment of the present invention;
[0050] Figure 6 2 is a network diagram of a content-focused self-attention module according to an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and technical effect of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0052] like Figure 1 and Figure 2 As shown, this embodiment discloses a panoramic image restoration method based on adaptive feature fusion and self-attention mechanism, comprising the following steps:
[0053] In step 1, the panoramic image is masked and then fed into the Adaptive Feature Fusion Transformer (ACB) module for feature fusion to generate a fused feature map. The ACB consists of three modules: the context module, the latent module, and the adaptive feature extraction module. The specific process includes:
[0054] Step 1.1: Use the mask image to divide the panoramic image features into two independent parts: visible area features F e and the feature F of the area to be repaired g , thereby explicitly separating the context information from the damage information, the expression is as follows:
[0055] F e =F⊙φ(M)
[0056] F g =F⊙φ(1-M)
[0057] F represents the panoramic image feature map, M represents the mask map, ⊙ represents element-by-element matrix multiplication, and φ represents the closest downscaling.
[0058] Step 1.2, as Figure 3 As shown, the visible area feature F e and the feature F of the area to be repaired g Input to the adaptive feature extraction module, pass through the 1*1 convolution layer and the sigmoid layer to obtain q, k, v, add q and k to v after matrix product, and then pass through the fully connected layer and the ReLu layer to output A p2p , the expression is as follows:
[0059]
[0060] Step 1.3, as Figure 4 As shown, the visible area feature F e and the feature F of the area to be repaired g Input to the context module, pass through the 1*1 convolution layer and the sigmoid layer to obtain q, k, v, multiply the q and k matrices and pass through the softmax layer and then multiply with the v matrix to output Rc .
[0061] Step 1.4: Output the context module to R c And the adaptive feature extraction module outputs A p2p Multiply to get F e ', the expression is as follows:
[0062] F e '=A p2p ⊙R c .
[0063] Step 1.5, as Figure 5 As shown, the feature map F e 'Input to the potential module, pass through layer normalization, convolution layer, ReLu layer, convolution layer, ReLu, fully connected layer, convolution layer, ReLu, convolution layer and output R l .
[0064] Step 1.6, output the potential module R l And the adaptive feature extraction module output A p2p Multiply to get R l * , the expression is as follows:
[0065] R l * =(1-A p2p )⊙R l .
[0066] Step 1.7, the output of the adaptive feature fusion Transformer module is F * , the expression is as follows:
[0067] F * =A p2p ⊙R c +(1-A p2p )⊙R l +F.
[0068] Step 2, such as Figure 6 As shown in Figure 1, a content-focused self-attention module (CFA) is added between each adaptive feature fusion Transformer module and the fused feature map is input, where the first CFA module is skip-connected to the last CFA module, the second CFA module is skip-connected to the second-to-last CFA module, and so on. The process of processing the fused feature map by the content-focused self-attention module specifically includes:
[0069] Step 2.1: Input fusion feature map F *Divide into four windows, calculate Q, K and V through 1*1 convolution and 3*3 convolution in sequence, and then divide the feature map into upper and lower channels for attention calculation.
[0070] Step 2.2: The upper channel calculates the channel self-attention in each window of the input multiple feature maps to obtain the feature representation F top , the lower channel calculates the channel self-attention on multiple feature maps of the input without dividing the window, and obtains the feature representation F bottom .
[0071] Step 2.3, add the output results of the upper and lower channels to obtain the output feature representation F out =w top *F top +w bottom *F bottom , w top , w bottom represents a learnable parameter.
[0072] In step three, a joint design of L1 loss, adversarial loss, feature matching loss and high receptive field perception loss is used to optimize model training.
[0073] Specifically, the L1 loss: represent the real image and the repaired image respectively, and ||·||1 represents the L1 norm.
[0074] The adversarial loss is composed of the discriminator loss L D and the generator loss L G In this embodiment, only the features from the masked area are considered as L D For fake samples in , the adversarial loss can be expressed as:
[0075]
[0076] L adv =L D +L G +λ GP L GP ,
[0077] L GP represents the gradient penalty, taking λ GP =1e-3, E[] represents the expected function, D() represents the feature map of the discriminator, Represents the gradient.
[0078] The feature matching loss L FM Based on the L1 loss between the discriminator features of real and fake samples, L FM It is usually used to stabilize training, and it can also slightly improve performance. The expression is as follows:
[0079]
[0080] N represents the number of layers of the discriminator, D (i) () represents the feature map of the i-th layer of the discriminator, M i It is the product of the number of channels, height and width of the feature map of the i-th layer, that is, M i =C i *H i *W i .
[0081] The high receptive field perception loss L hrf The expression is as follows:
[0082]
[0083] where φ hrf Use the pre-trained segmentation network ResNet50 with dilated convolutions.
[0084] Combining the above loss functions, we get the final loss function, which is expressed as follows:
[0085] L final =λ L1 L L1 +λ adv L adv +λ fm L fm +λ hrf L hrf .
[0086] The weights of different loss items are determined by experimental tuning, λ L1 =10,λ adv =10,λ FM =100,λ hrf =30.
[0087] The above examples are not intended to limit the present invention in any form. Although the implementation process of the present invention is described in detail above, those familiar with the art can still modify the technical solutions described in the above examples or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.
Claims
1. A panoramic image restoration method based on adaptive feature fusion and self-attention mechanism, characterized in that: include: A panoramic image restoration model is constructed, which includes an adaptive feature fusion Transformer module and a content-focused self-attention module. The original panoramic image is superimposed with a mask and then input into the panoramic image restoration model. The adaptive feature fusion Transformer module is used to extract and fuse features to obtain a fused feature map. The fusion feature map input content is focused on the self-attention module to perform dual-channel self-attention calculation, and then added to obtain the output feature representation; A joint loss function is used to train and optimize the panoramic image restoration model.
2. The panoramic image restoration method according to claim 1, wherein: Use the mask image to divide the original panoramic image into visible area features F e and the feature F of the area to be repaired g Two independent parts, expressed as follows: F e =F⊙φ(M) F g =F⊙φ(1-M) F represents the panoramic image feature map, M represents the mask map, ⊙ represents element-by-element matrix multiplication, and φ represents the closest downscaling.
3. The panoramic image restoration method according to claim 2, wherein: The adaptive feature fusion Transformer module includes a context module, a potential module and an adaptive feature extraction module.
4. The panoramic image restoration method according to claim 3, wherein: The adaptive feature extraction module inputs the visible area feature F e and the feature F of the area to be repaired g , after passing through the 1*1 convolution layer and the sigmoid layer, we get q, k, and v. We add q and k to v after matrix product, and then output feature A through the fully connected layer and the ReLu layer. p2p , the expression is as follows:
5. The panoramic image restoration method according to claim 4, wherein: The context module inputs the visible area feature F e and the feature F of the area to be repaired g After passing through the 1*1 convolution layer and the sigmoid layer, we get q, k, and v. We multiply the q and k matrices and pass them through the softmax layer and then multiply them with the v matrix to output the feature R. c .
6. The panoramic image restoration method according to claim 5, wherein: The output feature R of the context module c And the output feature A of the adaptive feature extraction module p2p Multiply to get the feature map F e ', the expression is as follows: F e '=A p2p ⊙R c , The latent module inputs the feature map F e ', and then pass through layer normalization, convolution layer, ReLu layer, convolution layer, ReLu, fully connected layer, convolution layer, ReLu, and convolution layer to output feature R l ; The output feature R of the latent module l And the output feature A of the adaptive feature extraction module p2p Multiply to get R l * , the expression is as follows: R l * =(1-A p2p )⊙R l , Finally, the original panoramic image feature F is combined with the output of the adaptive feature fusion Transformer module to obtain the fusion feature F * , the expression is as follows: F * =A p2p ⊙R c +(1-A p2p )⊙R l +F。 7. The panoramic image restoration method according to claim 1, wherein: In the panoramic image restoration model, the content-focused self-attention module is added between each adaptive feature fusion Transformer module, where the first content-focused self-attention module is jump-connected to the last content-focused self-attention module, the second content-focused self-attention module is jump-connected to the penultimate content-focused self-attention module, and so on.
8. The panoramic image restoration method according to claim 7, wherein: The content-focused self-attention module divides the input fusion feature map into four windows, calculates Q, K and V through 1*1 convolution and 3*3 convolution in sequence, and divides the feature map into upper and lower channels for self-attention calculation. Finally, the calculation results are added to obtain the output feature representation.
9. The panoramic image restoration method according to claim 8, wherein: The upper channel of the upper and lower channels calculates the channel self-attention in each window for the input multiple feature maps, and obtains the feature representation F top , the lower channel calculates the channel self-attention on multiple feature maps of the input without dividing the window, and obtains F bottom ; Add the output results of the upper and lower channels to get the output feature representation F out =w top *F top +w bottom *F bottom , w top , w bottom represents a learnable parameter.
10. The panoramic image restoration method according to claim 1, wherein: The joint loss function includes L1 loss, adversarial loss, feature matching loss and high receptive field perception loss. The L1 loss: represent the real image and the repaired image respectively, ||·||1 represents the L1 norm; The adversarial loss is composed of the discriminator loss L D and the generator loss L G Composition, only the features from the masked area are considered as L D For fake samples in , the adversarial loss is expressed as: L adv =L D +L G +λ GP L GP , L GP represents the gradient penalty, taking λ GP =1e-3, E[] represents the expected function, D() represents the feature map of the discriminator, represents the gradient; The feature matching loss L FM Based on the L1 loss between the discriminator features of real and fake samples, L FM For stable training, the expression is as follows: N represents the number of layers of the discriminator, D (i) () represents the feature map of the i-th layer of the discriminator, M i It is the product of the number of channels, height and width of the feature map of the i-th layer, that is, M i =C i *H i *W i ; The high receptive field perception loss L hrf The expression is as follows: where φ hrf Use the pre-trained segmentation network ResNet50 with dilated convolutions; Combining the above loss functions, we get the final loss function, which is expressed as follows: L final =λ L1 L L1 +λ adv L adv +λ fm L fm +λ hrf L hrf ; The weights of different loss items are determined by experimental tuning, λ L1 =10,λ adv =10,λ FM =100,λ hrf =30.