Regional attention mechanism and network based on information interaction of different regions

Abstract

The invention relates to a region attention mechanism based on mutual different region information interaction, and the method comprises the steps: carrying out the re-weighting of the information of a region through a region attention mechanism RA when a window comprises the information of the mutual different region needing to be adjusted; for a region which does not need to be adjusted, a region attention mechanism RA can perform pruning operation, and feature information basic operation of the region is maintained. The invention also relates to a regional attention mechanism network based on information interaction of different regions. According to the invention, the regional attention mechanism satisfies high efficiency by using a sliding window mechanism and a pruning technology; the property of the image is met to achieve flexibility; robustness is satisfied by using a large receptive field and an autonomous selection capability, a neural network model based on the regional attention mechanism has higher information integration and utilization efficiency, tradeoff is achieved in model complexity and model performance, the regional attention mechanism is verified in an image shadow removal task, and the image shadow removal efficiency is improved. The result shows that the RA has good compatibility and performance for information interaction tasks of different areas.

Description

Technical Field

[0001] The present invention belongs to the technical field of information interaction between different regions, and specifically relates to a regional attention mechanism and network based on information interaction between different regions. Background Art

[0002] AN IMAGE IS WORTH 16X16 WORDS:TRANSFORMERS FOR IMAGERECOGNITIONATSCALE [1] The work proposes to use the attention mechanism on the image and introduces the Vision Transformer (ViT) framework. This work mainly explores the direct application of the Transformer model to image recognition tasks to verify the performance of the Transformer on large-scale data. The work found that when pre-trained on large-scale datasets such as ImageNet-21k and JFT-300M, the Vision Transformer can reach or exceed the current state-of-the-art image recognition models. The innovation of this simple but scalable method is that it does not require specific image-aware inductive biases, but relies entirely on the learning ability of the Transformer framework. In addition, this work also conducted some preliminary self-supervised pre-training experiments to demonstrate the potential of ViT in this regard. In summary, this work demonstrates the powerful performance of the Transformer framework on large-scale data and provides new possibilities for applying Transformer to computer vision tasks. First, ViT divides the input image X into image blocks of size P×P and flattens them into vectors X p , and then project these image block vectors to the D-dimensional feature embedding block through a trainable linear mapping. At the same time, a learnable [class] category label is added as the image representation, and then the feature embedding block plus the position encoding is used as the input sequence of the ViT encoder. The ViT encoder is composed of multi-head self-attention (MHSA) layers and multi-layer perceptrons (MLP) alternating, and each layer has layer normalization LayerNorm and residual connection. The output state of the [class] label in the final output of the ViT encoder is the representation y of the image. A classification head is added to y for image classification tasks in both the pre-training and fine-tuning stages. Figure 1 This is a schematic diagram of ViT. Figure 1 On the left is the network structure diagram of the ViT overall model, and on the right is the structural diagram of the ViT encoder.

[0003] In ViT, the multi-head attention mechanism module is a core component, which draws on the Transformer model in natural language processing and applies it to image processing tasks. The multi-head attention mechanism uses multiple attention heads to calculate the attention distribution of different parts of the input sequence, thereby capturing more features and contextual information. Each attention head independently calculates a reduced-dimensional attention, and then the outputs of these attention heads are spliced ​​together and processed through linear transformation. Specifically, the feature embedding blocks that have undergone linear changes are mapped to query (Query, Q), key (Key, K), and value (Value, V) matrices respectively. Let the feature embedding block be X, then Q = XW Q ,K=XW K ,V=XW V , where XW, XW, XW are learnable weight matrices. Each attention head independently calculates the dot product of the query and the key to get the attention score, and then converts the score into an attention weight through scaling and softmax function. where d k is the dimension of the key. Finally, the input sequence is divided into h parts, and the attention score is calculated for each part MultiHead(Q,K,V)=Concat(head1,head2,…,head h )W O , where each head i Calculation method head i =Attention(Q i ,K i ,V i ), where W o is the output projection matrix. Through the multi-head attention mechanism, ViT can interact with the feature embedding blocks of different segmented image blocks and finally process the category information of the entire image.

[0004] Existing attention mechanism work: AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FORIMAGE RECOGNITION AT SCALE [1]A method is proposed to directly apply the Transformer model commonly used in natural language processing to image recognition tasks. By dividing the input image into fixed-size image blocks and embedding these image blocks into fixed-length vector sequences, the multi-head attention mechanism is used to model the relationship between image blocks, thereby achieving image classification and feature extraction. The mechanism proposed in this work can capture long-distance dependencies and is suitable for a variety of computer vision tasks. However, the disadvantage of ViT is that it has a high demand for large-scale data and computing resources, the training process is relatively complicated, and the complexity of the global multi-head attention mechanism is very high.

[0005] Working on: Swin Transformer: Hierarchical Vision Transformer using ShiftedWindows [2] A new visual Transformer model, Swin Transformer, is proposed as a general backbone network for computer vision tasks. Swin Transformer adopts a layered design and a moving window method, which can efficiently calculate self-attention within a local window and also realize cross-window connections. Experimental results show that Swin Transformer has achieved advanced performance in tasks such as image classification, object detection, and semantic segmentation. Compared with previous visual Transformers, Swin Transformer has lower computational complexity and is more suitable as a general visual backbone network, such as Figure 2 However, when designing the moving window attention, Swin Transformer uses the same key for attention mechanism calculation for queries in the same window, ignoring the impact of key position information on the task, and lacks certain flexibility.

[0006] Existing shadow removal models: Working ShadowFormer:Global Context Helps Image ShadowRemoval [3] A Retinex-based [4] The Transformer network with a multi-scale channel attention framework is designed to solve the shadow removal problem. It proposes a shadow interaction module SIM, such as Figure 3As shown. The shadow interaction attention mechanism SIA uses the global context information of the non-shadow area to help restore the shadow area. The motivation for doing this is that shadow removal is a locally damaged task, and the existing local attention mechanism will be limited, while SIM can make better use of global attention information. However, the method of re-weighting the attention map using the similarity map coefficients of shadows and non-shadows in SIA introduced in this work is not completely suitable because it does not fully consider the role of shadow information in the restoration of shadow areas. In addition, the attention mechanism of this work is still based on the traditional window attention mechanism. Although the technical design of SIA can alleviate the limitations of the window attention mechanism, it still has great limitations.

[0007] Working ShadowDiffusion:When Degradation PriorMeetsDiffusion ModelforShadowRemoval [6] A new shadow removal framework ShadowDiffusion based on diffusion is proposed. Figure 4 As shown. In order to better capture the complex lighting degradation and shadow feature distribution of the real world, this work uses a novel shadow model with reference to spatial variations. ShadowDiffusion models shadow removal as a task of jointly estimating a shadow-free image and fine-tuning a shadow mask, where mask fine-tuning serves as an auxiliary task of the diffusion generator to help estimate the shadow-free image. This work also proposes a novel diffusion sampling strategy that explicitly integrates image degradation priors into the diffusion framework. However, the computational cost of this type of network framework based on the diffusion model is very high. The model has a large number of parameters and high inference time complexity. It requires sufficient computing resources to support it and cannot cope with situations where computing resources are limited. Due to its huge time and space complexity, the model is difficult to quickly assist various downstream tasks.

[0008] References

[0009] [1]Alexey Dosovitskiy,Lucas Beyer,Alexander Kolesnikov,DirkWeissenborn,Xiaohua Zhai,Thomas Unterthiner,Mostafa Dehghani,MatthiasMinderer,Georg Heigold,Sylvain Gelly,Jakob Uszkoreit,&Neil Houlsby(2021).AnImage is Worth16x16 Words:Transformers for Image Recognition at Scale.InICLR.OpenReview.net.

[0010] [2]Ze Liu,Yutong Lin,Yue Cao,Han Hu,Yixuan Wei,Zheng Zhang,StephenLin,&Baining Guo(2021).Swin Transformer:Hierarchical Vision Transformer usingShiftedWindows.In ICCV(pp.9992–10002).IEEE.

[0011] [3]Lanqing Guo,Siyu Huang,Ding Liu,Hao Cheng,&Bihan Wen(2023).ShadowFormer:Global Context Helps Shadow Removal.In AAAI(pp.710–718).AAAIPress.

[0012] [4]Land,E.H.1977.The retinex theory of color vision.Scientificamerican,237(6):108–129.

[0013] [5]Yurui Zhu,Jie Huang,Xueyang Fu,Feng Zhao,Qibin Sun,&Zheng-Jun Zha(2022).Bijective Mapping Network for Shadow Removal.In CVPR(pp.5617–5626).IEEE.

[0014] [6]Lanqing Guo,Chong Wang,Wenhan Yang,Siyu Huang,Yufei Wang,HanspeterPfister,&Bihan Wen(2023).ShadowDiffusion:When Degradation Prior MeetsDiffusion Model for Shadow Removal.In CVPR(pp.14049–14058).IEEE.

[0015] [7]Alexey Dosovitskiy,Lucas Beyer,Alexander Kolesnikov,DirkWeissenborn,Xiaohua Zhai,Thomas Unterthiner,Mostafa Dehghani,MatthiasMinderer,Georg Heigold,Sylvain Gelly,Jakob Uszkoreit,&Neil Houlsby(2021).AnImage is Worth16x16 Words:Transformers for Image Recognition at Scale.InICLR.OpenReview.net.

[0016] [8]Jingyun Liang,Jiezhang Cao,Guolei Sun,Kai Zhang,Luc Van Gool,&RaduTimofte(2021).SwinIR:Image Restoration Using Swin Transformer.In ICCVW(pp.1833–1844).IEEE.

[0017] [9]Zhendong Wang,Xiaodong Cun,Jianmin Bao,Wengang Zhou,JianzhuangLiu,&Houqiang Li(2022).Uformer:A General U-Shaped Transformer for ImageRestoration.In CVPR(pp.17662–17672).IEEE.

[0018]

[10] Wei-Sheng Lai,Jia-Bin Huang,Narendra Ahuja,&Ming-Hsuan Yang(2019).Fast and Accurate Image Super-Resolution with Deep Laplacian PyramidNetworks.IEEE Trans.PatternAnal.Mach.Intell.,41(11),2599–2613.

[0019]

[11] Hieu M.Le,&Dimitris Samaras(2019).Shadow Removal via Shadow ImageDecomposition.In ICCV(pp.8577–8586).IEEE.

[0020]

[12] Liangqiong Qu,Jiandong Tian,Shengfeng He,Yandong Tang,&RynsonW.H.Lau(2017).DeshadowNet:AMulti-context Embedding Deep Network for ShadowRemoval.In CVPR(pp.2308–2316).IEEE Computer Society.

[0021]

[13] Xiaodong Cun,Chi-Man Pun,&Cheng Shi(2020).Towards Ghost-FreeShadow Removal via Dual Hierarchical Aggregation Network and Shadow MattingGAN.In AAAI(pp.10680–10687).AAAIPress.

[0022]

[14] Xiaowei Hu,Chi-Wing Fu,Lei Zhu,Jing Qin,&Pheng-Ann Heng(2020).Direction-Aware Spatial Context Features for Shadow Detection andRemoval.IEEE Trans.PatternAnal.Mach.Intell.,42(11),2795–2808.

[0023]

[15] Yeying Jin, Aashish Sharma, & Robby T.Tan(2021).DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-ClassifierGuidedNetwork.In ICCV(pp.5007–5016).IEEE.

[0024]

[16] Lan Fu, Changqing Zhou, Qing Guo, Felix Juefei-Xu, Hongkai Yu, Wei Feng, Yang Liu, & Song Wang (2021). Auto-Exposure Fusion for Single-Image Shadow Removal. In CVPR (pp. 10571–10580). Computer Vision Foundation / IEEE.

[0025]

[17] Yurui Zhu, Zeyu Xiao, Yanchi Fang, Xueyang Fu, Zhiwei Xiong, & Zheng-Jun Zha (2022). Efficient Model-Driven Network for Shadow Removal. In AAAI (pp. 3635–3643). AAAIPress.

[0026]

[18] Yeying Jin, Wei Ye, Wenhan Yang, Yuan Yuan, & Robby T.Tan (2024).DeS3:Adaptive Attention-Driven Self and Soft Shadow Removal Using ViTSimilarity.In AAAI(pp.2634–2642).AAAIPress.

[0027]

[19] Liu, Z., Yin, H., Wu, X., Wu, Z., Mi, Y., & Wang, S. (2021). From ShadowGeneration to Shadow Removal. In CVPR. Summary of the invention

[0028] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a regional attention mechanism and network based on the interaction of different regional information. The proposed regional attention mechanism RA (RegionalAttention, RA) can enable regional information of different properties to interact quickly and effectively. The regional attention mechanism RA adopts sliding window technology and pruning strategy to achieve efficient operation and solve the high complexity brought by traditional multi-head self-attention. At the same time, RA is designed to adapt to the inherent properties of the image, so that each query has a unique key closest to the query, which has better flexibility than traditional window attention. Its robustness is guaranteed by wide-range perception and enhanced adaptive selection characteristics. The neural network model designed with the help of this regional attention mechanism shows excellence in information fusion and application performance.

[0029] The present invention solves the technical problem by the following technical solutions:

[0030] A regional attention mechanism RA based on the interaction of different regional information. When a window contains different regional information that needs to be adjusted, the regional attention mechanism RA will reweight the information of the region; when facing a region that does not need to be adjusted, the regional attention mechanism RA will perform a pruning operation to maintain the basic operation of the feature information of the region while skipping the complex attention mechanism operation;

[0031] The regional attention mechanism RA is given a feature map Y∈R normalized by the layer C×H′×W′ , and reshape it into X∈R n×d , where n = H'×W', d = C; through linear projection X is transformed into query Q, key K and value V, as well as relative position deviation B(i,j), the attention weight of the i-th input in the region of size k is Defined as the query projection Q of the i-th input i The dot product with the key projection K of k elements in the surrounding area:

[0032]

[0033] Where: j (i) represents the jth element in the neighborhood of input i; the definition value is a k-row matrix whose content is the projection V of the region elements in the neighborhood of input i:

[0034]

[0035] The regional attention result of the i-th element of region size k is defined as:

[0036]

[0037] The regional attention mechanism RA promotes the feature map Y∈R C×H′×W′ Each feature element containing binary region mask information autonomously selects information from different surrounding regions to interact with, thereby achieving the purpose of regional information interaction. If the region involved in the window does not need to be adjusted, the regional attention mechanism RA will perform a pruning operation and skip the tedious attention calculation.

[0038] A regional attention mechanism network RASM based on information interaction of different regions, adopting the regional attention mechanism RA to remove image shadows, completing information interaction between image shadow regions and surrounding non-shadow regions; taking shadow image I and shadow mask M as model input, using spatial and channel context information of shadow regions and adjacent non-shadow regions with regional attention modules in the deep layer of the network, making shadow regions and surrounding non-shadow regions interact effectively through a given binary region mask, and performing shadow removal; the regional attention mechanism network RASM includes an image information input module, an encoder-decoder framework and a regional attention mechanism module, specifically:

[0039] First, the shadow image I s ∈R 3×H×W And the shadow mask I m ∈R H×W As input, where H, W are the length and width of the image, and the shadow mask I m is a single-channel binary image of the same size as the shadow image, where if the pixel at image coordinate (i, j) belongs to the shadow area, the shadow mask corresponds to the value I at the coordinate m The value of (i,j) is 1, otherwise it is 0, indicating two different regional information; in the initial stage of the network, a linear projection layer is applied to project the network input to generate the primary feature embedding X0∈R C ×H×W , where C represents the feature projection dimension;

[0040] Subsequently, X0 is processed through several encoder-decoder frameworks, each of which consists of two channel attention modules CA, both of which are connected to the scaling layer. The two channel attention modules CA are divided into an upsampling layer and a downsampling layer, which respectively reduce and enlarge the feature map. The channel attention module CA compresses the spatial information and then uses a feedforward multilayer perceptron MLP to capture long-distance correlations. After passing through L modules in the encoder, the hierarchical features {X1, X2, ..., X L},in At the network bottleneck stage, according to the set feature X LThe regional attention mechanism module calculates the correlation of information of different regions. The sub-blocks of the regional attention mechanism module perform attention mechanism operations by aggregating information of non-shadow areas around shadows, interacting with different regional information to adjust image feature information, so that each shadow position can more effectively utilize regional attention information across spatial and channel dimensions;

[0041] Next, the features of each channel attention module CA in the input decoder are concatenated with the features of the upsampling layer and the features of the corresponding channel dimension obtained by the encoder through skip links, and finally the resulting image after shadow removal is output.

[0042] Moreover, the training of the regional attention mechanism network RASM adopts content loss and perceptual loss. The content loss ensures the consistency between the output image and the real image training data. In the image field, the Charbonnier Loss algorithm is adopted, and the content loss is expressed as:

[0043]

[0044] in: is the output image, I gt is a real shadow-free image, ∈ is set to 10 -6 To ensure numerical stability;

[0045] Perceptual loss is used to preserve the high-level features and semantic information of the image while minimizing the difference between the restored image and the real image, and minimizing the output image in the {conv1_2,conv2_2,conv3_2,conv4_2,conv5_2} layer of the pre-trained VGG-19 model. I of features and real data lit The l1 difference of the features, denoting the i-th feature extractor as Ψ i (·), the perceptual loss is expressed as follows:

[0046]

[0047] Where: w i is the weight between different layers, and its value is set to {0.1,0.1,1,1,1};

[0048] The total loss function is:

[0049]

[0050] Among them: α1, α2 = {0.001, 1}, RASM uses the loss function L for end-to-end supervised training.

[0051] The positive effects that the present invention can produce are:

[0052] 1. The regional attention mechanism of the present invention can be used as a solution for information interaction tasks based on different regions. The regional attention mechanism can effectively promote efficient and flexible information interaction between different regions. Taking image shadow removal as an example, the image shadow model RASM based on regional attention of the present invention obtains almost all the best numerical comparison results in the widely used shadow removal dataset.

[0053] 2. The regional attention mechanism of the present invention adopts the form of a sliding window, in which the query after linear mapping is located at the center of the sliding window, and the sources of keys and values ​​are the remaining feature embedding blocks covered by the sliding window. The query at the center of the window and the surrounding keys and values ​​calculate the attention weights and re-weight them to complete the attention calculation, so that when calculating the regional attention, each query can interact with the surrounding nearest keys and values, and the keys and values ​​interacted by different queries are different. At the same time, the calculation process is carried out in the sliding window, thereby promoting more flexible, efficient and accurate information interaction.

[0054] 3. The regional attention mechanism RA of the present invention utilizes a sliding window mechanism and pruning technology to meet the requirements of high efficiency; meets the requirements of the image itself to achieve flexibility; and meets the requirements of robustness by utilizing a large receptive field and autonomous selection capabilities. The neural network model designed based on the regional attention mechanism has higher information integration and utilization efficiency. During training, a lightweight network can be designed to achieve superior performance, and a trade-off is achieved between model complexity and model performance. The regional attention mechanism is verified in the image shadow removal task. The results show that RA has good compatibility and performance for tasks in which information from different regions is interacted with. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the ViT network structure;

[0056] Figure 2 This is a visualization diagram of the Swin Transformer;

[0057] Figure 3 This is a schematic diagram of the ShadowFormer network structure;

[0058] Figure 4 This is a schematic diagram of the ShadowDiffusion network structure;

[0059] Figure 5 A visualization diagram of the attention mechanism of the present invention and the existing attention mechanism;

[0060] Figure 6 This is a graph showing the result of image shadow removal performed by the RASM network of the present invention;

[0061] Figure 7 It is a schematic diagram of the RASM network structure of the present invention;

[0062] Figure 8 Schematic diagram of the regional attention mechanism and the regional attention mechanism with expansion factor of the present invention;

[0063] Fig. 9 A comparison diagram of the visual effects of shadow removal of different network models of the present invention on the AISTD dataset;

[0064] Fig.10 A comparison diagram of the visual effects of shadow removal of different network models of the present invention on the SRD dataset;

[0065] Fig.11 Schematic diagram of weight allocation of regional attention mechanism of the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0067] In computer vision tasks, Transformer is a general visual framework. Its superior ability to capture long-distance dependency modeling makes the models derived from this framework have good performance. Models that use traditional self-attention mechanisms (SelfAttention) on visual tasks, represented by ViT, usually divide the image into several image blocks, and then perform feature mapping calculations on the image blocks through linear mapping. The feature embedding blocks of the image blocks are calculated to obtain queries, keys, and values. The self-attention mechanism calculates the similarity between each query and all other keys, obtains the attention map after softmax function and scaling, and adjusts the value of the feature after reweighting the value with the attention map, thereby effectively modeling long-distance dependencies. Swin Transformer is an improved Transformer model that achieves efficient image processing by introducing a hierarchical structure and a sliding window mechanism. The working principle of Swin Transformer includes two key parts: window attention (Window SelfAttention) and shift window attention (Shift Window SelfAttention). Window attention divides the image into non-overlapping fixed-size windows, and independently divides the image blocks in each window. The feature embedding blocks of the image blocks are linearly mapped to obtain queries, keys, and values. The query in the window only calculates the similarity with the key in the window, and finally obtains the attention map and the value in the window weighted to calculate the updated feature value, thereby reducing the computational complexity. Shifting window attention achieves cross-window information interaction by translating the window and enhancing the feature extraction capability. This mechanism not only retains the advantages of the global attention mechanism, but also significantly reduces the demand for computing resources and improves the performance and efficiency of the model.

[0068] The invention discloses a regional attention mechanism based on information interaction of different regions. The innovation of the invention is that each query will pay attention to the information of the neighboring regions, strengthen the process of information interaction with the surrounding different regions, and is different from the self-attention mechanism and the window attention mechanism. The self-attention mechanism projects the information of the whole image to obtain the query, key and value to perform the attention mechanism operation, but the self-attention mechanism over-relies on the global information and performs the attention operation on the information of all regions, which brings huge computational overhead; the window attention is to project the query, key and value in a specific window to perform the attention mechanism operation, but the information required for processing queries at different positions may be very different in position, and the practice of sharing keys and values ​​in the same window makes it limited to the window during information interaction, which lacks flexibility. Different from the attention mechanism used in previous works, the regional attention mechanism proposed in the invention adopts the form of a sliding window, in which the query after linear mapping is located at the center of the sliding window, and the source of the key and value is the remaining feature embedding blocks covered by the sliding window. The query at the center of the window and the key and value of the surrounding area calculate the attention weight and re-weight to complete the attention calculation. This allows each query to interact with the nearest keys and values ​​when calculating regional attention, and the keys and values ​​interacted with by different queries are different. At the same time, the calculation process is performed in a sliding window, thereby promoting more flexible, efficient and accurate information interaction. Figure 5 A visual diagram showing the self-attention mechanism SA, window attention mechanism WSA, shifted window attention mechanism SWSA and regional attention mechanism RA, where the highlighted squares represent the projected query elements, and the highlighted areas represent the element areas where the keys and values ​​are projected.

[0069] The regional attention mechanism RA based on information interaction of the present invention, specifically, given a feature map Y∈R normalized by layer C×H′×W′ , and reshape it into X∈R n×d , where n = H'×W', d = C. Through linear projection, X is transformed into query Q, key K and value V, as well as relative position deviation B(i,j). The attention weight of the i-th input in the region of size k is Defined as the query projection Q of the i-th input i The dot product with the key projection K of k elements in the surrounding area:

[0070]

[0071] where ρ j (i) represents the jth element in the neighborhood of input i. Then define the value is a k-row matrix whose content is the projection V of the region elements in the neighborhood of input i:

[0072]

[0073] The regional attention result of the i-th element of region size k is defined as:

[0074]

[0075] When the feature map contains binary region mask information, after learning the neural network model, the regional attention mechanism will prompt each feature element to autonomously select the information of different surrounding regions to interact, thereby achieving the purpose of regional information interaction. If the area involved in the window does not need to be adjusted, RA will perform pruning operations, skipping the tedious attention calculations to ensure the efficiency of RA.

[0076] The regional attention mechanism proposed in the present invention can handle computer vision tasks such as image shadow removal that require information interaction between different regions. Under normal environmental conditions, determining the specific location of the shadow mainly depends on the information differences in brightness, color, texture, etc. between the shadow area and its adjacent non-shadow area. The purpose of shadow removal is to make the reconstructed image similar to the surrounding non-shadow areas, so the information of the non-shadow area that is closer to the shadow area is more important. Therefore, analyzing the features of these areas can not only accurately identify the boundaries and positions of the shadows, but also provide necessary reference data for subsequent shadow removal work. The regional attention technology can well meet this requirement, allowing the shadow area to interact with the surrounding non-shadow area information. The present invention designs a regional attention-based network RASM, and the network structure is as follows: Figure 7 As shown in Figure 1, the shadow image I and the shadow mask M are used as model inputs. In the deep layer of the network, a module with regional attention is used to exchange the spatial and channel context information of the shadow area and the adjacent non-shadow area, so as to remove the shadow area. The effect of using the regional attention mechanism for shadow removal is shown in Figure 1. Figure 6 As shown, the given binary area mask RA can make the shadow area and the surrounding non-shadow area interact effectively to remove the shadow;

[0077] The present invention proposes a regional attention mechanism that can be applied to most tasks that require information interaction between different regions, such as image shadow removal. The present invention provides a regional attention mechanism network RASM based on information interaction between different regions. The model is described in detail below. First, the shadow image I s ∈R 3×H×W And the shadow mask I m ∈R H×WAs input, where H, W are the length and width of the image, and the shadow mask I m is a single-channel binary image of the same size as the shadow image, where if the pixel at image coordinate (i, j) belongs to the shadow area, the shadow mask corresponds to the value I at the coordinate m The value of (i,j) is 1, otherwise it is 0, indicating two different regions of information. In the initial stage of the network, a linear projection layer is applied to project the network input to generate the primary feature embedding X0∈R C×H×W , where C represents the feature projection dimension. Subsequently, X0 is processed by an encoder-decoder framework, each of which consists of a channel attention (CA) module to integrate multi-scale global features.

[0078] like Figure 7 As shown in Figure 1, each CA module has two CA sub-blocks, followed by a scaling layer. Specifically, there is a downsampling layer in the encoder CA module and an upsampling layer in the decoder CA module, which shrink and enlarge the feature map respectively. The function of the CA module is to compress spatial information through CA sub-blocks and then use a feedforward multi-layer perceptron MLP to capture long-range correlations. After L modules in the encoder, the hierarchical features {X1, X2, ..., X L},in At the network bottleneck stage, according to the set feature X L The correlation of information of different regions is calculated through the regional attention RA module. In the RA module, the RA sub-block performs attention mechanism operation by aggregating the information of non-shadow areas around the shadow, interacting with the information of different regions to adjust the image feature information, so that each shadow position can more effectively utilize the regional attention information across spatial and channel dimensions. Next, the features of each CA module in the input decoder are concatenated with the upsampled features and the features on the corresponding channel dimension obtained by the encoder through jump links, and finally the result image after shadow removal is output.

[0079] Shadows are usually not isolated, they are closely connected to the surrounding environment. A larger receptive field allows the model to more fully understand the relationship between shadow areas and non-shadow areas in the image. If the receptive field is too small, the model may only see part of the shadow or the object blocked by the shadow, which may lead to incorrect shadow perception and removal. A larger receptive field allows the model to observe the entire shadow area and its transition edges, so as to remove the shadow more accurately. In the process of removing shadows, it is crucial to maintain the naturalness and coherence of the image. A larger receptive field helps the model maintain the consistency and natural transition of the surrounding environment while removing shadows, avoiding unnatural patches or color differences in the processed image. Similarly, regional attention can also benefit from a larger receptive field because the markers calculated in this area can obtain more information. In order to balance the complexity and performance of the model, the present invention proposes a regional attention mechanism with a dilation factor. Specifically, by increasing the stride when selecting a region, the receptive field is expanded to a larger range. Using the regional attention mechanism with a dilation factor, the receptive field of regional attention can be expanded without increasing the complexity of the model, thereby further improving the performance. Figure 8 The regional attention mechanism with no dilation factor and the regional attention mechanism with dilation factor 2 are demonstrated.

[0080] Two loss terms are used when training RASM: content loss and perceptual loss. These two loss terms are explained in detail below.

[0081] Content loss ensures consistency between the output image and the real image training data. In the image field, Charbonnier Loss is used. [8] Algorithm. The content loss is expressed as:

[0082]

[0083] in, is the output image, I gt is a real shadow-free image. ∈ is set to 10 -6 to ensure numerical stability.

[0084] Perceptual loss has been widely used in various image restoration and generation tasks to preserve the high-level features and semantic information of the image while minimizing the difference between the restored image and the real image. Minimize the output image in the {conv1_2,conv2_2,conv3_2,conv4_2,conv5_2} layer of the pre-trained VGG-19 model I of features and real data lit The l1 difference of the feature. Denote the i-th feature extractor as Ψ i (·), the perceptual loss can be expressed as follows:

[0085]

[0086] Among them, w i is the weight between different layers, and its value is set to {0.1, 0.1, 1, 1, 1}.

[0087] The total loss function is:

[0088]

[0089] where α1, α2 = {0.001, 1}. RASM is trained end-to-end with supervision using the loss function L and achieves state-of-the-art results on shadow removal.

[0090] The regional attention mechanism proposed in this invention can be used as a solution for tasks based on information exchange between different regions. The regional attention mechanism can effectively promote efficient and flexible information exchange between different regions. Taking image shadow removal as an example, the image shadow model RASM based on regional attention invented in this invention has achieved almost all the best numerical comparison results in the widely used shadow removal datasets. It is comparable to previous work DHAN on two widely used shadow removal datasets.

[13] ,DSC

[14] 、G2R

[19] 、DC-ShadowNet

[15] 、AutoExposure

[16] BMNet [5] Unfolding

[15] 、ShadowFormer [3] 、ShadowDiffusion [6] 、DeS3

[18] For comparison, the root mean square error (RMSE) between the output image and the true shadow-free image in the CIE LAB color space is used as a quantitative indicator. In order to make the comparison more comprehensive, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) in the RGB space are measured and reported. The model operation volume FLOPs is tested and evaluated on a 256×256 image. First, in AISTD

[11] The results are shown in Table 1. The best results are marked in bold, the second best results are marked with underscores, and "-" indicates that this part of the data is not public. In the AISTD dataset, RASM surpasses existing methods and achieves the best performance in the comparison of PSNR and RMSE indicators in shadow areas, complete images, and non-shadow areas. In non-shadow areas, RASM's SSIM is basically equivalent to the best reported results.

[0091] Table 1. Comparison of numerical results of shadow removal of different network models on the AISTD dataset

[0092]

[0093] In SRD

[12] The results are shown in Table 2. In shadow regions, RASM outperforms the state-of-the-art in all metrics. In non-shadow regions, RASM performs on par with ShadowDiffusion, with the slight performance loss likely due to the region attention mechanism following the guidance of the binary region mask more strictly. It is worth noting that despite the inaccurate shadow mask, RASM is still the best among competitors in terms of root mean square error (RMSE) across all images.

[0094] Table 2. Numerical comparison results of shadow removal of different network models on the SRD dataset

[0095]

[0096] In order to verify the high efficiency of RASM, FLOPs and the number of parameters are compared. As shown in Table 3, RASM has a small number of parameters, low FLOPs, and only occupies very little computing resources, but achieves excellent performance, which shows that RASM effectively balances model complexity and model performance. It also reflects that the efficient and flexible regional information interaction capabilities of the regional attention mechanism allow simple network designs to achieve excellent performance.

[0097] Table 3. Comparison of different model parameters, computational complexity, and performance

[0098]

[0099] Fig. 9 and Fig.10 The visual comparison of the output results of different network models in the two datasets AISTD and SRD is shown, where the input is the shadow image input and GT is the real shadow-free image. RASM achieves state-of-the-art performance with fewer residual shadow components and no visual artifacts. In addition, the total parameters of RASM are significantly less than most previous works, which proves the effectiveness of the regional attention mechanism design proposed in this invention.

[0100] In order to verify whether the regional attention mechanism can really enable information from different regions to interact, the present invention performs a visual analysis, selects several points on the image, and visualizes their attention weight distribution, such as Fig.11As shown in the figure, the star mark is the selected point, and the yellow box centered on the star mark is the regional attention range of the point. The red heat map color in the box indicates that the attention weight of the corresponding position is from light to heavy from light to dark. It can be seen that in the fully illuminated area or shadow area, the attention weights of these points are relatively low and evenly distributed. These areas do not need to be adjusted. When performing the regional attention mechanism, the complex attention calculation will be skipped through pruning technology; when the attention point is located in the shadow area and its regional attention range includes the surrounding non-shadow area, the attention weight allocated to the non-shadow area is much larger. In addition, points at different shadow positions will pay attention to non-shadow areas at different positions, which corresponds to the fact that the information of each shadow area interacts with the information of the adjacent non-shadow area, which is consistent with the proposed region-based attention mechanism.

[0101] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A regional attention mechanism based on information interaction between different regions, characterized by: When the window contains different regional information that needs to be adjusted, the regional attention mechanism RA will reweight the information of the region; when facing the region that does not need to be adjusted, the regional attention mechanism RA will perform a pruning operation to maintain the basic operation of the feature information of the region while skipping the complex attention mechanism operation; The regional attention mechanism RA is given a feature map Y∈R normalized by the layer C×H’×W’ , and reshape it into X∈R n×d , where n = H'×W', D = C; through linear projection X is transformed into query Q, key K and value V, as well as relative position deviation B(i,j), the attention weight of the i-th input in the region of size k is Defined as the query projection Q of the i-th input i The dot product with the key projection K of k elements in the surrounding area: Where: j (i) represents the jth element in the neighborhood of input i; the definition value is a k-row matrix whose content is the projection V of the region elements in the neighborhood of input i: The regional attention result of the i-th element of region size k is defined as: The regional attention mechanism RA promotes the feature map Y∈R C×H’×W’ Each feature element containing binary region mask information autonomously selects information from different surrounding regions to interact with, thereby achieving the purpose of regional information interaction. If the region involved in the window does not need to be adjusted, the regional attention mechanism RA will perform a pruning operation and skip the tedious attention calculation.

2. A regional attention mechanism network based on information interaction between different regions, characterized by: The regional attention mechanism RA described in claim 1 is used to remove image shadows, and the information interaction between the image shadow area and the surrounding non-shadow areas is completed; the shadow image I and the shadow mask M are used as model inputs, and the spatial and channel context information of the shadow area and the adjacent non-shadow area are interacted with by the regional attention module in the deep layer of the network, and the shadow area and the surrounding non-shadow area are effectively interacted through a given binary area mask to remove the shadow; the regional attention mechanism network RASM includes an image information input module, an encoder-decoder framework and a regional attention mechanism module, specifically: First, the shadow image I s ∈R 3×H×W And the shadow mask I m ∈R H×W As input, where H, W are the length and width of the image, and the shadow mask I m is a single-channel binary image of the same size as the shadow image, where if the pixel at image coordinate (i, j) belongs to the shadow area, the shadow mask corresponds to the value I at the coordinate m The value of (i,j) is 1, otherwise it is 0, indicating two different regional information; in the initial stage of the network, a linear projection layer is applied to project the network input to generate the primary feature embedding X0∈R C×H×W , where C represents the feature projection dimension; Subsequently, X0 is processed through several encoder-decoder frameworks, each of which consists of two channel attention modules CA, both of which are connected to the scaling layer. The two channel attention modules CA are divided into an upsampling layer and a downsampling layer, which respectively reduce and enlarge the feature map. The channel attention module CA compresses the spatial information and then uses a feedforward multilayer perceptron MLP to capture long-distance correlations. After passing through L modules in the encoder, the hierarchical features {X1, X2, ..., X L },in At the network bottleneck stage, according to the set feature X L The regional attention mechanism module calculates the correlation of information of different regions. The sub-blocks of the regional attention mechanism module perform attention mechanism operations by aggregating information of non-shadow areas around shadows, interacting with different regional information to adjust image feature information, so that each shadow position can more effectively utilize regional attention information across spatial and channel dimensions; Next, the features of each channel attention module CA in the input decoder are concatenated with the features of the upsampling layer and the features of the corresponding channel dimension obtained by the encoder through skip links, and finally the resulting image after shadow removal is output.

3. The regional attention mechanism network RASM based on mutual information interaction of different regions according to claim 1, characterized in that: The training of the regional attention mechanism network RASM adopts content loss and perceptual loss. The content loss ensures the consistency between the output image and the real image training data. In the image field, the Charbonnier Loss algorithm is used, and the content loss is expressed as: in: is the output image, I gt is a real shadow-free image, ∈ is set to 10 -6 To ensure numerical stability; Perceptual loss is used to preserve the high-level features and semantic information of the image while minimizing the difference between the restored image and the real image, and minimizing the output image in the {conv1_2,conv2_2,conv3_2,conv4_2,conv5_2} layer of the pre-trained VGG-19 model. The ll1 difference between the feature and the Ilit feature of the real data, the i-th feature extractor is represented as Ψ i (·), the perceptual loss is expressed as follows: Where: w i is the weight between different layers, and its value is set to {0.1,0.1,1,1,1}; The total loss function is: Among them: α1, α2 = {0.001, 1}, RASM uses the loss function L for end-to-end supervised training.

Images