Multi-scale encoding and decoding essential image decomposition network, method and application based on self-attention

Abstract

The present invention relates to a self-attention multi-scale encoding and decoding intrinsic image decomposition network, method and application, and belongs to the field of image recognition technology. The network uses a parallel encoding and decoding strategy, that is, two sets of identical but independent encoders and decoders are used to generate images of reflectance maps and illumination maps. Each network includes three parts: an encoder, a decoder and a skip connection. The encoder and decoder are both composed of multiple self-attention modules and downsampling or upsampling modules. The adaptive window self-attention proposed in this method can adaptively adjust the window according to the image content to strengthen the network's attention to different types of features and reduce the resource usage of the model; the channel enhancement self-attention module enhances the effective information in the feature map by expanding the self-attention calculation between different dimensions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and relates to a network and method for completing essential image decomposition for a single image using two self-attention modules through a multi-scale encoding and decoding network. Background Art

[0002] With the rapid development of computer technology, significant progress has been made in the field of image processing related to computer vision. Image feature extraction is a fundamental task in image processing. The color, texture, and other information displayed in an image are often dependent on the lighting environment. This means that the color and texture of a scene will appear differently as the lighting changes. These variations often pose significant challenges to feature extraction algorithms and significantly impact image analysis and understanding.

[0003] Computer vision aims to recognize and understand the content in images and videos. Today, over 70% of data on the internet is images or videos, and the number of surveillance cameras worldwide exceeds the population, generating over 800 million hours of surveillance video data daily. This massive amount of data urgently requires automated visual understanding and analysis technologies, and a key step is the ability to extract features that are invariant to the environment. Because information such as color and texture displayed in an image is often dependent on the lighting environment, the color and texture of a scene will appear differently in the image as the lighting changes. By analyzing the inherent color and texture of the target, and making the proposed features robust to lighting, it is possible to effectively identify the same target in different scenes. This analysis method is known as essential image decomposition.

[0004] Based on different algorithm design approaches, current methods for essential feature analysis can be divided into three categories: those based on implicit constraints, those based on explicit constraints, and those based on deep learning. The design philosophy of implicit constraint-based essential feature analysis methods is to assume the invariance of image essential features. Without considering inherent connections, they directly perform pattern analysis on a set of images of the same scene under the influence of different environmental factors, thereby estimating the spatial distribution of the scene's essential features. Most methods based on explicit constraints employ prior constraints and solve optimization problems within the constraint domain. Their algorithmic performance is determined by the rationality of the prior constraints and the convergence of the optimization function. Deep learning-based methods are essentially also based on implicit constraints, but due to the development of deep learning in depth and breadth, they perform the best among all essential image decomposition algorithms and are currently the mainstream algorithms. Summary of the Invention

[0005] The technical problems to be solved by the present invention are:

[0006] In view of the fact that existing deep networks lack the ability to accurately extract global detail information, in order to avoid the shortcomings of the existing technology, the present invention provides an essential image decomposition network and method implemented by a multi-scale encoding and decoding Transformer network.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A self-attention-based multi-scale encoding and decoding intrinsic image decomposition network is characterized in that the network uses a parallel encoding and decoding strategy, that is, using two sets of identical but independent encoders and decoders to generate images of reflectance maps or illumination maps.

[0009] A further technical solution of the present invention is that there is a jump connection between the encoder and the decoder.

[0010] A further technical solution of the present invention is: the encoder and decoder are both composed of multiple self-attention modules and downsampling or upsampling modules.

[0011] A multi-scale encoding and decoding essential image decomposition method based on self-attention is characterized by the following steps:

[0012] Step 1: Data Preparation

[0013] Use the original image as the input of the essential image decomposition model, and use the reflectance map or illumination map as the label during model training; randomly extract a certain number of input images from the test image dataset, and then extract the corresponding reflectance map or illumination map;

[0014] Step 2: Build a self-attention Tansformer network

[0015] Step 21: Build an adaptive window self-attention module

[0016] Adaptive window self-attention consists of two parts: a window adaptation stage guided by an edge detection operator and a remapping stage based on regression scores. The window adaptation module first filters the image using a k×k Sobel edge detection operator implemented by convolution to obtain a smoothness score for each k×k window area. Then, based on the score, the window in the image is adaptively divided into a larger "long-range window" and a smaller "local window"; the size of the "local window" is k×k, and the size of the "long-range window" is Nk×Nk.

[0017] In the remapping stage, the two types of windows obtained in the window adaptation stage are sorted according to the scores of the Sobel operator filter. In the sorting, all windows are sorted in k×k sizes, among which the "long-distance window" is evenly divided into 4 parts and kept adjacent. After the sorting, all k×k windows are restored to the original image size according to the sorting, and further self-attention calculation is performed on the window of size 2k×2k.

[0018] Step 22: Build a window-enhanced self-attention module

[0019] When calculating the window-enhanced self-attention, the feature map is first mapped into three feature matrices Q, K, and V. Then, the Q matrix and the K matrix are used to calculate the channel attention matrix with a size of the number of channels along the channel direction. Then, the channel attention matrix is ​​multiplied with the Q matrix and the K matrix respectively for enhancement. Finally, the enhanced Q matrix and the K matrix are used to calculate the self-attention matrix along the spatial direction.

[0020] Step 3: Build a multi-scale codec network

[0021] The multi-scale codec consists of a four-level network, each of which is composed of multiple sets of adaptive window self-attention and channel-enhanced self-attention in series to process feature maps of different scales. The four-level codec network uses downsampling and upsampling from encoder to decoder to complete the encoding and decoding according to the above scales, and outputs an image with the same size and dimensions as the input image.

[0022] Step 4: Set the loss function and input the training dataset to train the multi-scale encoder-decoder network. Use the reflectance map labels in the training of the reflectance map decomposition network and the illumination map labels in the training of the illumination map decomposition network.

[0023] Step 5: Use the trained multi-scale codec network to decompose the image and obtain a reflectance map or illumination map.

[0024] A further technical solution of the present invention is: in order to keep the window size consistent, the "local window" is directly obtained by dividing the temporary window during the Sobel operator filtering, and the "long-distance window" is obtained by splicing the four temporary windows with the closest spatial distances.

[0025] A further technical solution of the present invention: in step 3, the first-level network processes a feature map with a size of 256×256 and a number of channels of 32; the second-level network processes a feature map with a size of 128×128 and a number of channels of 64; the third-level network processes a feature map with a size of 64×64 and a number of channels of 128; and the fourth-level network processes a feature map with a size of 32×32 and a number of channels of 256.

[0026] A further technical solution of the present invention is to use mean square error, local mean square error and structural dissimilarity as output evaluation indicators.

[0027] A further technical solution of the present invention is as follows: during training, random seeds are used to fix the initialization parameters, the network optimizer uniformly uses the Adam optimizer, the optimizer parameters β1 and β2 are set to 0.9 and 0.999 respectively, and the weight decay rate is set to 1e-8; the number of network epochs is set to 250 times, and the initial learning rate is 0.0002.

[0028] A further technical solution of the present invention: the loss function in training includes Charbonnier loss, perceptual loss and Sobel loss.

[0029] An application of the above-mentioned image decomposition method is characterized in that the texture, color, material, shape information, lighting information and environmental information of the target are separated from the image, so that the part of the image affected by the environment is filtered out, and a more accurate feature description of the target is obtained, which is conducive to improving the performance of other visual perception tasks.

[0030] The beneficial effects of the present invention are:

[0031] Because the correlation between illumination or material information in an image is not limited to local regions, traditional convolutional networks, limited by the receptive field of the convolution kernel, have difficulty extracting relevant features over a wide range. Therefore, an improved Transformer network is proposed to enhance the model's perception of global detail information. The proposed adaptive window self-attention method adaptively adjusts the window based on image content to strengthen the network's focus on different types of features and reduce the model's resource usage. The channel-enhanced self-attention module enhances the effective information in the feature map by performing self-attention calculations across different dimensions. Furthermore, a multi-scale network structure is proposed to enhance the high-frequency and low-frequency components of features at different scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0033] Figure 1 Multi-scale encoding and decoding Transformer network structure;

[0034] Figure 2 Schematic diagram of the division of local window and long-distance window;

[0035] Figure 3 Use channel self-attention to enhance the Q and K matrices respectively;

[0036] Figure 4 Schematic diagram of the visualization results of training on the ShapeNet dataset;

[0037] Figure 5 Schematic diagram of the visualization results of different self-attention modules. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0039] The embodiment of the present invention proposes a multi-scale encoding and decoding essential image decomposition network based on self-attention. The network uses a parallel encoding and decoding strategy, that is, two sets of identical but independent encoders and decoders are used to generate the image of the reflectance map and the illumination map. Its structure is as follows Figure 1 As shown in Figure 2, each network consists of three parts: encoder, decoder, and skip connection. Both the encoder and decoder are composed of multiple self-attention modules and downsampling or upsampling modules.

[0040] Based on the above network, the embodiment of the present invention also proposes a multi-scale encoding and decoding essential image decomposition method based on self-attention, and the specific steps are as follows:

[0041] Step 1: Data Preparation

[0042] This paper uses the original image as input to the intrinsic image decomposition model, and uses the reflectance map and illumination map as labels during model training. A certain number of input images are randomly sampled from the test image dataset, and the corresponding reflectance maps and illumination maps are then extracted. Image embedding is performed on each sampled input image, which is then processed as an input token for the subsequent Transformer network.

[0043] Step 2: Build a self-attention Tansformer network

[0044] (1) Constructing an adaptive window self-attention module

[0045] Adaptive window self-attention consists of two parts: a window adaptation stage guided by an edge detection operator and a remapping stage based on regression scores. The window adaptation module first filters the image using a sobel edge detection operator of size k×k implemented by convolution to obtain a smoothness score for each window area of ​​size k×k. Then, based on the score, the windows in the image are adaptively divided into larger "long-distance windows" and smaller "local windows". Among them, the size of the "local window" is k×k, and the size of the "long-distance window" is Nk×Nk, and the present invention takes N=2. In order to keep the window size consistent, the "local window" can be directly obtained from the temporary window divided during the sobel operator filtering, while the "long-distance window" is obtained by splicing the four temporary windows with the closest spatial distance. Figure 2 Describes how the windows are divided, with local windows marked in orange and long-range windows marked in blue.

[0046] The remapping stage sorts the two types of windows obtained in the window adaptation stage according to their Sobel filter scores. All windows are sorted in k×k order, with the "long-range window" evenly divided into four parts, maintaining contiguousness. After sorting, all k×k windows are restored to their original image size, and self-attention calculations are performed on 2k×2k windows. Furthermore, the remapping stage preserves the corresponding window mappings, allowing each window to be restored to its original spatial position after the window self-attention calculations are complete.

[0047] (2) Constructing a window-enhanced self-attention module

[0048] When calculating the window enhanced self-attention, the feature map is first mapped into three feature matrices Q, K, and V. Then, the Q matrix and the K matrix are used to calculate the channel attention matrix with the number of channels along the channel direction. Then, the channel attention matrix is ​​multiplied with the Q matrix and the K matrix respectively for enhancement. Finally, the enhanced Q matrix and the K matrix are used to calculate the self-attention matrix along the spatial direction. The process is as follows. Figure 3 shown.

[0049] Step 3: Build a multi-scale codec network

[0050] The multi-scale codec consists of a four-level network. Each level of the network is composed of multiple sets of adaptive window self-attention and channel-enhanced self-attention in series to process feature maps of different scales. The first level of the network processes feature maps with a size of 256×256 and 32 channels. The second level of the network processes feature maps with a size of 128×128 and 64 channels. The third level of the network processes feature maps with a size of 64×64 and 128 channels. The fourth level of the network processes feature maps with a size of 32×32 and 256 channels. The four-level codec network uses downsampling and upsampling from encoder to decoder to complete the encoding and decoding according to the above scales, and outputs an image with the same size and dimension as the input image. The schematic diagram of the multi-scale codec Transformer network is shown below. Figure 1 shown.

[0051] Step 4: Define the loss function

[0052] The loss function of the multi-scale encoding and decoding Transformer network is defined as follows:

[0053] L=L C +L P +L S

[0054] Among them L C is the Charbonnier loss, L P is the perceptual loss, L S is the Sobel loss.

[0055] The Charbonnier loss is calculated as follows:

[0056]

[0057] The perceptual loss is calculated as follows:

[0058] L P (x,x′)=L MSE (vgg(x),vgg(x′))

[0059] To make the image restoration model more robust to shape, this paper proposes the Sobel loss. The Sobel loss first uses the Sobel operator to extract edges from the real image and the generated image, then calculates the mean squared error (MSE) of the generated image. The Sobel loss can constrain the shape and contour of the image, reducing the degree of damage to the restored image. Its formula is as follows:

[0060] L S (x,x′)=L MSE (sobel(x),sobel(x′))

[0061] Step 5: Network training

[0062] For all experiments, we used fixed initialization parameters using random seeds. The network optimizer used the same Adam optimizer, with parameters β1 and β2 set to 0.9 and 0.999, respectively, and a weight decay rate of 1e-8. The number of network epochs was set to 250, with an initial learning rate of 0.0002. The batch size for model training was set to 24. The input image resolution for both training and testing was 256×256×3.

[0063] Example

[0064] Step 1: Data preparation and indicator selection

[0065] Datasets used in the essential image decomposition task can be divided into datasets based on real-world images and computer-generated datasets based on the acquisition method; and into datasets with rich labels and datasets with sparse labels based on the annotation level. Due to the differences in image acquisition difficulty and supervision performance between datasets, the mainstream datasets are computer-generated datasets with rich labels and real-world image datasets with sparse labels. Real-world image datasets with rich labels offer good supervision for supervised essential image decomposition frameworks, but the high cost of labeling, which requires controlling lighting conditions and scene alignment, limits the scale of such datasets. To increase the number of samples for model training, large-scale computer-generated datasets with rich labels, such as MPI-Sintel and ShapeNet, are used.

[0066] The images in the dataset are processed into original images, reflection images, and illumination images, which are used as input and two labels in training respectively. The image size is uniformly processed to 256×256×3.

[0067] Then, a suitable evaluation index is selected. The present invention uses mean square error, local mean square error and structural dissimilarity as evaluation indexes.

[0068] 1) Mean Squared Error (MSE)

[0069] The mean square error is often used in visual tasks to evaluate the degree of difference between the predicted image and the real image on a global scale. Its mathematical definition is shown as follows:

[0070]

[0071] The present invention normalizes the image before calculating the MSE, and the final MSE value range is (0, 1).

[0072] 2) Local Mean Square Error (LMSE)

[0073] The mean square error can roughly evaluate the degree of difference in the global range of the image, but it cannot accurately measure the difference in the local area of ​​the image. To evaluate the degree of difference in the local range of the image, the local mean square error (LMSE) was proposed based on the mean square error. LMSE sets a sliding window in the image and calculates the mean square error of the image block in each window. The mean square error of all windows is then combined to obtain the LMSE. LMSE uses scale-invariant mean square error (MSE) in the calculation. The scale-invariant covariance adjusts the error value by introducing the global brightness of the predicted image. Its formula is as follows:

[0074]

[0075] Specifically, when calculating LMSE, a sliding window of size k×k is first selected, and then it slides across the image in steps of k / 2. Each slide calculates the scale-invariant covariance of a local region ω. If the set of local regions of the image is represented by Ω, then the LMSE is equal to the cumulative sum of the scale-invariant covariances of all local regions within the set Ω. Mathematically, this is:

[0076] LMSE k (x,x′)=∑ ω∈Ω siMSE(x ω ,x′ ω )

[0077] In actual calculation, the present invention sets the window size k to 20. The value range of LMSE is (0, 1), and the smaller the value within the value range, the smaller the local difference of the essential image.

[0078] 3) Structural dissimilarity (DSSIM)

[0079] Structural similarity (SSIM) is a quantitative metric that measures the degree of image similarity in terms of structure, contrast, and brightness. Given two images X and Y, the structural similarity between X and Y can be derived using the following formula:

[0080] SSIM(X,Y)=L(X,Y)*C(X,Y)*S(X,Y)

[0081]

[0082]

[0083]

[0084] where μ X and μ Y represents the mean of the image, σ X and σ YRepresents the standard deviation of the image, σ XY Represents the covariance of the image, C1, C2 and C2 are constants. In order to avoid the occurrence of zero division phenomenon, C1 is usually taken as (k1L) 2 , C2=(k2L) 2 , Where k1 = 0.01, k2 = 0.03, and L = 255. Substituting the formulas for L(X, Y), C(X, k), and S(X, Y) into the structural similarity formula, we can derive the complete formula for structural similarity as shown below.

[0085]

[0086] The larger the structural similarity, the more similar the prediction result is to the label; conversely, the smaller the structural similarity, the less similar the generated image is to the label image. In order to make this indicator the same as the other two indicators, the smaller the value, the better, this paper uses structural dissimilarity instead of structural similarity.

[0087] Structural dissimilarity (DSSIM) is a linear mapping of structural similarity, and its conversion method can be described by the following formula:

[0088]

[0089] The value range of DSSIM is [0,1], and the smaller the value, the closer the prediction result is to the label.

[0090] Step 2: Build a multi-scale encoding and decoding Transformer network

[0091] The present invention uses PyTorch to build a multi-scale codec Transformer network and trains it using the dataset created in step 1. The model is trained for a total of 250 epochs.

[0092] Step 3: Experimental results analysis

[0093] 1) Experimental results on the ShapeNet dataset

[0094] Table 1 compares the performance of our proposed multi-scale encoding / decoding Transformer network with other intrinsic image decomposition methods on the ShapeNet dataset. This dataset uses 10,000 image pairs consisting of original images, reflectance maps, and illumination maps. Among the algorithms used for comparison, SIRFS and IIW are traditional handcrafted feature-based methods, while DI, Shi, LapPyrNet, and NCCNet are deep learning-based algorithms.

[0095] According to the comparison of quantitative indicators in the experimental results, the multi-scale encoding and decoding Transformer network performs better than other essential image decomposition methods in terms of mean square error (MSE), local mean square error (LMSE) and structural dissimilarity (DSSIM) when restoring illumination maps. When restoring reflectance maps, the mean square error indicator is only slightly inferior to NCCNet, and it is better than the remaining other methods. Its performance in local mean square error is comparable to NCCNet.

[0096] Table 1 Training performance on the ShapeNet dataset

[0097]

[0098]

[0099] Figure 4 The comparison of visualization effects of reflection maps, illumination maps and labeled images obtained by image restoration using various methods is shown. It can be seen that the illumination map restoration results of traditional algorithms based on manual features, such as IIW and SIFRS, are relatively smooth, while the reflection map restoration results of SIFRS show that its illumination information is not completely removed. In deep learning-based methods, some early algorithms such as DI showed a large number of blurred pixels in the illumination map and reflection map restoration results, resulting in poor image smoothness and texture consistency. Later work such as LapPyrNet performed better in image smoothness and texture consistency, but there is still room for improvement in detail information. The self-attention-based encoding and decoding network proposed in the present invention can be found to have improved image smoothness in the restoration of illumination maps, and the restoration results of reflection maps are also supplemented in detail information.

[0100] 2) Experimental results on the MPI-Sintel dataset

[0101] The proposed method is compared with similar works on the MPI-Sintel dataset (based on image segmentation). The results are shown in Table 2.

[0102] Table 2 Training performance of different methods on the MPI dataset

[0103]

[0104] Experimental results show that the proposed self-attention-based encoder-decoder network outperforms other essential image decomposition methods in terms of mean squared error (MSE). Regarding the local mean squared error (LMSE) metric, the proposed method performs better than other methods for illumination map recovery, and its performance for reflectance map recovery is comparable to the best results among other methods. However, the performance of the self-attention-based encoder-decoder network is slightly inferior to the state-of-the-art methods in terms of structural dissimilarity.

[0105] 3) Evaluation of the Self-Attention Module

[0106] The encoder and decoder in the generator, which consists of an encoder-decoder network, share a similar basic model: a self-attention module followed by a dimension mapping. This experiment aims to verify the effectiveness of improvements to the self-attention module. To measure the effectiveness of using "adaptive attention" and "channel-enhanced attention" in the generator, a controlled experiment was designed using VIT and Swin-Transformer as control groups. The specific experimental setup is as follows:

[0107] (a) The basic module of the generator consists only of “windowed self-attention”;

[0108] (b) The upper layer of the generator is composed of "window self-attention" and the lower layer is composed of "global self-attention";

[0109] (c) The basic module of the generator consists only of “adaptive attention”;

[0110] (d) The basic module of the generator consists only of “channel-enhanced attention”;

[0111] (e) The basic module of the generator is composed of a stack of “adaptive attention” and “channel-enhanced attention”;

[0112] Due to the limitations of the experimental environment, it is difficult to fully use "global self-attention" as a basic network module under the premise of controlling variables. The solution provided by the present invention is to use "window self-attention" in the upper layer with higher feature map resolution and "global self-attention" in the bottom two layers with lower resolution to form a contrast. In this experiment, except for the self-attention module, the other variable conditions are the same. The present invention trains the above five cases on the ShapeNet dataset, and the results are shown in Table 3.

[0113] The experimental results in the table show that the MSE of the reconstructed image using "Adaptive Attention" alone is better than that of "Window Self-Attention," but slightly worse than that of "Global Self-Attention." Using "Channel-Enhanced Self-Attention" alone also slightly outperforms the MSE of the reconstructed image. When using both "Adaptive Window Self-Attention" and "Channel-Enhanced Self-Attention" simultaneously, the experimental results are similar to those using "Global Self-Attention." Furthermore, DSSIM performs poorly when using "Adaptive Window Self-Attention," but using "Channel-Enhanced Self-Attention" compensates for this.

[0114] Table 3 Experimental results using different attention modules

[0115]

[0116] Figure 5 The visualization results of essential image decomposition using different self-attention modules are shown. The figure shows the restored reflection and illumination images of three groups of objects. The left column of each group shows the reflection image, and the right column shows the illumination image. Comparing the results in the above figure shows that simply using global self-attention (VIT) in the network does not necessarily achieve good visual results in reflectance image restoration. Introducing a sliding window into the self-attention mechanism can improve the local visualization of the restored reflectance image. Further introducing channel-enhanced attention and adaptive window attention further improves the local accuracy of the restored reflectance image. By combining channel-enhanced attention and adaptive window attention, as shown in the image in the first column of the sixth row, the restored reflectance image exhibits better color consistency and smoothness.

[0117] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. A multi-scale encoding and decoding essential image decomposition method based on self-attention, characterized by Here are the steps: Step 1: Data Preparation Use the original image as the input of the essential image decomposition model, and use the reflectance map or illumination map as the label during model training; randomly extract a certain number of input images from the test image dataset, and then extract the corresponding reflectance map or illumination map; Step 2: Build a self-attention Tansformer network Step 21: Build an adaptive window self-attention module The adaptive window self-attention consists of two parts: the window adaptation stage guided by the edge detection operator and the remapping stage based on the regression score; the window adaptation module first uses the size of The sobel edge detection operator is used to filter the image and obtain each image of size The smoothness score of the window area is obtained; then the window in the image is adaptively divided into a larger "long-distance window" and a smaller "local window" according to the score; the size of the "local window" is , the size of the "long-distance window" is ; In the remapping stage, the two types of windows obtained in the window adaptation stage are sorted according to the scores of the Sobel operator filter. In the sorting, all windows are sorted by The size of the "long distance window" is evenly divided into 4 parts and kept adjacent; all the The window is restored to the original image size according to the sorting, and further resized Window self-attention calculation; Step 22: Build a window-enhanced self-attention module When calculating the window-enhanced self-attention, the feature map is first mapped into three feature matrices Q, K, and V. Then, the Q matrix and the K matrix are used to calculate the channel attention matrix with a size of the number of channels along the channel direction. Then, the channel attention matrix is ​​multiplied with the Q matrix and the K matrix respectively for enhancement. Finally, the enhanced Q matrix and the K matrix are used to calculate the self-attention matrix along the spatial direction. Step 3: Build a multi-scale codec network The multi-scale codec consists of a four-level network, each of which is composed of multiple sets of adaptive window self-attention and channel-enhanced self-attention in series to process feature maps of different scales. The four-level codec network uses downsampling and upsampling from encoder to decoder to complete the encoding and decoding according to the above scales, and outputs an image with the same size and dimensions as the input image. Step 4: Set the loss function and input the training dataset to train the multi-scale encoder-decoder network. Use the reflectance map labels in the training of the reflectance map decomposition network and the illumination map labels in the training of the illumination map decomposition network. Step 5: Use the trained multi-scale codec network to decompose the image and obtain a reflectance map or illumination map.

2. The multi-scale encoding and decoding essential image decomposition method based on self-attention according to claim 1, characterized in that: To keep the window size consistent, the "local window" is directly obtained by dividing the temporary window during the Sobel operator filtering, while the "long-distance window" is obtained by splicing the four temporary windows with the closest spatial distance.

3. The multi-scale encoding and decoding essential image decomposition method based on self-attention according to claim 1, characterized in that: In step 3, the first-level network processes feature maps with a size of 256×256 and 32 channels; the second-level network processes feature maps with a size of 128×128 and 64 channels; the third-level network processes feature maps with a size of 64×64 and 128 channels; and the fourth-level network processes feature maps with a size of 32×32 and 256 channels.

4. The multi-scale encoding and decoding essential image decomposition method based on self-attention according to claim 1, characterized in that: The mean square error, local mean square error and structural dissimilarity are used as output evaluation indicators.

5. The multi-scale encoding and decoding essential image decomposition method based on self-attention according to claim 1, characterized in that: During training, random seeds are used to initialize fixed parameters, and the network optimizer uniformly uses the Adam optimizer. The optimizer parameters are and Set them to 0.9 and 0.999 respectively, and the weight decay rate is set to 1e-8; the number of network epochs is set to 250 times, and the initial learning rate is 0.0002.

6. The multi-scale encoding and decoding essential image decomposition method based on self-attention according to claim 1, characterized in that The loss functions used in training include Charbonnier loss, perceptual loss, and Sobel loss.

7. An application of the image decomposition method according to claim 1, characterized in that: The target's texture, color, material, shape information, lighting information and environmental information are separated from the image, so that the part of the image affected by the environment is filtered out, and a more accurate feature description of the target is obtained, which is beneficial to improving the performance of other visual perception tasks.

Images