Image defogging model training method and image defogging processing method and device

Abstract

The invention discloses an image defogging model training method and an image defogging processing method and device. The image defogging model training method comprises the following steps: acquiring a training sample set and a sample label set; an initial network model is constructed, and the model at least comprises M Transformer modules which are used for carrying out global context feature extraction on the foggy image and are combined with content perception and recursive interaction, and a multi-scale frequency domain attention module which is used for carrying out frequency domain feature extraction on global context feature maps output by the M Transformer modules which are combined with content perception and recursive interaction, m is a positive integer; and performing iterative training on the initial network model by using the training sample set and the sample label set to obtain an image defogging model. The image defogging method and device solve the technical problem that the defogged image is poor in quality due to the fact that a related image defogging model is poor in capability of processing a low-frequency foggy part and recovering high-frequency details.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image dehazing model training method and an image dehazing processing method and apparatus. Background Technology

[0002] In real life, light interacts with suspended particles in the atmosphere (such as water droplets, dust, and pollutants) during its propagation, causing images captured by sensors to appear blurry, with reduced contrast and color distortion. Currently, image dehazing algorithms are mainly divided into traditional dehazing algorithms and deep learning-based dehazing algorithms. Traditional dehazing algorithms are further divided into image enhancement-based methods and filtering-based methods. Image enhancement-based methods mainly employ histogram equalization and Retinex theory, but they neglect physical models, leading to the introduction of noise or distortion in the dehazed image. Filtering-based methods include dark channel priors, guided filtering, and bilateral filtering, but these methods rely on statistical prior estimates of transparency and cannot dehaze sky areas (bright areas), resulting in high computational complexity.

[0003] In contrast to traditional methods, deep learning-based algorithms directly learn the mapping between foggy images and clear, fog-free images without requiring physical modeling, thus enabling them to handle diverse fog distributions in various scenes. However, due to the high computational complexity of most deep learning models and their poor ability to handle low-frequency fog components and recover high-frequency details, image details become blurred and colors are distorted, easily producing halo effects. Currently, no effective solution has been proposed to address these issues. Summary of the Invention

[0004] This application provides an image dehazing model training method and an image dehazing processing method and apparatus to at least solve the technical problem that the relevant image dehazing model has poor ability to process low-frequency foggy parts and restore high-frequency details, resulting in poor image quality after dehazing.

[0005] According to one aspect of the embodiments of this application, an image dehazing model training method is provided, comprising: acquiring a training sample set and a sample label set, wherein the training sample set includes multiple first hazy images as training samples, and the sample label set includes a first dehazed image corresponding to each first hazy image as a sample label corresponding to the corresponding training sample; constructing an initial network model, wherein the initial network model includes at least: M Transformer modules combining content awareness and recursive interaction for global context feature extraction of hazy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules combining content awareness and recursive interaction, where M is a positive integer; and iteratively training the initial network model using the training sample set and the sample label set to obtain an image dehazing model.

[0006] Optionally, the initial network model further includes: a feature extraction module, N cascaded downsampling modules, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where N is a positive integer. The initial network model is iteratively trained using a training sample set and a sample label set to obtain an image dehazing model. This includes: for each training sample in the training sample set, the initial network model performs the following analysis on the training sample to obtain the corresponding first predicted dehazed image: the training sample is sequentially input into the primary feature extraction module and the N cascaded downsampling modules within the initial network model to obtain the feature map output by each downsampling module, wherein the resolution of the feature maps output by the N cascaded downsampling modules decreases sequentially; the feature map output by the Nth downsampling module is input into M Transformer modules combining content awareness and recursive interaction to obtain a global context feature map; the global context feature map is input into a multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map. The frequency domain enhanced feature map is input into the (N+1)th upsampling module out of N+1 upsampling modules to obtain the corresponding initial reconstructed feature map. The following steps are repeated until the target reconstructed feature map output by the first upsampling module is obtained: The feature map output by the nth downsampling module and the reconstructed feature map output by the (N+1)th upsampling module are input into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstructed feature map, and the reconstructed feature map is input into the nth upsampling module to obtain the reconstructed feature map output by the nth upsampling module, where n is a positive integer between 1 and N. The target reconstructed feature map output by the first upsampling module is analyzed using the output module to obtain the first predicted dehazed image corresponding to the training sample. A loss function is constructed based on the first predicted dehazed image and sample label corresponding to each training sample, and the loss function is calculated using the backpropagation algorithm. When the loss function satisfies the preset convergence condition, the trained image dehazing model is obtained.

[0007] Optionally, each downsampling module includes: a first downsampling unit, a second downsampling unit, a convolutional layer, and an average pooling layer. The first downsampling unit is composed of a first depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The second downsampling unit is composed of a second depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The convolutional kernel of the first depthwise separable convolutional layer is larger than the convolutional kernel of the second depthwise separable convolutional layer. The training samples are sequentially input into the N downsampling modules within the initial network model to obtain the feature map output by each downsampling module. This includes: inputting the training samples into the primary feature extraction module within the initial network model to obtain an initial local feature map; for the first downsampling module among the N downsampling modules, the first local feature map obtained by the first downsampling unit in the first downsampling module processing the initial local feature map is compared with the first local feature map obtained by the second downsampling unit in the first downsampling module processing the initial local feature map. The first local feature map is concatenated with the second local feature map obtained from processing the first local feature map. The concatenated result is then added to the feature map obtained from the initial local feature map after processing by the average pooling layer in the first downsampling module. Finally, the concatenated result is processed by the convolutional layer in the first downsampling module to obtain the feature map output by the first downsampling module. For the other downsampling modules among the N downsampling modules (excluding the first downsampling module), the first local feature map obtained from processing the feature map output by the first downsampling unit in the current downsampling module with the second local feature map obtained from processing the first local feature map with the second downsampling unit in the current downsampling module is concatenated with the feature map obtained from processing the first local feature map with the second downsampling unit in the current downsampling module. The concatenated result is then added to the feature map obtained from processing by the average pooling layer in the current downsampling module. Finally, the concatenated result is processed by the convolutional layer in the current downsampling module to obtain the feature map output by the current downsampling module.

[0008] Optionally, each Transformer module combining content awareness and recursive interaction includes at least: a normalization layer, a content-aware sparse self-attention unit, and a recursively gated interactive unit. The feature map output by the Nth downsampling module is input into M Transformer modules combining content awareness and recursive interaction to obtain the corresponding global context feature map. This includes: for the first Transformer module combining content awareness and recursive interaction among the M modules, the feature map output by the Nth downsampling module is sequentially input into the normalization layer and the content-aware sparse self-attention unit within the first Transformer module, resulting in a content-sensitive feature map output by the content-aware sparse self-attention unit. This content-sensitive feature map is then residually concatenated with the feature map output by the Nth downsampling module to obtain the corresponding residual feature map. The residual feature map is then sequentially input into the normalization layer and the recursively gated interactive unit within the first Transformer module, resulting in an interactive fusion feature map output by the recursively gated interactive unit. The residual feature map is then concatenated with the residual feature map to obtain the corresponding global context feature map. For the M Transformer modules combining content awareness and recursive interaction (excluding the first one), the global context feature map output by the previous Transformer module is sequentially input into the normalization layer and content-aware sparse self-attention unit in the current Transformer module, resulting in the content-sensitive feature map output by the content-aware sparse self-attention unit. This is then residually concatenated with the global context feature map output by the previous Transformer module to obtain the corresponding residual feature map. The residual feature map is then sequentially input into the normalization layer and recursively gated interactive unit in the current Transformer module, resulting in the interactive fusion feature map output by the recursively gated interactive unit. This is then residually concatenated with the residual feature map to obtain the corresponding global context feature map.

[0009] Optionally, the content-aware sparse self-attention unit includes at least: a convolutional layer, an average pooling layer, a second activation function layer, a depthwise convolutional layer, and a multi-head self-attention layer. The global contextual feature map output from the previous Transformer module combining content awareness and recursive interaction is sequentially input into the normalization layer and the content-aware sparse self-attention unit in the current Transformer module combining content awareness and recursive interaction to obtain the content-sensitive feature map output by the content-aware sparse self-attention unit. This includes inputting the global contextual feature map output from the previous Transformer module combining content awareness and recursive interaction into the normalization layer in the current Transformer module combining content awareness and recursive interaction. A normalized global context feature map is obtained. The normalized global context feature map is then processed using convolutional layers, average pooling layers, and a second activation function layer in the content-aware sparse self-attention unit of the Transformer module, which combines content awareness and recursive interaction, to obtain the content-aware mask score for each pixel within the normalized global context feature map. Based on the relationship between the content-aware mask score of each pixel and a preset threshold value, the normalized global context feature map is divided into foggy and fog-free regions. A multi-head self-attention layer is used to analyze the feature map of the foggy region, and a deep convolutional layer is used to analyze the feature map of the fog-free region, resulting in a content-sensitive feature map output by multiple content-aware sparse self-attention units.

[0010] Optionally, the recursive gated interactive unit includes: a first recursive subunit, a second recursive subunit, and a recursive update subunit. The first recursive subunit is composed of a stacked convolutional layer, a batch normalization layer, and a first activation function layer; the second recursive subunit is composed of a stacked convolutional layer and a second activation function layer; and the recursive update subunit is composed of a stacked global average pooling layer, a third activation function layer, and a convolutional layer. The residual feature map is sequentially input into the normalization layer and the recursive gated interactive unit in the current Transformer module combining content awareness and recursive interaction to obtain the interactive fusion feature map output by the recursive gated interactive unit. This includes: inputting the residual feature map into the normalization layer in the current Transformer module combining content awareness and recursive interaction to obtain a normalized residual feature map; during the first round of recursion, inputting the normalized residual feature map into the first recursive subunit in the current Transformer module combining content awareness and recursive interaction to obtain the corresponding local feature map; and inputting the residual feature map into the current Transformer module combining content awareness and recursive interaction. In the second recursive subunit and recursive update subunit of the Transformer module, the gating weights and global feature maps are obtained. The gating weights are multiplied by the local feature maps and global feature maps obtained in the first round of recursion, and then added element-wise. The feature map obtained by concatenating the summed feature map with the normalized residual feature map is used as the input to the second round of recursion. In each round of recursion from the second round onwards, the feature map output from the previous round of recursion is input into the second recursive subunit of the Transformer module, which combines content awareness and recursive interaction. In the recursive update subunit of the recursive gating interactive unit, new gating weights and a new global feature map are obtained. The new gating weights are used to multiply the feature map output from the previous recursive process and the new global feature map, and then the elements are added together. The feature map obtained by concatenating the summed feature map with the normalized residual feature map is used as the input for the next recursive process. The feature map obtained from the last recursive process is used as the interactive fusion feature map output by the recursive gating interactive unit in the Transformer module that combines content awareness and recursive interaction.

[0011] Optionally, the multi-scale frequency domain attention module includes at least: multiple dilated convolutional layers, multiple frequency domain attention units, convolutional layers, and average pooling layers. The process of inputting the global context feature map into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map includes: inputting the global context feature map into multiple dilated convolutional layers to obtain local feature maps at different scales, wherein each dilated convolutional layer has a different dilation rate; for local feature maps at different scales, inputting the local feature maps into the frequency domain attention units to obtain the corresponding local enhanced feature maps, wherein the frequency domain attention units are used to perform Fast Fourier Transform processing on the local feature maps and divide them into... The residual weights of the local feature maps are analyzed to obtain the corresponding query vector, key vector, value vector, and residual weight map. The dot product of the key vector and value vector is processed by the Softmax function to obtain the weight map. The weight map is then multiplied by the query vector and processed by the inverse Fast Fourier Transform to obtain the frequency domain weight map. The frequency domain weight map, the weight map, and the residual weight map are concatenated to obtain the corresponding local enhanced feature map. The local enhanced feature maps corresponding to local feature maps at different scales are concatenated by channels and added to the feature maps obtained by processing the global context feature map through convolutional layers and average pooling layers, respectively, to obtain the corresponding frequency domain enhanced feature map.

[0012] Optionally, the multi-scale feature selection module includes at least: a convolutional layer, a batch normalization layer, a first activation function layer, and compression and activation units. The feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are input into the nth multi-scale feature selection module, which is skip-connected to the nth downsampling module, to obtain the corresponding reconstructed feature map. This includes: concatenating the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module to obtain a concatenated feature map, which is then sequentially processed by the nth multi-scale feature selection module. The convolutional layer, normalization layer, first activation function layer, compression and activation unit in the selection module are used to obtain the channel attention feature map. The feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are multiplied by the channel attention feature map to obtain the first important feature in the feature map output by the nth downsampling module and the second important feature in the reconstructed feature map output by the (n+1)th upsampling module. The first important feature and the second important feature are concatenated to obtain the reconstructed feature map output by the nth multi-scale feature selection module.

[0013] Optionally, the output module may include at least multiple convolutional layers and a fourth activation function layer.

[0014] According to another aspect of the embodiments of this application, an image dehazing processing method is also provided, comprising: acquiring a second hazy image to be processed; processing the second hazy image using a pre-trained image dehazing model to obtain a corresponding second predicted dehazing image, wherein the image dehazing model is trained using the image dehazing model training method described above.

[0015] Optionally, the image dehazing model includes: N cascaded downsampling modules, M Transformer modules combining content awareness and recursive interaction, a multi-scale frequency domain attention module, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where M and N are both positive integers. The pre-trained image dehazing model is used to process the second hazy image to obtain the corresponding second predicted dehazed image. This includes: sequentially inputting the second hazy image into the primary feature extraction module and the N cascaded downsampling modules within the initial network model to obtain the feature map output by each downsampling module, wherein the resolution of the feature maps output by the N cascaded downsampling modules decreases sequentially; inputting the feature map output by the Nth downsampling module into the M Transformer modules combining content awareness and recursive interaction to obtain a global context feature map; and inputting the global upsampling module into the M Transformer modules combining content awareness and recursive interaction to obtain a global context feature map. The following steps are repeated: the feature map is input into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map; the frequency domain enhanced feature map is input into the (N+1)th upsampling module of the N+1 upsampling modules to obtain the corresponding initial reconstructed feature map; the following steps are repeated until the target reconstructed feature map output by the first upsampling module is obtained; the feature map output by the nth downsampling module and the reconstructed feature map output by the (N+1)th upsampling module are input into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstructed feature map, and the reconstructed feature map is input into the nth upsampling module to obtain the reconstructed feature map output by the nth upsampling module, where n is a positive integer between 1 and N; the target reconstructed feature map output by the first upsampling module is analyzed using the output module to obtain the second predicted dehazed image corresponding to the second foggy image.

[0016] Optionally, after obtaining the corresponding second predicted dehazed image, the method further includes: updating the training sample set and sample label set of the image dehazing model using the second hazy image and the second predicted dehazed image, and performing self-supervised training of the image dehazing model using the updated training sample set and the updated sample label set.

[0017] According to another aspect of the embodiments of this application, an image dehazing model training apparatus is also provided, comprising: a first acquisition module, configured to acquire a training sample set and a sample label set, wherein the training sample set includes multiple first hazy images as training samples, and the sample label set includes a first dehazed image corresponding to each first hazy image as a sample label corresponding to the corresponding training sample; a construction module, configured to construct an initial network model, wherein the initial network model includes at least: M Transformer modules combining content awareness and recursive interaction for global context feature extraction of hazy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules combining content awareness and recursive interaction, where M is a positive integer; and a training module, configured to iteratively train the initial network model using the training sample set and the sample label set to obtain an image dehazing model.

[0018] According to another aspect of the embodiments of this application, an image dehazing processing apparatus is also provided, including: a second acquisition module, configured to acquire a second hazy image to be processed; and a processing module, configured to process the second hazy image using a pre-trained image dehazing model to obtain a corresponding second predicted dehazed image, wherein the image dehazing model is trained using the image dehazing model training method described above.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described image dehazing model training method or image dehazing processing method.

[0020] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described image dehazing model training method or image dehazing processing method through the computer program.

[0021] In this embodiment, an image dehazing model is trained that includes at least a Transformer module combining content awareness and recursive interaction, and a multi-scale frequency domain attention module. The Transformer module, combining content awareness and recursive interaction, enhances the model's ability to extract global contextual features, improving its processing and feature understanding of foggy areas. The multi-scale frequency domain attention module efficiently identifies and attenuates low-frequency energy related to fog, restoring high-frequency details. Therefore, this image dehazing model, during the dehazing process of foggy images, not only accurately captures long-range dependencies but also works collaboratively in the frequency and spatial domains, improving the quality and detail preservation of the dehazed image, avoiding halo and color distortion, and effectively removing fog interference from the image, thus improving visual clarity and image quality. This solves the technical problem that related image dehazing models have poor ability to process low-frequency foggy parts and restore high-frequency details, resulting in poor image quality after dehazing. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart illustrating an optional image dehazing model training method according to an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of an optional initial network model according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of an optional downsampling module according to an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the structure of an optional Transformer module that combines content awareness and recursive interaction according to an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the structure of an optional recursive gating interactive unit according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an optional multi-scale frequency domain attention module according to an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of an optional frequency domain attention unit according to an embodiment of this application;

[0030] Figure 8This is a schematic diagram of the structure of an optional multi-scale feature selection module according to an embodiment of this application;

[0031] Figure 9 This is a schematic flowchart of an optional image dehazing method according to an embodiment of this application;

[0032] Figure 10 This is a schematic diagram of an optional image dehazing model training device according to an embodiment of this application;

[0033] Figure 11 This is a schematic diagram of an optional image dehazing processing apparatus according to an embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the hardware structure of a computer terminal for implementing an image dehazing model training method or an image dehazing processing method, according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] According to an embodiment of this application, an image dehazing model training method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] Figure 1 This is a flowchart illustrating an image dehazing model training method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0040] Step S102: Obtain the training sample set and the sample label set. The training sample set includes multiple first hazy images used as training samples, and the sample label set includes a first dehazed image corresponding to each first hazy image as the sample label for that training sample.

[0041] Step S104: Construct an initial network model, wherein the initial network model includes at least: M Transformer modules combining content awareness and recursive interaction for global context feature extraction of foggy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules combining content awareness and recursive interaction, where M is a positive integer.

[0042] Step S106: Iteratively train the initial network model using the training sample set and sample label set to obtain the image dehazing model.

[0043] Based on the scheme defined in steps S102 to S106 above, it can be understood that in this embodiment, the trained image dehazing model, through M Transformer modules combining content awareness and recursive interaction, enables the model to effectively extract global contextual features, enhancing its processing and feature understanding capabilities for dense fog areas; while the multi-scale frequency domain attention module can efficiently identify and attenuate low-frequency energy related to haze, restoring high-frequency details. Therefore, the image dehazing model trained by the method provided in this embodiment can not only accurately capture long-range dependencies during image dehazing, but also work collaboratively in the frequency and spatial domains, improving the quality and detail preservation of dehazed images, avoiding halo and color distortion, and achieving the goal of effectively removing fog interference from images and improving visual clarity and image quality.

[0044] The following section explains each step of the image dehazing model training method in conjunction with the specific implementation process.

[0045] In the technical solution provided in step S102 above, the system can first collect a large number of first foggy images from indoor and outdoor environments, and then preprocess these first foggy images, such as by cropping, flipping, and enhancing contrast, to enhance the diversity of the training sample set. This allows the model to learn more different image features and fog distribution patterns during training, thereby improving the model's generalization ability and robustness. Next, for each training sample in the training sample set, the training sample can be processed by manual dehazing or other known dehazing algorithms to obtain a clear first dehazed image as the corresponding sample label.

[0046] Through the above acquisition process, it is ensured that the first foggy image in the training sample set corresponds one-to-one with the first defogging image in the sample label set. This allows the model to learn the transformation rules from foggy to fog-free images during training, by using the first foggy image as input and its corresponding first defogging image as the expected output.

[0047] In the technical solution provided in step S104 above, the system can establish an initial network model, which includes: a primary feature extraction module, N cascaded downsampling modules, M Transformer modules combining content awareness and recursive interaction, a multi-scale frequency domain attention module, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where N is a positive integer.

[0048] For example, Figure 2 This is a schematic diagram of the structure of an optional initial network model, such as... Figure 2 As shown, the initial network model includes a primary feature extraction module, three downsampling modules, M Transformer modules that combine content awareness and recursive interaction, a multi-scale frequency domain attention module, three multi-scale feature selection modules, four upsampling modules, and an output module.

[0049] In the technical solution provided in step S106 above, the system can use the training sample set and sample label set to iteratively train the initial network model to obtain the image dehazing model.

[0050] As an optional implementation, the above model iterative training process may include the following steps:

[0051] Step S1061: For each training sample in the training sample set, the initial network model can analyze the training sample in the following way to obtain the corresponding first predicted dehazed image:

[0052] Step 1: Input the training samples sequentially into the primary feature extraction module and N cascaded downsampling modules within the initial network model for feature extraction and downsampling operations, obtaining the feature map output by each downsampling module. Wherein:

[0053] The primary feature extraction module consists of multiple convolutional layers (e.g., multiple 3x3 convolutional layers stacked together). Each convolutional layer learns features at different levels. Therefore, the primary feature extraction module can capture local feature maps of training samples. Specifically, the primary feature extraction module effectively extracts local features of the image, such as edges, textures, and local structural information, by performing a sliding window operation on the training samples.

[0054] Each of the N downsampling modules includes: a first downsampling unit, a second downsampling unit, a convolutional layer, and an average pooling layer. The first downsampling unit is composed of a first depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The second downsampling unit is composed of a second depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The convolutional kernel of the first depthwise separable convolutional layer is larger than the convolutional kernel of the second depthwise separable convolutional layer. Specifically:

[0055] For the first downsampling module among N downsampling modules, the first local feature map obtained by the first downsampling unit in the first downsampling module processing the initial local feature map is concatenated with the second local feature map obtained by the second downsampling unit in the first downsampling module processing the first local feature map. The concatenated result is then added to the feature map after the initial local feature map has been processed by the average pooling layer in the first downsampling module, and then processed by the convolutional layer in the first downsampling module to obtain the feature map output by the first downsampling module.

[0056] For all downsampling modules except the first one, the first local feature map obtained by the first downsampling unit in the current downsampling module processing the feature map output by the previous downsampling module is concatenated with the second local feature map obtained by the second downsampling unit in the current downsampling module processing the first local feature map. The concatenated result is then added to the feature map output by the previous downsampling module after processing by the average pooling layer in the current downsampling module. Finally, the result is processed by the convolutional layer in the current downsampling module to obtain the feature map output by the current downsampling module.

[0057] Therefore, the N cascaded downsampling modules reduce the resolution of the feature maps through downsampling operations, thereby obtaining multi-scale feature maps. As a result, the depth of the feature maps output by each downsampling module in the N cascaded downsampling modules increases sequentially, while the resolution decreases sequentially.

[0058] Specifically, Figure 3 This is a schematic diagram of an optional downsampling module, such as... Figure 3 As shown, the first downsampling unit in this downsampling module is composed of a 3x3 depthwise separable convolutional layer, a normalization layer, and a LeakReLU activation function layer stacked together, while the second downsampling unit is composed of a 1x1 depthwise separable convolutional layer, a normalization layer, and a LeakReLU activation function layer stacked together. Figure 3 The structural diagram shown above illustrates that the downsampling process described above can be understood as: taking the feature map P output by the previous downsampling module... total The first local feature map P′ is obtained by sequentially inputting a 3×3 separable convolutional layer conv1, a normalization layer, and a LeakyReLU activation function layer. total =LK(LN(conv1(P total Then, the first local feature map P′ total With the first local feature map P′ total The second local feature map P″ is obtained by sequentially processing a 1×1 depth separable convolutional layer conv2, a normalization layer, and a LeakyReLU activation function layer. total =LK(LN(conv2(P′) total The splicing process is performed, and the resulting splicing result P″′ is obtained. total =concat(P′ total ,P″ total The feature map P output by the previous downsampling module total The feature map Avg(P) after processing by the average pooling layer in the downsampling module total After addition, the sums are processed by the convolutional layer conv3 in the downsampling module to obtain the feature map output by the downsampling module. The final expression of the output feature map can be written as: P″″ total =conv3(Avg(P total )+P″ total ).

[0059] In the above expression, LN(·) is the layer normalization operation, and LK(·) is the LeakyReLU activation function.

[0060] Step 2: Input the feature map output by the Nth downsampling module into M Transformer modules that combine content awareness and recursive interaction to obtain the global context feature map.

[0061] Each Transformer module combining content-aware and recursive interaction includes: a normalization layer, a content-aware sparse self-attention unit, and a recursively gated interactive unit. These units analyze the feature map output by the Nth downsampling module to extract long-range semantic information and enhance the model's global feature extraction capabilities. Figure 4 This is a schematic diagram of the structure of an optional Transformer module that combines content awareness and recursive interaction.

[0062] Combination such as Figure 4 The structural diagram shown illustrates how each Transformer module, combining content awareness and recursive interaction, processes the input feature map according to a workflow to extract multi-granularity contextual information and obtain the corresponding global contextual feature map.

[0063] The first Transformer module, which combines content awareness and recursive interaction, sequentially inputs the feature map output by the Nth downsampling module into the normalization layer and the content-aware sparse self-attention unit in the first Transformer module, obtaining the content-sensitive feature map output by the content-aware sparse self-attention unit. This feature map is then residually concatenated with the feature map output by the Nth downsampling module to obtain the corresponding residual feature map. The residual feature map is then sequentially input into the normalization layer and the recursive gated interaction unit in the first Transformer module, obtaining the interaction fusion feature map output by the recursive gated interaction unit. This residual feature map is then residually concatenated with the residual feature map to obtain the corresponding global context feature map.

[0064] Except for the first Transformer module that combines content awareness and recursive interaction, other Transformer modules that combine content awareness and recursive interaction sequentially input the global context feature map output by the previous Transformer module into the normalization layer and content-aware sparse self-attention unit in the current Transformer module, to obtain the content-sensitive feature map output by the content-aware sparse self-attention unit. This feature map is then residually concatenated with the global context feature map output by the previous Transformer module to obtain the corresponding residual feature map. The residual feature map is then sequentially input into the normalization layer and recursively gated interactive unit in the Transformer module, to obtain the interactive fusion feature map output by the recursively gated interactive unit. This residual feature map is then residually concatenated with the residual feature map to obtain the corresponding global context feature map.

[0065] Optionally, the content-aware sparse self-attention unit includes at least: a convolutional layer, an average pooling layer, a second activation function layer, a depthwise convolutional layer, and a multi-head self-attention layer. Therefore, the content-sensitive feature map output by the content-aware sparse self-attention unit in the current Transformer module combining content awareness and recursive interaction can be obtained as follows:

[0066] First, the global context feature map output by the previous Transformer module that combines content awareness and recursive interaction is input into the normalization layer in the current Transformer module that combines content awareness and recursive interaction to obtain a normalized global context feature map.

[0067] Next, the normalized global context feature map is processed using the convolutional layer, average pooling layer, and second activation function layer in the content-aware sparse self-attention unit of the Transformer module, which combines content awareness and recursive interaction. This yields the content-aware mask score for each pixel within the normalized global context feature map. The convolutional kernels of the convolutional layers can be 1x1 in size, and the second activation function is the sigmoid activation function. A sliding window operation is then performed on the normalized global context feature map, applying 1x1 convolution, average pooling, and sigmoid activation transformation to a portion of the feature map within the window. This transforms the value of each pixel in the portion of the feature map within the window into the probability of whether that pixel belongs to a foggy region (i.e., the content-aware mask score, which ranges from 0 to 1).

[0068] Then, based on the relationship between the content-aware mask score of each pixel in the normalized global context feature map and a preset threshold value, the normalized global context feature map is divided into foggy regions and fog-free regions. The preset threshold value can be 0.5; if the probability (content-aware mask score) of a pixel is higher than 0.5, it indicates that the pixel belongs to a foggy region; conversely, if the probability (content-aware mask score) of a pixel is not higher than 0.5, it indicates that the pixel belongs to a fog-free region. Therefore, by judging through the preset threshold value, the normalized global context feature map can be divided into two types of regions: foggy regions (i.e., foggy windows) and fog-free regions (i.e., fog-free windows).

[0069] Finally, a multi-head self-attention layer is used to analyze the feature map of the foggy region, and a deep convolutional layer is used to analyze the feature map of the fog-free region, resulting in a content-sensitive feature map output by the multi-content-aware sparse self-attention unit. Specifically, the multi-head self-attention layer performs linear projection and normalization on the features corresponding to the foggy region (i.e., the foggy window) to obtain the query vector, key vector, and value vector. The query vector and key vector are multiplied and then processed through Softmax to obtain the multi-head attention weight map. Multiplying this weight map by the value vector yields the multi-head attention feature map. Therefore, the expression for the multi-head attention feature map can be written as: In the formula, [Q,K,V]=[FW q FW k FW v ], W q W k W v Let F represent the learnable matrix, d represent the linear projection parameter matrix, P represent the partial feature map corresponding to the foggy input region, Q, K, and V represent the query vector, key vector, and value vector, respectively, and ⊙ represent the dot product operation.

[0070] This processing method allows the model to automatically adjust its processing approach during the process, achieving optimal allocation and efficient utilization of computing resources. For foggy regions, multi-head self-attention layers can delve deeper into details, while for fog-free regions, deep convolutional layers are sufficient to meet computational needs, thus avoiding unnecessary computational overhead.

[0071] Furthermore, the recursive gated interactive unit includes a first recursive subunit, a second recursive subunit, and a recursive update subunit. The first recursive subunit is composed of a convolutional layer, a batch normalization layer, and a first activation function layer stacked together; the second recursive subunit is composed of a convolutional layer and a second activation function layer stacked together; and the recursive update subunit is composed of a global average pooling layer, a third activation function layer, and a convolutional layer stacked together. Therefore, the interactive fusion feature map output by the recursive gated interactive unit in the Transformer module combining content awareness and recursive interaction can be obtained in the following manner according to the embodiments of this application:

[0072] First, the residual feature map is input into the normalization layer in the current Transformer module that combines content awareness and recursive interaction to obtain the normalized residual feature map;

[0073] In the first round of recursion, the normalized residual feature map is input into the first recursive sub-unit of the Transformer module that combines content awareness and recursive interaction to obtain the corresponding local feature map. The residual feature map is input into the second recursive sub-unit and the recursive update sub-unit of the Transformer module that combines content awareness and recursive interaction to obtain the gating weight and the global feature map. The gating weight is multiplied by the local feature map and the global feature map obtained in the first round of recursion, and then added element by element. The feature map obtained by concatenating the summed feature map with the residual of the normalized residual feature map is used as the input for the second round of recursion.

[0074] In each subsequent recursive round, the feature map output from the previous recursive round is input into the second recursive subunit and the recursive update subunit in the recursive gating interactive unit of the Transformer module that combines content awareness and recursive interaction, to obtain new gating weights and a new global feature map. The new gating weights are used to multiply the feature map output from the previous recursive round and the new global feature map respectively, and then the elements are added together. The feature map obtained by concatenating the summed feature map with the normalized residual feature map is used as the input for the next recursive round.

[0075] Finally, the feature map obtained from the last round of recursion is used as the interaction fusion feature map output by the recursive gating interactive unit in the Transformer module that combines content awareness and recursive interaction.

[0076] Specifically, Figure 5 This is a schematic diagram of the structure of an optional recursive gated interactive unit, such as... Figure 5 As shown, the first recursive subunit has a 3x3 convolutional layer and a LeakyReLU activation function as its first activation layer. The second recursive subunit has a 1x1 convolutional layer and a Sigmoid activation function as its second activation layer. The recursive update subunit has a GeLU activation function as its third activation layer and a 1x1 convolutional layer. Therefore, combined with... Figure 5 The processing procedure of the recursive gated interactive unit can be described as follows:

[0077] In the first round of recursion, the normalized residual feature map P is processed by the first recursive subunit (a 3x3 convolutional layer conv3, a batch normalization layer, and a LeakyReLU activation function layer) to obtain the local feature P. l 1=LK(BN(conv3(P))), which is an invariant path, where LK(·) is the LeakyReLU activation function and BN(·) is the batch normalization operation; at the same time, the normalized residual feature map P is processed by recursively updating sub-units (global average pooling layer, third activation function layer, 1*1 convolutional layer conv1) to obtain global features. This is the recursive update path; the normalized residual feature map P is processed by the second recursive subunit (a 1*1 convolutional layer conv1 followed by a Sigmoid activation function layer) to obtain the gate weights. 1 =σ(conv1(P)); This modulates the gate weights by their respective values ​​and the output local features P. l 1 and global features The product is multiplied and then added element-wise. This process is then gated and fused with the normalized residual feature map P of the original input to obtain the output feature.

[0078] In the second round of recursion, the feature map obtained in the first round of recursion is... New global features are obtained by recursively updating sub-units (global average pooling layer, third activation function layer, and 1*1 convolutional layer conv1). To update the global features in the recursive path; then, the feature map The new gated weights are obtained by processing the second recursive subunit (a 1x1 convolutional layer conv1 followed by a Sigmoid activation function layer). The new gate weights are respectively compared with the local features P of the output. l 1 and global features The product is multiplied and then added element-wise. This process is then gated and fused with the normalized residual feature map P of the original input to obtain the output feature.

[0079] It should be noted that the parameter update method of each subsequent recursive process is consistent with that of the second recursive process, so as to achieve the purpose of cyclic recursion. Through the recursive gating mechanism, the interaction between local and global features is gradually refined through multiple iterations to form deep correction. In each iteration, the gating weight is dynamically generated from the previous step to achieve adaptive fusion and improve the accuracy of image restoration.

[0080] Step 3: Input the global context feature map into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map.

[0081] The multi-scale frequency domain attention module includes at least: multiple dilated convolutional layers, multiple frequency domain attention units, convolutional layers, and average pooling layers. These are used to extract frequency domain features from the global context feature map, improving the model's ability to extract features collaboratively in the frequency and spatial domains, thereby more accurately obtaining information about haze distribution. Therefore, the multi-scale frequency domain attention module can process the global context feature map according to a specific process to obtain the corresponding enhanced frequency domain feature map.

[0082] First, the global context feature map is input into multiple dilated convolutional layers to obtain local feature maps of different scales, where the dilation rate of each dilated convolutional layer is different.

[0083] Next, for local feature maps of different scales, the local feature maps are input into the frequency domain attention unit to obtain the corresponding local enhanced feature maps. The frequency domain attention unit is used to perform fast Fourier transform on the local feature maps and analyze the residual weights of the local feature maps to obtain the corresponding query vector, key vector, value vector, and residual weight map. The dot product of the key vector and value vector is processed by the Softmax function to obtain the weight map. The weight map is then multiplied by the query vector and processed by inverse fast Fourier transform to obtain the frequency domain weight map. The frequency domain weight map, weight map, and residual weight map are concatenated to obtain the corresponding local enhanced feature map.

[0084] The local enhanced feature maps corresponding to local feature maps of different scales are concatenated and added to the feature maps obtained by processing the global context feature map through convolutional layers and average pooling layers, respectively, to obtain the corresponding frequency domain enhanced feature map.

[0085] Specifically, Figure 6 This is a schematic diagram of an optional multi-scale frequency domain attention module. Figure 7 This is a schematic diagram of an optional frequency domain attention unit. Combined with... Figure 6 and Figure 7 The processing of the global context feature map by the multi-scale frequency domain attention module described above can be understood as follows:

[0086] First, the global context feature map is processed by three dilated convolutional layers with dilation rates of 1, 2 and 4, respectively, to obtain three local feature maps, which have different scales.

[0087] For each local feature map, it is input into the frequency domain attention unit. The frequency domain attention unit performs a fast Fourier transform on the local feature map and calculates its residual weights to obtain the corresponding query vector, key vector, value vector, and residual weight map. The dot product of the key vector and value vector is processed by the Softmax function to obtain the weight map. The weight map is then multiplied by the query vector and processed by an inverse fast Fourier transform to obtain the frequency domain weight map. The frequency domain weight map, the weight map, and the residual weight map are concatenated to obtain the corresponding local enhanced feature map.

[0088] Finally, the local enhanced feature maps corresponding to local feature maps of different scales are concatenated by channels, and the feature maps obtained by processing the global context feature map through 1*1 convolutional layers and average pooling layers are added together to obtain a more comprehensive frequency domain enhanced feature map.

[0089] Therefore, through the above processing operations, the suppression of low-frequency information related to fog and the enhancement of high-frequency details are achieved, ensuring that the frequency domain enhancement feature map can better reflect the frequency domain characteristics of the clear image, that is, reducing the blur caused by fog and lowering the low-frequency energy, while enhancing the details and high-frequency components of the image.

[0090] Step 4: Input the frequency domain enhanced feature map into the (N+1)th upsampling module of the N+1 upsampling modules to obtain the corresponding initial reconstructed feature map.

[0091] Since the frequency domain enhanced feature map is processed by the aforementioned series of modules, the feature information is condensed into a small size. Therefore, this application proposes to perform an upsampling operation on the frequency domain enhanced feature map using an upsampling module, so that the frequency domain processed feature map is restored to the same spatial size as the input image, thereby obtaining the initial reconstructed feature map. The upsampling process involves expanding the information in the frequency domain enhanced feature map in the spatial dimension, which is equivalent to inserting additional detail information at each location in the image to restore the image details obscured by fog.

[0092] It should be noted that the frequency domain enhanced feature map emphasizes high-frequency details and suppresses low-frequency components related to fog. Therefore, these enhanced details are magnified and reconstructed into the image, resulting in higher detail clarity in the output initial reconstructed feature map.

[0093] Step 5: Repeat the following steps until the target reconstructed feature map output by the first upsampling module is obtained: Input the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstructed feature map, and input the reconstructed feature map into the nth upsampling module to obtain the reconstructed feature map output by the nth upsampling module, where n is a positive integer between 1 and N.

[0094] The role of skip connections is to directly "skip" the intermediate processing layers in the downsampling process and fuse them with the later feature maps in the upsampling process. This means directly connecting low-level features (which typically contain more detailed information) and high-level features (which typically contain more semantic information), ensuring that the model can fully utilize this information at different stages of the dehazing process. Simultaneously, the detailed information lost during downsampling can be recovered from the earlier feature maps through skip connections, which is crucial for maintaining sharp edges and texture details after image dehazing.

[0095] In addition, each multi-scale feature selection module includes at least: a convolutional layer, a batch normalization layer, a first activation function layer, and compression and activation units, used for feature fusion of multi-scale features captured at different levels, and through a channel attention mechanism, enabling shallow and deep layers to focus more on their respective important features, achieving adaptive fusion of local details and global context. Therefore, the nth multi-scale feature selection module can analyze the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module according to the following process to obtain the reconstructed feature map:

[0096] First, the concatenated feature map obtained by concatenating the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module is sequentially passed through the convolutional layer, normalization layer, first activation function layer, compression and activation unit in the nth multi-scale feature selection module to obtain the channel attention feature map.

[0097] Next, the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are multiplied by the channel attention feature map to obtain the first important feature in the feature map output by the nth downsampling module and the second important feature in the reconstructed feature map output by the (n+1)th upsampling module, respectively.

[0098] Finally, the first and second most important features are concatenated to obtain the reconstructed feature map output by the nth multi-scale feature selection module.

[0099] Specifically, Figure 8 This is a schematic diagram of the structure of an optional multi-scale feature selection module, where shallow details are... Figure 2The feature map output by the third downsampling module in the process has a deep semantic meaning. Figure 2 The reconstructed feature map is output by the fourth upsampling module. Therefore, the processing of the third multi-scale feature selection module can be understood as follows: first, the feature map output by the third downsampling module and the reconstructed feature map output by the fourth upsampling module are processed through a 4*4 convolutional layer, a normalization layer, a first activation function layer, a compression and activation unit to obtain the channel attention feature map, denoted as: P′3=SE(RL(BN(conv4(concat(P3,P′ total The feature map output by the third downsampling module and the reconstructed feature map output by the fourth upsampling module are multiplied by the channel attention feature map P′3 respectively, dynamically selecting the most important information from each. Finally, the reconstructed feature map is output by concatenation, denoted as: P″3=concat(P3+μ·P′3,(1-μ)·P′3+P total )

[0100] Step 6: Analyze the target reconstruction feature map output by the first upsampling module using the output module to obtain the first predicted dehazed image corresponding to the training sample.

[0101] Specifically, the target reconstruction feature map after N+1 upsampling can be passed through the output model to obtain the first predicted dehazed image corresponding to the training sample. The output module includes at least multiple convolutional layers and a fourth activation function layer. The fourth activation function layer can be the Tham activation function, and the convolutional layers can include a 3*3 group of convolutional layers and a 1*1 group of convolutional layers.

[0102] Step S1062: Construct a loss function based on the first predicted dehazed image and sample label corresponding to each training sample, and calculate the loss function through backpropagation algorithm. If the loss function meets the preset convergence condition, the trained image dehazed model is obtained.

[0103] Based on the structure of the initial network model described above, the loss function used in this embodiment is composed of a mean squared error loss function, a structural loss function, and a frequency domain loss function. Therefore, the expression for the loss function of the image dehazing model can be written as:

[0104] L total =γ1×L mse +γ2×L ssim +γ3×L freq

[0105] In the formula, L mse Let L represent the mean squared error loss function, L represent the structural loss function, and L represent the mean squared error loss function. freqLet γ1, γ2, and γ3 represent the frequency domain loss function, where γ1, γ2, and γ3 are the weight coefficients of these three loss functions, with γ1 being greater than γ2, and γ2 being greater than γ3. These loss functions can be used to constrain the first predicted dehazed image and sample label corresponding to each training sample.

[0106] Furthermore, embodiments of this application can also load the image dehazing model network into memory. For example, the raw data of the image dehazing model can be loaded from non-volatile memory into volatile memory so that the processor can run the image dehazing model. The raw data of the image dehazing model refers to unprocessed data, which typically includes the parameters and structural data of the image dehazing model. The structural data can be the calculation relationships based on the parameters, such as the forward propagation calculation relationships between intermediate layers and between neurons. Specifically, the structural data can include the structure-related code of the image dehazing model, such as code used to perform related calculations between intermediate layers and between neurons.

[0107] In one implementation, a region in memory can be partitioned off for loading the image dehazing model, which may include a structure data storage area and a parameter storage area. The structure data storage area stores structure-related code, and the parameters referenced by it can be pointed to by pointers to the addresses of specific parameters in the parameter storage area. During the training of the image dehazing model, the parameters may need to be updated frequently, which can be done by updating the parameter values ​​in the parameter storage area.

[0108] The image dehazing model trained as described above utilizes a content-aware sparse self-attention unit within the Transformer module, which combines content awareness and recursive interaction. This unit dynamically samples the self-attention mechanism to analyze foggy regions and samples depthwise convolution to analyze fog-free regions, reducing computation and making the model more lightweight. Simultaneously, the recursive gating interactive unit within the Transformer module refines features through recursive gating, enhancing its ability to model complex fog distributions and improving image dehazing, resulting in clearer and more realistic reconstructed images. Furthermore, the multi-scale frequency domain attention module transforms the global contextual features output from the content-aware and recursive interaction Transformer module into the frequency domain to extract frequency domain features. This significantly enhances the model's ability to model fog distributions, improves its ability to handle global dependencies, low-frequency fog components, and high-frequency details, avoids misjudgments of semantic information by traditional frequency domain methods, and improves robustness and computational efficiency. Moreover, the multi-scale feature selection module further integrates shallow detail information and deep semantic information at different scales, preserving high-frequency details while integrating global fog distribution information.

[0109] Example 2

[0110] According to an embodiment of this application, an image dehazing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0111] Figure 9 This is a schematic flowchart of an image dehazing method provided according to an embodiment of this application, as shown below. Figure 9 As shown, the method includes the following steps:

[0112] Step S902: Obtain the second foggy image to be processed.

[0113] Step S904: The second foggy image is processed using a pre-trained image dehazing model to obtain the corresponding second predicted dehazed image. The training process of the image dehazing model has been specifically described in Embodiment 1 above, therefore, this embodiment will not elaborate on it further.

[0114] Specifically, the image dehazing model includes: N cascaded downsampling modules, M Transformer modules combining content awareness and recursive interaction, a multi-scale frequency domain attention module, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where M and N are both positive integers. Therefore, in this embodiment, the pre-trained image dehazing model can be used to process the second hazy image as follows to obtain the corresponding second predicted dehazed image:

[0115] Step 1: Input the second foggy image into the primary feature extraction module and N cascaded downsampling modules in the initial network model in sequence to obtain the feature map output by each downsampling module. The resolution of the feature maps output by the N cascaded downsampling modules decreases sequentially.

[0116] Step 2: Input the feature map output by the Nth downsampling module into M Transformer modules that combine content awareness and recursive interaction to obtain the global context feature map;

[0117] Step 3: Input the global context feature map into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map;

[0118] Step 4: Input the frequency domain enhanced feature map into the (N+1)th upsampling module out of the N+1 upsampling modules to obtain the corresponding initial reconstructed feature map;

[0119] Step 5: Repeat the following steps until the target reconstruction feature map output by the first upsampling module is obtained: Input the feature map output by the nth downsampling module and the reconstruction feature map output by the (n+1)th upsampling module into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstruction feature map, and input the reconstruction feature map into the nth upsampling module to obtain the target reconstruction feature map output by the nth upsampling module, where n is a positive integer between 1 and N;

[0120] Step 6: Analyze the target reconstruction feature map output by the first upsampling module using the output module to obtain the second predicted dehazed image corresponding to the second foggy image.

[0121] Furthermore, after obtaining the second predicted dehazed image corresponding to the second foggy image through the above method, the embodiments of this application can also use the second foggy image and the second predicted dehazed image to update the training sample set and sample label set of the image dehazing model, and use the updated training sample set and the updated sample label set to perform self-supervised training of the image dehazing model.

[0122] Example 3

[0123] According to an embodiment of this application, an image dehazing model training apparatus is also provided for implementing the image dehazing model training method in Embodiment 1, such as... Figure 10 As shown, the image dehazing model training device includes at least: a first acquisition module 102, a construction module 104, and a training module 106, wherein:

[0124] The first acquisition module 102 is used to acquire a training sample set and a sample label set. The training sample set includes multiple first fogged images as training samples, and the sample label set includes a first defogging image corresponding to each first fogged image as a sample label corresponding to the corresponding training sample.

[0125] Module 104 is used to build an initial network model, wherein the initial network model includes at least: M Transformer modules that combine content awareness and recursive interaction for global context feature extraction of foggy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules that combine content awareness and recursive interaction, where M is a positive integer.

[0126] Training module 106 is used to iteratively train the initial network model using the training sample set and sample label set to obtain the image dehazing model.

[0127] It should be noted that each module in the image dehazing model training device in this application embodiment corresponds one-to-one with each implementation step of the image dehazing model training method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0128] Example 4

[0129] According to an embodiment of this application, an image dehazing processing apparatus for implementing the image dehazing processing method in Embodiment 2 is also provided, such as... Figure 11 As shown, the image dehazing processing device includes at least: a second acquisition module 112 and a processing module 114, wherein:

[0130] The second acquisition module 112 is used to acquire the second foggy image to be processed;

[0131] The processing module 114 is used to process the second hazy image using a pre-trained image dehazing model to obtain a corresponding second predicted dehazed image. The training process of the image dehazing model has been specifically described in Embodiment 1 above, and therefore will not be specifically described in this embodiment.

[0132] Specifically, the image dehazing model includes: N cascaded downsampling modules, M Transformer modules combining content awareness and recursive interaction, a multi-scale frequency domain attention module, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where M and N are both positive integers.

[0133] It should be noted that each module in the image dehazing processing device in this embodiment corresponds one-to-one with each implementation step of the image dehazing processing method in embodiment 2. Since embodiment 2 has been described in detail, some details not shown in this embodiment can be referred to embodiment 2, and will not be elaborated further here.

[0134] Example 5

[0135] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the image dehazing model training method in Embodiment 1 or the image dehazing processing method in Embodiment 2.

[0136] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the image dehazing model training method in Embodiment 1 or the image dehazing processing method in Embodiment 2 by running the computer program.

[0137] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the image dehazing model training method in Embodiment 1 or the image dehazing processing method in Embodiment 2.

[0138] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the image dehazing model training method of Embodiment 1 or the image dehazing processing method of Embodiment 2 through the computer program.

[0139] Optionally, the computer program executes the following steps during runtime: obtaining a training sample set and a sample label set, wherein the training sample set includes multiple first hazy images used as training samples, and the sample label set includes a first dehazed image corresponding to each first hazy image as a sample label corresponding to the corresponding training sample; constructing an initial network model, wherein the initial network model includes at least: M Transformer modules combining content awareness and recursive interaction for global context feature extraction of hazy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules combining content awareness and recursive interaction, where M is a positive integer; iteratively training the initial network model using the training sample set and the sample label set to obtain an image dehazing model.

[0140] Optionally, the computer program executes the following steps during runtime: acquiring a second foggy image to be processed; processing the second foggy image using a pre-trained image dehazing model to obtain a corresponding second predicted dehazed image.

[0141] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 12 A hardware block diagram of a computer terminal for implementing an image dehazing model training method or an image dehazing processing method is shown. Figure 12 As shown, the computer terminal 120 may include one or more processors 1202 (shown as 1202a, 1202b, ..., 1202n in the figure) 1202 (processor 1202 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1204 for storing data, and a transmission device 1206 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 12The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, computer terminal 120 may also include... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown.

[0142] It should be noted that the aforementioned one or more processors 1202 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 120. As involved in the embodiments of this application, this data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0143] The memory 1204 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the [Subject] method in this embodiment. The processor 1202 executes various functional applications and data processing by running the software programs and modules stored in the memory 1204, thereby implementing the above-mentioned application vulnerability detection method. The memory 1204 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1204 may further include memory remotely located relative to the processor 1202, and these remote memories can be connected to the computer terminal 120 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The transmission device 1206 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 120. In one example, the transmission device 1206 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 1206 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0145] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 120.

[0146] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0152] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for training an image dehazing model, characterized in that, include: Obtain a training sample set and a sample label set, wherein the training sample set includes multiple first foggy images as training samples, and the sample label set includes a first defogging image corresponding to each first foggy image as a sample label corresponding to the corresponding training sample. Construct an initial network model, wherein the initial network model includes at least: M Transformer modules combining content awareness and recursive interaction for global context feature extraction of foggy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules combining content awareness and recursive interaction, where M is a positive integer; The initial network model is iteratively trained using the training sample set and the sample label set to obtain an image dehazing model.

2. The method according to claim 1, characterized in that, The initial network model further includes: a feature extraction module, N cascaded downsampling modules, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where N is a positive integer. The initial network model is iteratively trained using the training sample set and the sample label set to obtain an image dehazing model, including: For each training sample in the training sample set, the initial network model is used to analyze the training sample as follows to obtain the corresponding first predicted dehazed image: The training sample is sequentially input into the primary feature extraction module and N cascaded downsampling modules in the initial network model to obtain the feature map output by each downsampling module, wherein the resolution of the feature maps output by the N cascaded downsampling modules decreases sequentially; The feature map output by the Nth downsampling module is input into the M Transformer modules that combine content awareness and recursive interaction to obtain the global context feature map; The global context feature map is input into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhancement feature map; The frequency domain enhancement feature map is input into... In the (N+1)th upsampling module of the N+1 upsampling modules, the corresponding initial reconstructed feature map is obtained; the following steps are repeated until the target reconstructed feature map output by the first upsampling module is obtained: the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are input into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstructed feature map, and the reconstructed feature map is input into the nth upsampling module to obtain the reconstructed feature map output by the nth upsampling module, where n is a positive integer between 1 and N; the target reconstructed feature map output by the first upsampling module is analyzed using the output module to obtain the first predicted dehazed image corresponding to the training sample; A loss function is constructed based on the first predicted dehazed image and sample label corresponding to each training sample, and the loss function is calculated through backpropagation algorithm. When the loss function satisfies the preset convergence condition, the trained image dehazed model is obtained.

3. The method according to claim 2, characterized in that, Each downsampling module includes: a first downsampling unit, a second downsampling unit, a convolutional layer, and an average pooling layer. The first downsampling unit is composed of a first depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The second downsampling unit is composed of a second depthwise separable convolutional layer, a normalization layer, and a first activation function layer stacked together. The convolutional kernel of the first depthwise separable convolutional layer is larger than that of the second depthwise separable convolutional layer. The training samples are sequentially input into the primary feature extraction module and N cascaded downsampling modules within the initial network model to obtain the feature map output by each downsampling module, including: The training samples are input into the primary feature extraction module within the initial network model to obtain an initial local feature map; For the first downsampling module among the N downsampling modules, the first local feature map obtained by the first downsampling unit in the first downsampling module processing the initial local feature map is concatenated with the second local feature map obtained by the second downsampling unit in the first downsampling module processing the first local feature map. The concatenated result is added to the feature map after the initial local feature map is processed by the average pooling layer in the first downsampling module, and then processed by the convolutional layer in the first downsampling module to obtain the feature map output by the first downsampling module. For the N downsampling modules other than the first downsampling module, the first local feature map obtained by the first downsampling unit in the current downsampling module processing the feature map output by the previous downsampling module is concatenated with the second local feature map obtained by the second downsampling unit in the current downsampling module processing the first local feature map. The concatenated result is then added to the feature map output by the previous downsampling module after processing by the average pooling layer in the current downsampling module. Finally, the feature map is processed by the convolutional layer in the current downsampling module to obtain the feature map output by the current downsampling module.

4. The method according to claim 2, characterized in that, Each of the aforementioned Transformer modules combining content awareness and recursive interaction includes at least: a normalization layer, a content-aware sparse self-attention unit, and a recursive gated interaction unit. The feature map output from the Nth downsampling module is input to the M Transformer modules combining content awareness and recursive interaction to obtain the corresponding global context feature map, including: For the first Transformer module combining content awareness and recursive interaction among the M Transformer modules combining content awareness and recursive interaction, the feature map output by the Nth downsampling module is sequentially input into the normalization layer and the content-aware sparse self-attention unit in the first Transformer module combining content awareness and recursive interaction to obtain the content-sensitive feature map output by the content-aware sparse self-attention unit, and residually concatenated with the feature map output by the Nth downsampling module to obtain the corresponding residual feature map; the residual feature map is sequentially input into the normalization layer and the recursive gated interaction unit in the first Transformer module combining content awareness and recursive interaction to obtain the interaction fusion feature map output by the recursive gated interaction unit, and residually concatenated with the residual feature map to obtain the corresponding global context feature map; For the M Transformer modules combining content awareness and recursive interaction (excluding the first one), the global context feature map output by the previous Transformer module is sequentially input into the normalization layer and content-aware sparse self-attention unit in the current Transformer module, resulting in a content-sensitive feature map output by the content-aware sparse self-attention unit. This content-sensitive feature map is then residually concatenated with the global context feature map output by the previous Transformer module to obtain a corresponding residual feature map. This residual feature map is then sequentially input into the normalization layer and recursively gated interactive unit in the current Transformer module, resulting in an interaction fusion feature map output by the recursively gated interactive unit. This interaction fusion feature map is then residually concatenated with the residual feature map to obtain a corresponding global context feature map.

5. The method according to claim 4, characterized in that, The content-aware sparse self-attention unit includes at least: a convolutional layer, an average pooling layer, a second activation function layer, a deep convolutional layer, and a multi-head self-attention layer. The global context feature map output by the previous Transformer module combining content awareness and recursive interaction is sequentially input into the normalization layer and the content-aware sparse self-attention unit in the current Transformer module combining content awareness and recursive interaction, resulting in a content-sensitive feature map output by the content-aware sparse self-attention unit, including: The global context feature map output by the previous Transformer module that combines content awareness and recursive interaction is input into the normalization layer in the current Transformer module that combines content awareness and recursive interaction to obtain a normalized global context feature map. The normalized global context feature map is processed by the convolutional layer, the average pooling layer, and the second activation function layer in the content-aware sparse self-attention unit of the Transformer module that combines content awareness and recursive interaction, to obtain the content-aware mask score of each pixel in the normalized global context feature map. Based on the relationship between the content-aware mask score of each pixel in the normalized global context feature map and a preset threshold value, the normalized global context feature map is divided into foggy regions and fog-free regions. The feature map of the foggy region is analyzed using the multi-head self-attention layer, and the feature map of the fog-free region is analyzed using the deep convolutional layer, to obtain the content-sensitive feature map output by the content-aware sparse self-attention unit.

6. The method according to claim 4, characterized in that, The recursive gated interactive unit includes: a first recursive subunit, a second recursive subunit, and a recursive update subunit. The first recursive subunit is composed of a convolutional layer, a batch normalization layer, and a first activation function layer stacked together. The second recursive subunit is composed of a convolutional layer and a second activation function layer stacked together. The recursive update subunit is composed of a global average pooling layer, a third activation function layer, and a convolutional layer stacked together. The residual feature map is sequentially input into the normalization layer and the recursive gated interactive unit in the current Transformer module combining content awareness and recursive interaction to obtain the interaction fusion feature map output by the recursive gated interactive unit, including: The residual feature map is input into the normalization layer in the Transformer module that combines content awareness and recursive interaction to obtain a normalized residual feature map. In the first round of recursion, the normalized residual feature map is input into the first recursive subunit of the Transformer module that combines content awareness and recursive interaction to obtain the corresponding local feature map; the residual feature map is input into the second recursive subunit and the recursive update subunit of the Transformer module that combines content awareness and recursive interaction to obtain the gating weight and the global feature map; the gating weight is multiplied by the local feature map and the global feature map obtained in the first round of recursion, and then added element by element; the feature map obtained by concatenating the summed feature map with the residual of the normalized residual feature map is used as the input of the second round of recursion. In each subsequent recursive round, the feature map output from the previous recursive round is input into the second recursive subunit and the recursive update subunit in the recursive gating interactive unit of the Transformer module that combines content awareness and recursive interaction, to obtain new gating weights and a new global feature map. The new gating weights are used to multiply the feature map output from the previous recursive round and the new global feature map, and then the elements are added together. The feature map obtained by concatenating the summed feature map with the normalized residual feature map is used as the input for the next recursive round. The feature map obtained from the last round of recursion is used as the interactive fusion feature map output by the recursive gating interactive unit in the current Transformer module that combines content awareness and recursive interaction.

7. The method according to claim 2, characterized in that, The multi-scale frequency domain attention module includes at least: multiple dilated convolutional layers, multiple frequency domain attention units, convolutional layers, and average pooling layers. The global context feature map is input into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map, including: The global context feature map is input into multiple dilated convolutional layers to obtain local feature maps of different scales, wherein the dilation rate of each dilated convolutional layer is different. For local feature maps of different scales, the local feature maps are input into the frequency domain attention unit to obtain corresponding local enhanced feature maps. The frequency domain attention unit performs a Fast Fourier Transform on the local feature maps and analyzes the residual weights of the local feature maps to obtain the corresponding query vector, key vector, value vector, and residual weight map. The dot product of the key vector and the value vector is processed by the Softmax function to obtain a weight map. The weight map is then multiplied by the query vector and subjected to an inverse Fast Fourier Transform to obtain a frequency domain weight map. The frequency domain weight map, the weight map, and the residual weight map are concatenated to obtain the corresponding local enhanced feature map. The local enhanced feature maps corresponding to local feature maps of different scales are concatenated by channels, and then added to the feature maps obtained by processing the global context feature map through convolutional layers and average pooling layers, respectively, to obtain the corresponding frequency domain enhanced feature map.

8. The method according to claim 2, characterized in that, The multi-scale feature selection module includes at least: a convolutional layer, a batch normalization layer, a first activation function layer, and a compression and activation unit. The feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are input to the nth multi-scale feature selection module, which is skip-connected to the nth downsampling module, to obtain the corresponding reconstructed feature map, including: The concatenated feature map, obtained by concatenating the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module, is sequentially passed through the convolutional layer, normalization layer, first activation function layer, compression and activation unit in the nth multi-scale feature selection module to obtain the channel attention feature map. The feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module are multiplied by the channel attention feature map to obtain the first important feature in the feature map output by the nth downsampling module and the second important feature in the reconstructed feature map output by the (n+1)th upsampling module, respectively. The first important feature and the second important feature are concatenated to obtain the reconstructed feature map output by the nth multi-scale feature selection module.

9. The method according to claim 2, characterized in that, The output module includes at least multiple convolutional layers and a fourth activation function layer.

10. An image dehazing method, characterized in that, include: Obtain the second foggy image to be processed; The second foggy image is processed using a pre-trained image dehazing model to obtain a corresponding second predicted dehazing image, wherein the image dehazing model is trained using the image dehazing model training method described in any one of claims 1 to 9.

11. The method according to claim 10, characterized in that, The image dehazing model includes: N cascaded downsampling modules, M Transformer modules combining content awareness and recursive interaction, a multi-scale frequency domain attention module, N multi-scale feature selection modules, N+1 upsampling modules, and an output module, where M and N are both positive integers. The pre-trained image dehazing model is used to process the second hazy image to obtain the corresponding second predicted dehazed image, including: The second foggy image is sequentially input into the primary feature extraction module and N cascaded downsampling modules in the initial network model to obtain the feature map output by each downsampling module, wherein the resolution of the feature maps output by the N cascaded downsampling modules decreases sequentially. The feature map output by the Nth downsampling module is input into the M Transformer modules that combine content awareness and recursive interaction to obtain the global context feature map; The global context feature map is input into the multi-scale frequency domain attention module to obtain the corresponding frequency domain enhanced feature map; The frequency domain enhanced feature map is input into the (N+1)th upsampling module among the N+1 upsampling modules to obtain the corresponding initial reconstructed feature map; Repeat the following steps until the target reconstructed feature map output by the first upsampling module is obtained: input the feature map output by the nth downsampling module and the reconstructed feature map output by the (n+1)th upsampling module into the nth multi-scale feature selection module that is skip-connected to the nth downsampling module to obtain the corresponding reconstructed feature map, and input the reconstructed feature map into the nth upsampling module to obtain the reconstructed feature map output by the nth upsampling module, where n is a positive integer between 1 and N; The output module is used to analyze the target reconstruction feature map output by the first upsampling module to obtain the second predicted dehazed image corresponding to the second foggy image.

12. The method according to claim 10, characterized in that, After obtaining the corresponding second predicted dehazed image, the method further includes: The training sample set and sample label set of the image dehazing model are updated using the second foggy image and the second predicted dehazing image, and the image dehazing model is then trained in a self-supervised manner using the updated training sample set and the updated sample label set.

13. An image dehazing model training device, characterized in that, include: The first acquisition module is used to acquire a training sample set and a sample label set, wherein the training sample set includes multiple first foggy images as training samples, and the sample label set includes a first defogging image corresponding to each first foggy image as a sample label corresponding to the corresponding training sample. A construction module is used to construct an initial network model, wherein the initial network model includes at least: M Transformer modules that combine content awareness and recursive interaction for global context feature extraction of foggy images, and a multi-scale frequency domain attention module for frequency domain feature extraction of the global context feature maps output by the M Transformer modules that combine content awareness and recursive interaction, where M is a positive integer; The training module is used to iteratively train the initial network model using the training sample set and the sample label set to obtain an image dehazing model.

14. An image dehazing processing apparatus, characterized in that, include: The second acquisition module is used to acquire the second foggy image to be processed; The processing module is used to process the second foggy image using a pre-trained image dehazing model to obtain a corresponding second predicted dehazing image, wherein the image dehazing model is trained using the image dehazing model training method described in any one of claims 1 to 9.

15. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the image dehazing model training method according to any one of claims 1 to 9 or the image dehazing processing method according to any one of claims 10 to 12.

16. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute, via the computer program, the image dehazing model training method of any one of claims 1 to 9 or the image dehazing processing method of any one of claims 10 to 12.

Images