A lightweight image defogging method, model, medium and system

Abstract

The application discloses a kind of lightweight image defogging method, model, medium and system, method includes the following steps: obtaining foggy image, extracting the image features of foggy image, the image features are respectively input into at least three different no-fog image prediction models, obtain at least three initial no-fog images, the model parameters of at least three no-fog image prediction models are different, according to the weight map of each initial no-fog image generated according to image features, according to all weight maps, the final no-fog image is generated by weighted summation to initial no-fog image.The application can reduce the complexity of model, save computing resources, and can be widely applied in computer vision technology field.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a lightweight image dehazing method, model, medium, and system. Background Technology

[0002] With the development of deep learning technology, vision systems such as intelligent surveillance and drone inspections are becoming increasingly common. These systems rely on high-quality visual input to achieve accurate image analysis and decision support. However, in hazy weather, image clarity and contrast are significantly reduced, severely impacting the performance of vision systems. Image dehazing methods aim to remove haze from images and restore clear, high-quality images.

[0003] Most existing image dehazing methods improve the dehazing performance by increasing the complexity of the model, which often requires a lot of computing resources and is difficult to meet the needs of real-time processing or resource-constrained environments. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide a lightweight image dehazing method, model, medium, and system that can reduce model complexity and save computational resources.

[0005] On one hand, the present invention provides a lightweight image dehazing method, comprising the following steps:

[0006] Acquire a foggy image and extract its image features;

[0007] The image features are input into at least three different fog-free image prediction models to obtain at least three initial fog-free images; the model parameters of the at least three fog-free image prediction models are different.

[0008] A weight map is generated for each initial fog-free image based on the image features.

[0009] The initial fog-free image is weighted and summed according to the weighted graph to generate the final fog-free image.

[0010] Optionally, extracting the image features of the foggy image specifically includes:

[0011] Extract image features at several scales from the foggy image;

[0012] Image features of the foggy image are obtained by fusing image features at all scales.

[0013] Optionally, the at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model; the step of inputting the image features into the at least three different fog-free image prediction models to obtain at least three initial fog-free images specifically includes:

[0014] The image features are input into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model; the model parameters of the at least two first-type haze-free image prediction models are different;

[0015] The image features are input into at least one second-type fog-free image prediction model to obtain an initial fog-free image of the second type output by each second-type fog-free image prediction model; the initial fog-free image of the first type has a different model structure than the fog-free image prediction model of the second type.

[0016] Optionally, the step of inputting the image features into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model specifically includes:

[0017] The image features are input into at least two first-type fog-free image prediction models respectively. The first feature is extracted from the image features individually by each first-type fog-free image prediction model to obtain several first extracted features. The several first extracted features are stacked to obtain a first stacked feature map. Based on the first stacked feature map and the image features, the initial fog-free image of the first type output by each first-type fog-free image prediction model is obtained.

[0018] Optionally, the step of inputting the image features into at least one second-type haze-free image prediction model to obtain an initial haze-free image of the second type output by each second-type haze-free image prediction model specifically includes:

[0019] The image features are input into at least one second-type haze-free image prediction model. The residuals of the image features are extracted separately by each second-type haze-free image prediction model. Based on the residuals and the hazy image, the initial haze-free image of the second type output by each second-type haze-free image prediction model is obtained.

[0020] Optionally, at least three different fog-free image prediction models are trained using the following method:

[0021] Obtain foggy sample images and corresponding fog-free label images of foggy sample images, and extract image sample features from the foggy sample images;

[0022] The image sample features are respectively input into at least three initial models for predicting hazy images to obtain at least three initial hazy sample images;

[0023] Based on the image sample features, a weighted sample map is generated for each initial haze-free sample image; the initial haze-free sample images are weighted and summed based on the weighted sample maps to generate the final haze-free sample image.

[0024] The parameters of the initial fog-free image prediction model are updated based on the initial fog-free sample image, the fog-free label image, and / or the final fog-free sample image until the preset requirements are met, resulting in at least three trained fog-free image prediction models.

[0025] Optionally, at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model, wherein the at least two first-type fog-free image prediction models include at least one initial fog-free image prediction model with a first loss and at least one initial fog-free image prediction model with a second loss; the step of updating the parameters of the initial fog-free image prediction model based on the initial fog-free sample image, the fog-free label image, and / or the final fog-free sample image specifically includes:

[0026] If the first instruction is received, a first loss function is generated based on the haze-free sample image and the haze-free label image, and the parameters of the initial haze-free image prediction model are updated using the first loss function; the first instruction is used to characterize the initial haze-free image prediction model with the first loss or the second type of haze-free image prediction model.

[0027] If the second instruction is received, a first loss function is generated based on the haze-free sample image and the haze-free label image, and a second loss function is generated based on the final haze-free sample image. The parameters of the initial model for haze-free image prediction are updated using the first loss function and the second loss function. The second instruction is used to characterize the initial haze-free image prediction model with the second loss.

[0028] On the other hand, the present invention provides a lightweight image dehazing model, including a feature extraction module, a multi-model prediction module, a weight map generation module, and a final prediction module: wherein,

[0029] The feature extraction module is used to acquire a foggy image and extract image features from the foggy image;

[0030] The multi-model prediction module is used to input the image features into at least three different fog-free image prediction models to obtain at least three initial fog-free images; the model parameters of the at least three fog-free image prediction models are different.

[0031] The weight map generation module is used to generate a weight map for each initial fog-free image based on the image features.

[0032] The final prediction module is used to perform a weighted summation of the initial fog-free image based on the weight map to generate the final fog-free image.

[0033] On the other hand, the present invention provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the aforementioned lightweight image dehazing method.

[0034] On the other hand, the present invention provides a lightweight image dehazing system, comprising:

[0035] At least one processor;

[0036] At least one memory for storing at least one program;

[0037] When the at least one program is executed by the at least one processor, the at least one processor implements the lightweight image dehazing method as described above.

[0038] The present invention offers the following advantages: By acquiring a foggy image, extracting its image features, and inputting these features into several different fog-free image prediction models, several initial fog-free images are obtained. These models have different parameters. A weight map is generated for each initial fog-free image based on its image features. The initial fog-free images are then weighted and summed based on all the weight maps to generate the final fog-free image. By sharing the extracted image features among multiple different fog-free image prediction models and using multiple different models to predict fog-free images, and employing a weighted map prediction method to weightedly fuse the fog-free images obtained from these models, the invention can adaptably handle various complex foggy environments, reduce model complexity, save computational resources, and quickly and accurately remove fog from images, improving the visual effect of the image and its usability in visual systems. Attached Figure Description

[0039] Figure 1 This is a flowchart of the steps of a lightweight image dehazing method provided by the present invention;

[0040] Figure 2 This is an execution flowchart of a lightweight image dehazing method provided by the present invention;

[0041] Figure 3 This is a flowchart of the steps for extracting image features from a foggy image provided by the present invention;

[0042] Figure 4This is a flowchart of the steps to obtain three initial haze-free images provided by the present invention;

[0043] Figure 5 This is a flowchart of the steps for training a fog-free image prediction model provided by the present invention;

[0044] Figure 6 This is a flowchart of the steps for updating the parameters of an initial model for predicting fog-free images, provided by the present invention.

[0045] Figure 7 This is a network structure diagram of a lightweight image dehazing model provided by the present invention;

[0046] Figure 8 This is a network structure diagram of an adaptive receptive field feature extraction block provided by the present invention;

[0047] Figure 9 This is a schematic diagram of the structure of a lightweight image dehazing system provided by the present invention;

[0048] Figure 10 This is a schematic diagram of another lightweight image dehazing system provided by the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0050] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0051] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first / S100," "second / S200," "third / S300," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. In the description of this application, A and / or B can mean that it can be A, it can be B, or it can be both A and B. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0052] To better understand this invention, the following technical terms are explained:

[0053] Multi-model collaborative prediction: Model features are extracted using perceptual adaptive feature extraction blocks, and multiple lightweight haze-free image prediction models are integrated for collaborative prediction through adaptive estimation of weight maps, achieving adaptive processing for different haze intensities and types.

[0054] Image dehazing: Image dehazing is an image analysis and processing method designed to meet specific scenario needs, highlight image details, and enhance image quality. In foggy or hazy weather conditions, a large number of tiny suspended particles in the outdoor air refract and scatter light, causing a significant decrease in the clarity and contrast of images captured by outdoor image acquisition equipment, and even causing color shifts and a large loss of detail.

[0055] It is understood that the lightweight image dehazing method provided in this embodiment of the invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0056] like Figure 1 and Figure 2 As shown, Figure 1This is a flowchart of the steps involved in a lightweight image dehazing method. Figure 2 This invention provides a lightweight image dehazing method, comprising the following steps:

[0057] S100. Obtain the foggy image and extract its features.

[0058] Specifically, after obtaining the foggy image, image features at several scales are extracted from the foggy image, and the image features at all scales are fused to obtain the image features of the foggy image. It is worth noting that "several" can be two or more.

[0059] S200. Input the image features into at least three different fog-free image prediction models to obtain at least three initial fog-free images; the model parameters of the at least three fog-free image prediction models are different.

[0060] Among them, at least three of the fog-free image prediction models have different model parameters.

[0061] At least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model.

[0062] At least two Type I fog-free image prediction models have the same model structure but different model parameters; at least one Type II fog-free image prediction model has the same model structure but different model parameters.

[0063] The first type of fog-free image prediction model and the second type of fog-free image prediction model have different model structures. The first type of fog-free image prediction model includes a receptive field adaptive feature extraction block and a feature mapping layer, while the second type of fog-free image prediction model includes a receptive field adaptive feature extraction block, a feature mapping layer, and a pixel position summation layer.

[0064] Specifically, image features are input into each of the first-type haze-free image prediction models to obtain the initial haze-free image of the first type output by each model. Image features are then input into at least one second-type haze-free image prediction model to obtain the initial haze-free image of the second type output by each model. It is worth noting that each haze-free image prediction model processes the image features independently to obtain its output initial haze-free image.

[0065] It is worth noting that the ratio of the first type of fog-free image prediction model to the second type of fog-free image prediction model is not limited here among at least three different fog-free image prediction models.

[0066] S300. Generate a weight map for each initial fog-free image based on the image features.

[0067] Specifically, according to steps S100 and S200, a weight map is generated for each initial hazy image by fusing the image features of the hazy image based on the image features of all scales.

[0068] If at least three initial fog-free images are generated in step S200, then at least three weighted images can be generated.

[0069] The features of the fused image are extracted using two residual blocks. The extracted results are mapped to at least three weight maps using a weight map mapping layer. The features of each three channels of the weight map mapping layer are taken as a weight map. For example, if three initial haze-free images are generated, the features of the first three channels are taken as the first weight map, the features of the middle three channels are taken as the second weight map, and the features of the last three channels are taken as the third weight map.

[0070] In this step, the present invention obtains a weight map based on image features, and uses the weight map to adaptively fuse different fog-free image prediction models, which can better adapt to a variety of different foggy scenes.

[0071] S400. The initial fog-free image is weighted and summed according to the weight map to generate the final fog-free image.

[0072] Specifically, finally, based on the initial fog-free image obtained from each fog-free image prediction model and the weight map corresponding to each initial fog-free image, all the initial fog-free images are integrated in a weighted sum manner to generate the final fog-free image.

[0073] The weighted summation method can be, but is not limited to, multiplying the initial fog-free image obtained by each fog-free image prediction model and the weight map corresponding to each initial fog-free image pixel by pixel, obtaining the multiplication result, and then summing the multiplication results to generate the final fog-free image.

[0074] In some embodiments, such as Figure 3 As shown, Figure 3 This is a flowchart of steps for extracting image features from a foggy image. Step S100, which extracts image features from a foggy image, specifically includes:

[0075] S110. Extract image features at several scales from the foggy image.

[0076] Specifically, image features are extracted using a multi-scale feature extraction module, and the outputs of several intermediate layers of the multi-scale feature extraction module are selected and assigned as image features of several scales. The multi-scale feature extraction module can be, but is not limited to, the Efficient model.

[0077] For example, the first 8 layers of the Efficient model are used as a multi-scale feature extraction module, and the outputs of the 2nd, 4th and 7th layers of the Efficient model are selected as image features at three scales, which are denoted as F1, F2 and F3 respectively.

[0078] In some embodiments, an image pyramid model may also be included, and the outputs of several layers of the image pyramid model during downsampling may be assigned to image features at several scales.

[0079] S120. The image features of all scales are fused to obtain the image features of the foggy image.

[0080] Specifically, to fuse image features at all scales, a sampling operation is needed to sample features at each scale to the same resolution. In this embodiment, considering that the final prediction needs to be a fog-free image with the same resolution as the foggy image, the resolution of image features at all scales is sampled to the same resolution as the foggy image. Simultaneously, to avoid introducing trainable parameters during the sampling process, bilinear interpolation is used to sample the resolution of image features at all scales to the same resolution as the foggy image. The number of channels of the sampled image features is converted into a preset number of channels using several convolutional kernels, resulting in converted image features. These channel-converted image features are then fused by addition to obtain the image features of the foggy image.

[0081] For example, for the image features F1, F2, and F3 at the three scales mentioned above, the resolutions of F1, F2, and F3 are 1 / 4, 1 / 8, and 1 / 16 of the resolution of the hazy image, respectively. Using bilinear interpolation, the resolutions of F1, F2, and F3 are sampled to the resolution of the hazy image, resulting in sampled F1, F2, and F3. These sampled F1, F2, and F3 are then input into three convolutional layers. Each convolutional layer has a 1*1 kernel size and 32 output channels. The three convolutional layers convert the number of channels of F1, F2, and F3 to 32. Finally, the channel-converted F1, F2, and F3 are added together to obtain the fused image feature F. f That is, the image features of a foggy image.

[0082] In some embodiments, such as Figure 4 As shown, Figure 4 This is a flowchart of the steps to obtain three initial haze-free images. Step S200 involves inputting image features into at least three different haze-free image prediction models to obtain at least three initial haze-free images, specifically including:

[0083] S210. Input the image features into at least two first-type haze-free image prediction models to obtain the first-type initial haze-free image output by each first-type haze-free image prediction model.

[0084] The explanation for the first type of haze-free image prediction model is the same as described above. At least two first-type haze-free image prediction models are trained with different loss functions, resulting in different parameters for each model. It is worth noting that because the model parameters of each first-type haze-free image prediction model are different, the initial haze-free image output by each model is also different.

[0085] Specifically, step S210 includes:

[0086] S211. Input the image features into at least two first-type fog-free image prediction models respectively, and extract the first feature from the image features individually through each first-type fog-free image prediction model to obtain several first extracted features; stack the several first extracted features to obtain a first stacked feature map; and obtain the first-type initial fog-free image output by each first-type fog-free image prediction model based on the first stacked feature map and the image features.

[0087] Specifically, image features are input into each of the first-type haze-free image prediction models. Each model uses several cascaded convolutional layers to extract first features from the image features. The output of each convolutional layer is assigned to one of the extracted features, resulting in several extracted features. For example, four cascaded convolutional layers are used to extract image features, with the output of each layer serving as the first extracted features X1, X2, X3, and X4. The first two convolutional layers have a 3x3 kernel size and 32 output channels, used for extracting local features, while the last two layers have an 11x11 kernel size and 32 output channels. Therefore, X1 and X2 contain more local information, while X3 and X4 contain more global information.

[0088] Several first extracted features are integrated in a stacked manner to obtain a first stacked feature map with several channels. For example, X1, X2, X3 and X4 are integrated in a stacked manner to obtain a feature map with 128 channels.

[0089] The weight of each channel of the first stacked feature map is dynamically calculated based on several first extracted features. The first stacked feature map is then input into the channel attention mechanism layer, and the weight of each channel is used to weight the first stacked feature map to obtain the weighted feature map. For example, substituting the first extracted features X1, X2, X3 and X4 into formula (1) yields the weight α of all channels:

[0090] α=S igmoid(f c2 (f c1 (G A ([X1,X2,X3,X4])))) (1)

[0091] Where [·] indicates feature stacking along the channel dimension; G A (·) indicates global average pooling; f c1 (·) and f c2 (·) are all fully connected layers, f c1 The output dimension of (·) is 16, f c2 The output dimension of (·) is 128; S igmoid (·) represents the Sigmoid activation function, and α is a 128-dimensional vector representing the weights of the 128 channels of the first stacked feature map. Since α is a variable related to the first extracted features, it is not static but dynamically generated based on the first extracted features. α can adaptively select channels that are beneficial for dehazing, that is, assign a larger weight to channels that are beneficial for dehazing, which can improve the dehazing effect.

[0092] After obtaining the weighted feature map, a convolutional layer (channel transformation layer) is used to reduce the dimension of the weighted feature map to a preset dimension, resulting in a dimension-reduced feature map. A residual layer (residual operation) is then used to fuse the dimension-reduced feature map with the aforementioned image features (the input of the first fog-free image prediction model) in an additive manner, yielding a fused feature map. For example, a convolutional layer with a kernel size of 1*1 and 32 output channels can be used as a channel transformation layer. This channel transformation layer reduces the dimension of the weighted feature map to 32, resulting in a dimension-reduced feature map.

[0093] The fused feature map is mapped to an initial haze-free image of type 1 using a feature mapping layer.

[0094] S220. Input the image features into at least one second-type haze-free image prediction model to obtain the second-type initial haze-free image output by each second-type haze-free image prediction model.

[0095] The second type of fog-free image prediction model includes a receptive field adaptive feature extraction block, a feature mapping layer, and a pixel position summation layer.

[0096] Specifically, the receptive field adaptive feature extraction block of the second type of fog-free image prediction model processes image features in the same way as the first type of fog-free image prediction model. The difference is that the feature mapping layer of the second type of fog-free image prediction model maps the fused feature map as the residual between the image features (the input of the second type of fog-free image prediction model) and the fused feature map.

[0097] The pixel position addition layer of the second type of fog-free image prediction model fuses the residual with the foggy image (obtained in step S100) to obtain the initial fog-free image of the second type. The fusion method can be, but is not limited to, pixel position addition.

[0098] In steps S210-S220, the receptive field adaptive feature extraction block of this method has a dynamic receptive field, which can adaptively adjust the receptive field through an attention mechanism according to different inputs, thus having better feature extraction capabilities.

[0099] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the steps for training a haze-free image prediction model. In step S200, several different haze-free image prediction models are trained using the following methods:

[0100] S230. Obtain the foggy sample image and the corresponding fog-free label image, and extract the image sample features of the foggy sample image.

[0101] Specifically, an image dehazing database is obtained, which contains several hazy sample images and their corresponding hazy-free labeled images. In this embodiment, the ITS database is used, including 1339 hazy-free labeled images and their corresponding 13990 hazy sample images. The hazy sample images are generated by simulation from the corresponding hazy-free labeled images.

[0102] The process of extracting image sample features from foggy sample images is the same as step S100 and its subordinate steps, except that the object is a foggy sample image, which will not be described in detail here.

[0103] S240. Input the image sample features into the initial prediction model of at least three haze-free images respectively to obtain at least three initial haze-free sample images.

[0104] The initial model for predicting fog-free images is one that has not been trained for defogging.

[0105] Specifically, the process of inputting image sample features into at least three initial models for predicting fog-free images to obtain at least three initial fog-free sample images is the same as the process described in steps S210-S220 above. The difference is that the input is image sample features, which will not be described again here.

[0106] S250. Generate a weighted sample map for each initial haze-free sample image based on the image sample features; sum the initial haze-free sample images based on all the weighted sample maps to generate the final haze-free sample image.

[0107] Specifically, the processing procedure in this step is the same as that in S300-S400. The difference is that this step processes the image sample features and the initial haze-free sample image, which will not be described in detail here.

[0108] S260. Update the parameters of the initial fog-free image prediction model based on the initial fog-free sample image, the fog-free label image, and / or the final fog-free sample image until the preset conditions are met, and obtain at least three trained fog-free image prediction models.

[0109] Specifically, according to the description of steps S210-220, the trained fog-free image prediction model can have different parameters, such as each first-type fog-free image prediction model, or it can have different structures, such as a first-type fog-free image prediction model and a second-type fog-free image prediction model. In this embodiment, different loss functions can be used to train the initial fog-free image prediction model during the training process, so that the parameters of the trained fog-free image prediction model are different, thus allowing the model to better adapt to more complex defogging environments.

[0110] like Figure 6 As shown, Figure 6 This is a flowchart of steps for updating the parameters of an initial model for haze-free image prediction. At least three different haze-free image prediction models are included, comprising at least two first-type haze-free image prediction models and at least one second-type haze-free image prediction model. The at least two first-type haze-free image prediction models include at least one initial haze-free image prediction model with a first loss and at least one initial haze-free image prediction model with a second loss. The ratio of the first-loss initial haze-free image prediction model to the second-loss initial haze-free image prediction model is not limited. Step S260 specifically includes:

[0111] S261a. If the first instruction is received, a first loss function is generated based on the fog-free sample image and the fog-free label image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function.

[0112] The first instruction is used to characterize either an initial fog-free image prediction model that yields a first loss or a second type of fog-free image prediction model. For example, the first instruction is used to characterize either obtaining the aforementioned first fog-free image prediction model or obtaining the aforementioned third fog-free image prediction model.

[0113] Specifically, for the first loss model with different structures and the second type of fog-free image prediction model, the first loss function L1 is generated based on the fog-free sample image Y1 (output of the initial model) and the fog-free label image Y2, as shown in formula (2):

[0114] L1(Y1,Y2)=‖Y1-Y2‖1 (2)

[0115] Then, based on the first loss function L1, the parameters of the initial model for predicting fog-free images are updated using backpropagation.

[0116] If an initial fog-free image prediction model includes multiple first-loss methods, multiple fog-free image prediction models can be obtained based on different training accuracy and other metrics.

[0117] S261b: If the second instruction is received, generate a first loss function based on the haze-free sample image and the haze-free label image, generate a second loss function based on the final haze-free sample image, and update the parameters of the initial model for predicting haze-free images using the first loss function and the second loss function.

[0118] The second instruction is used to characterize the initial haze-free image prediction model that yields the second loss. For example, the second instruction is used to characterize the aforementioned second haze-free image prediction model.

[0119] Specifically, upon receiving the second instruction, a first loss function L1 is first generated based on the haze-free sample image and the haze-free label image, and a second loss function L is generated based on the final haze-free sample image Y3. GAN As shown in formula (3):

[0120] L GAN (Y3)=-D(Y3) (3)

[0121] Where D(·) represents the discriminator, which consists of multiple convolutional layers, an average pooling layer, and a cascaded sigmoid activation function. The number of convolutional layers can be set as needed.

[0122] Using the first loss function L1 and the second loss function L GAN The parameters of the initial model for predicting fog-free images are updated by combining the results.

[0123] In some embodiments, if the model includes a first fog-free image prediction model, a second fog-free image prediction model, a third fog-free image prediction model, ..., an Nth fog-free image prediction model, then K1 initial models with the same structure as the first type of fog-free image prediction model can be selected. These models use the aforementioned first loss function L1 as the loss function, and different preset conditions can be set to obtain K1 different fog-free image prediction models. Similarly, K2 initial models with the same structure as the second type of fog-free image prediction model can also use the aforementioned first loss function L1 as the loss function, and different preset conditions can be set to obtain K2 different fog-free image prediction models. The remaining N-K1-K2 initial models can have the same model structure as the first type of fog-free image prediction model, using the aforementioned first loss function L1 and second loss function L... GANUsing these as a loss function, different preset conditions can be set to obtain N-K1-K2 different fog-free image prediction models.

[0124] In some embodiments, the present invention provides different fog-free image prediction models, including two first-type fog-free image prediction models and one second-type fog-free image prediction model. The two first-type fog-free image prediction models are respectively a first fog-free image prediction model and a second fog-free image prediction model, and the second-type fog-free image prediction model is a third fog-free image prediction model. Image features are individually input into the first fog-free image prediction model, the second fog-free image prediction model, and the third fog-free image prediction model, and the above steps S210-S220 are performed to obtain a first initial fog-free image, a second initial fog-free image, and a third initial fog-free image.

[0125] Based on the image features, a first weight map of the first initial fog-free image, a second weight map of the second initial fog-free image, and a third weight map of the third initial fog-free image are generated, and then processed according to step S400 to obtain the final fog-free image.

[0126] like Figure 7 As shown, Figure 7 This is a network structure diagram of a lightweight image dehazing model. In some embodiments, the present invention also provides a lightweight image dehazing model, including a feature extraction module, a multi-model prediction module, a weight graph generation module, and a final prediction module: wherein,

[0127] The feature extraction module is used to acquire foggy images and extract their features.

[0128] The multi-model prediction module is used to input image features into at least three different fog-free image prediction models to obtain at least three initial fog-free images; the model parameters of the at least three fog-free image prediction models are different.

[0129] The weight map generation module is used to generate a weight map for each initial haze-free image based on the image features.

[0130] The final prediction module is used to perform a weighted summation of the initial fog-free image based on the weight map to generate the final fog-free image.

[0131] Specifically, the feature extraction module is used to implement step S100, the multi-model prediction module is used to implement step S200, the weight map generation module is used to implement step S300, and the final prediction module is used to implement step S400. The specific process is the same as the aforementioned lightweight image dehazing method, and will not be repeated here.

[0132] In some embodiments, such as Figure 7As shown, the feature extraction module includes a multi-scale feature extraction unit and a multi-scale feature fusion unit. The multi-scale feature fusion unit includes a bilinear interpolation layer, a convolutional layer, and a pixel-wise summation layer. The size of the convolutional layer can be, but is not limited to, 1*1. Specifically, the multi-scale feature extraction unit extracts image features at several scales from the foggy image; the multi-scale feature fusion unit fuses the image features at all scales to obtain the image features of the foggy image. More specifically, the bilinear interpolation layer samples the resolution of the image features at all scales to the resolution of the foggy image; the convolutional layer converts the number of channels of the sampled image features to 32; and the pixel-wise summation layer adds the image features after channel conversion to obtain the image features of the foggy image.

[0133] In some embodiments, the multi-model prediction module includes a first fog-free image prediction model, a second fog-free image prediction model, and a third fog-free image prediction model, wherein each of the first, second, and third fog-free image prediction models includes a receptive field adaptive feature extraction block and a feature mapping layer. Figure 8 As shown, Figure 8 This is a network structure diagram of a receptive field adaptive feature extraction block. The receptive field adaptive feature extraction block includes four cascaded convolutional layers, one channel attention mechanism layer, one channel transform layer, and one residual block. The functions of the receptive field adaptive feature extraction block and its sub-layers are the same as those described above, and will not be repeated here.

[0134] The feature mapping layer can include a convolutional layer with a kernel size of 3*3 and 3 output channels. The feature mapping layers of the first and second haze-free image prediction models map the output of the receptive field adaptive feature extraction block to the first and second initial haze-free images. The feature mapping layer of the third haze-free image prediction model maps the output of the receptive field adaptive feature extraction block to the residual between the output of the receptive field adaptive feature extraction block and the input (image features) of the receptive field adaptive feature extraction block. The third haze-free image prediction model also includes a pixel position summing layer, which has the same function as described above and will not be repeated here.

[0135] In some embodiments, the weight map generation module includes two residual blocks and a weight map mapping layer. Each residual block includes two cascaded convolutional layers and one residual operation layer. The convolutional kernels of the two convolutional layers are 3*3, and the output channels are 32. The weight map mapping layer includes one convolutional layer with a 3*3 kernel and 9 output channels. The features of the first three channels are used as the first weight map, the features of the middle three channels are used as the second weight map, and the features of the last three channels are used as the third weight map. If more weight maps need to be generated, the convolutional kernels and output channels of the convolutional layers can be determined according to actual needs.

[0136] The lightweight image dehazing model proposed in this invention shares a feature extraction module between the multi-model prediction module and the weight map generation module. The multi-model prediction module obtains the feature by cascading small-sized convolutional kernels and large-sized convolutional kernels and combining them with a channel attention mechanism. It also uses weight prediction to weight and fuse the dehazing results of multiple sub-models. This approach can handle various complex foggy environments while reducing the complexity of the model, thus achieving lightweight and efficient image dehazing.

[0137] Implementing this invention has the following beneficial effects: This invention shares extracted image features through multiple different fog-free image prediction models, and uses multiple different fog-free image prediction models to predict fog-free images. By adopting a weighted graph prediction method, the fog-free images obtained by multiple different fog-free image prediction models are weighted and fused, which can adaptably handle various complex foggy environments, reduce model complexity, and save computing resources.

[0138] Each fog-free image prediction model of the present invention reduces model complexity by cascading small-sized convolutional kernels and large-sized convolutional kernels and combining them with a channel attention mechanism.

[0139] This invention can be applied to intelligent monitoring or drone inspection. In the future, this model can be generalized to fields such as low-light image enhancement, image deraining, and image super-resolution. It can be applied to intelligent video surveillance systems, autonomous driving systems, drone patrol systems, and other systems under severe weather conditions.

[0140] For the lightweight image dehazing model and method proposed in this invention, the SOTS dataset was used to evaluate the dehazing performance, and the evaluation results are shown in Table 1. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used to evaluate the similarity between the haze-free image estimated by each model and the haze-free label image. Model parameter metrics are used to evaluate model size, and the number of multiply-accumulate operations (MACs) reflects the complexity of the model performing one dehazing operation. As can be seen, compared to existing dehazing models such as All-in-one Dehazing Network (AOD-Net), Gated Fusion Network (GFN), GridDehazeNet (GDN), Fusion Feature Attention Network (FFA-Net), Multi-stream Fusion Network (MSFNet), Knowledge Distilling Dehazing Network (KDDN), and Multi-Scale Boosted Dehazing Network (MSBDN), this invention has fewer parameters, lower complexity, and better dehazing performance. The data in the table show that this invention has fewer parameters, lower complexity, and better dehazing performance.

[0141] Table 1

[0142]

[0143] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0144] The present invention also provides a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the above-described method.

[0145] It is understood that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0146] like Figure 9 As shown, Figure 9 This is a schematic diagram of a lightweight image dehazing system. The present invention also provides a lightweight image dehazing system, comprising:

[0147] At least one processor;

[0148] At least one memory for storing at least one program;

[0149] When the at least one program is executed by the at least one processor, the at least one processor implements the lightweight image dehazing method as described above.

[0150] Specifically, such as Figure 10 As shown, Figure 10This is a schematic diagram of another lightweight image dehazing system. A lightweight image dehazing system may also include an image acquisition device and a computer device connected to the image acquisition device. The computer device includes at least one processor and at least one memory. The image acquisition device is used to acquire a foggy image and upload the foggy image to the computer device.

[0151] The image acquisition device is mainly implemented through a camera or a webcam, and may specifically include at least one camera or webcam; while the computer device may be different types of electronic devices, including but not limited to desktop computers, laptops and other terminals.

[0152] It is evident that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0153] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A lightweight image dehazing method, characterized in that, Includes the following steps: Acquire a foggy image and extract its image features; The image features are input into at least three different fog-free image prediction models to obtain at least three initial fog-free images. The model parameters of the at least three fog-free image prediction models are different. Specifically, the at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model. The process of inputting the image features into the at least three different fog-free image prediction models to obtain at least three initial fog-free images includes: The image features are input into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model; the model parameters of the at least two first-type haze-free image prediction models are different; wherein, the step of inputting the image features into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model specifically includes: The image features are input into at least two first-type fog-free image prediction models respectively. The first feature is extracted from the image features individually by each first-type fog-free image prediction model to obtain a number of first extracted features. The number of first extracted features are stacked to obtain a first stacked feature map. Based on the first stacked feature map and the image features, the initial fog-free image of the first type output by each first-type fog-free image prediction model is obtained. The image features are input into at least one second-type haze-free image prediction model to obtain an initial haze-free image of the second type output by each second-type haze-free image prediction model; the initial haze-free image of the first type has a different model structure from the haze-free image prediction model of the second type; wherein, the step of inputting the image features into at least one second-type haze-free image prediction model to obtain an initial haze-free image of the second type output by each second-type haze-free image prediction model specifically includes: The image features are input into at least one second-type haze-free image prediction model. The residuals of the image features are extracted separately by each second-type haze-free image prediction model. Based on the residuals and the hazy image, the initial haze-free image of the second type output by each second-type haze-free image prediction model is obtained. A weight map is generated for each initial fog-free image based on the image features. The initial fog-free image is weighted and summed according to the weighted graph to generate the final fog-free image.

2. The method according to claim 1, characterized in that, The extraction of image features from the foggy image specifically includes: Extract image features at several scales from the foggy image; Image features of the foggy image are obtained by fusing image features at all scales.

3. The method according to any one of claims 1-2, characterized in that, At least three different fog-free image prediction models were trained using the following method: Obtain foggy sample images and corresponding fog-free label images of foggy sample images, and extract image sample features from the foggy sample images; The image sample features are respectively input into at least three initial models for predicting hazy images to obtain at least three initial hazy sample images; Based on the image sample features, a weighted sample map is generated for each initial haze-free sample image; the initial haze-free sample images are weighted and summed based on the weighted sample maps to generate the final haze-free sample image. The parameters of the initial fog-free image prediction model are updated based on the initial fog-free sample image, the fog-free label image, and / or the final fog-free sample image until the preset requirements are met, resulting in at least three trained fog-free image prediction models.

4. The method according to claim 3, characterized in that, At least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model. The at least two first-type fog-free image prediction models include at least one initial fog-free image prediction model with a first loss and at least one initial fog-free image prediction model with a second loss. The step of updating the parameters of the initial fog-free image prediction model based on the initial fog-free sample image, the fog-free label image, and / or the final fog-free sample image specifically includes: If the first instruction is received, a first loss function is generated based on the haze-free sample image and the haze-free label image, and the parameters of the initial haze-free image prediction model are updated using the first loss function; the first instruction is used to characterize the initial haze-free image prediction model with the first loss or the second type of haze-free image prediction model. If the second instruction is received, a first loss function is generated based on the haze-free sample image and the haze-free label image, and a second loss function is generated based on the final haze-free sample image. The parameters of the initial model for haze-free image prediction are updated using the first loss function and the second loss function. The second instruction is used to characterize the initial haze-free image prediction model with the second loss.

5. A lightweight image dehazing model, characterized in that, It includes a feature extraction module, a multi-model prediction module, a weight graph generation module, and a final prediction module: among which, The feature extraction module is used to acquire a foggy image and extract image features from the foggy image; The multi-model prediction module is used to input the image features into at least three different fog-free image prediction models to obtain at least three initial fog-free images. The model parameters of the at least three fog-free image prediction models are different, wherein the at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model. Specifically, inputting the image features into the at least three different fog-free image prediction models to obtain at least three initial fog-free images includes: The image features are input into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model; the model parameters of the at least two first-type haze-free image prediction models are different; wherein, the step of inputting the image features into at least two first-type haze-free image prediction models to obtain an initial haze-free image of the first type output by each first-type haze-free image prediction model specifically includes: The image features are input into at least two first-type fog-free image prediction models respectively. The first feature is extracted from the image features individually by each first-type fog-free image prediction model to obtain a number of first extracted features. The number of first extracted features are stacked to obtain a first stacked feature map. Based on the first stacked feature map and the image features, the initial fog-free image of the first type output by each first-type fog-free image prediction model is obtained. The image features are input into at least one second-type haze-free image prediction model to obtain an initial haze-free image of the second type output by each second-type haze-free image prediction model; the initial haze-free image of the first type has a different model structure from the haze-free image prediction model of the second type; wherein, the step of inputting the image features into at least one second-type haze-free image prediction model to obtain an initial haze-free image of the second type output by each second-type haze-free image prediction model specifically includes: The image features are input into at least one second-type haze-free image prediction model. The residuals of the image features are extracted separately by each second-type haze-free image prediction model. Based on the residuals and the hazy image, the initial haze-free image of the second type output by each second-type haze-free image prediction model is obtained. The weight map generation module is used to generate a weight map for each initial fog-free image based on the image features. The final prediction module is used to perform a weighted summation of the initial fog-free image based on the weight map to generate the final fog-free image.

6. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-4.

7. A lightweight image dehazing system, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-4.

Images