Lightweight image defogging method, model, medium and system

Abstract

The invention discloses a lightweight image defogging method, model, medium and system, and the method comprises the following steps: obtaining a foggy image, extracting the image features of the foggy image, respectively inputting the image features into at least three different fogless image prediction models, obtaining at least three initial fogless images, and carrying out the defogging of the fogless images. The model parameters of the at least three fogless image prediction models are different, a weight map of each initial fogless image is generated according to the image features, weighted summation is carried out on the initial fogless images according to all the weight maps, and a final fogless image is generated. The method can reduce the complexity of the model, saves computing resources, and can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a lightweight image defogging method, model, medium and system. Background Art

[0002] With the development of deep learning technology, vision systems such as intelligent monitoring and drone inspection are becoming increasingly common. These systems rely on high-quality visual input for accurate image analysis and decision support. However, in hazy weather, the clarity and contrast of the image are significantly reduced, which seriously affects the performance of the vision system. Image dehazing methods aim to remove haze from images and restore clear, high-quality images.

[0003] Most existing image dehazing methods improve the dehazing performance of the model by increasing the complexity of the model, which often requires a large amount of computing resources and is difficult to meet the needs of real-time processing or resource-constrained environments. Summary of the invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a lightweight image defogging method, model, medium and system, which can reduce the complexity of the model and save computing resources.

[0005] In one aspect, the present invention provides a lightweight image defogging method, comprising the following steps:

[0006] Acquire a foggy image and extract image features of the foggy image;

[0007] Inputting the image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of the at least three haze-free image prediction models are different;

[0008] Generating a weight map for each initial haze-free image according to the image features;

[0009] The initial haze-free image is weightedly summed according to the weight map to generate a final haze-free image.

[0010] Optionally, extracting image features of the foggy image specifically includes:

[0011] Extracting image features of the foggy image at several scales;

[0012] Image features of all scales are fused to obtain image features of the foggy image.

[0013] Optionally, the at least three different haze-free image prediction models include at least two haze-free image prediction models of the first type and at least one haze-free image prediction model of the second type; and the inputting the image features into the at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images specifically includes:

[0014] Inputting the image features into at least two of the first-type haze-free image prediction models to obtain a first-type initial haze-free image output by each of the first-type haze-free image prediction models; at least two of the first-type haze-free image prediction models have different model parameters;

[0015] The image features are input into at least one of the second-type haze-free image prediction models to obtain a second-type initial haze-free image output by each of the second-type haze-free image prediction models; the first-type initial haze-free image has a different model structure from the second-type haze-free image prediction model.

[0016] Optionally, inputting the image features into at least two of the first-type haze-free image prediction models to obtain the first-type initial haze-free image output by each of the first-type haze-free image prediction models specifically includes:

[0017] The image features are respectively input into at least two of the first-type haze-free image prediction models, and first features are extracted from the image features by each of the first-type haze-free image prediction models to obtain a plurality of first extracted features; the plurality of first extracted features are stacked to obtain a first stacked feature map; and based on the first stacked feature map and the image features, a first-type initial haze-free image output by each of the first-type haze-free image prediction models is obtained.

[0018] Optionally, inputting the image features into at least one second-type haze-free image prediction model to obtain a second-type initial haze-free image output by each second-type haze-free image prediction model specifically includes:

[0019] The image features are input into at least one second type of haze-free image prediction model, and the residuals of the image features are separately extracted through each of the second type of haze-free image prediction models. Based on the residuals and the haze image, a second type of initial haze-free image output by each second type of haze-free image prediction model is obtained.

[0020] Optionally, at least three different haze-free image prediction models are trained by the following method:

[0021] Obtaining a foggy sample image and a fog-free label image corresponding to the foggy sample image, and extracting image sample features of the foggy sample image;

[0022] Inputting the image sample features into at least three initial haze-free image prediction models respectively to obtain at least three initial haze-free sample images;

[0023] Generating a weighted sample map for each initial haze-free sample image according to the image sample features; performing weighted summation on the initial haze-free sample images according to the weighted sample map to generate a final haze-free sample image;

[0024] The parameters of the initial model for predicting the fog-free image are updated according to the initial fog-free sample image, the fog-free label image and / or the final fog-free sample image until preset requirements are met, thereby obtaining at least three trained fog-free image prediction models.

[0025] Optionally, the at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model, and the at least two first-type fog-free image prediction models include at least one first-loss initial fog-free image prediction model and at least one second-loss initial fog-free image prediction model; updating the parameters of the initial fog-free image prediction model according to the initial fog-free sample image, the fog-free label image and / or the final fog-free sample image specifically includes:

[0026] If a first instruction is received, a first loss function is generated according to the fog-free sample image and the fog-free label image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function; the first instruction is used to characterize the initial fog-free image prediction model or the second type of fog-free image prediction model that obtains the first loss;

[0027] If a second instruction is received, a first loss function is generated based on the fog-free sample image and the fog-free label image, a second loss function is generated based on the final fog-free sample image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function and the second loss function; the second instruction is used to characterize the initial fog-free image prediction model that obtains the second loss.

[0028] On the other hand, the present invention provides a lightweight image defogging model, including a feature extraction module, a multi-model prediction module, a weight map generation module and a final prediction module: wherein,

[0029] The feature extraction module is used to obtain a foggy image and extract image features of the foggy image;

[0030] The multi-model prediction module is used to input the image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of at least three haze-free image prediction models are different;

[0031] The weight map generation module is used to generate a weight map of each initial haze-free image according to the image features;

[0032] The final prediction module is used to perform weighted summation on the initial haze-free image according to the weight map to generate a final haze-free image.

[0033] On the other hand, the present invention provides a computer-readable storage medium storing a program executable by a processor, wherein the program executable by the processor is used to execute the lightweight image defogging method described above when executed by the processor.

[0034] On the other hand, the present invention provides a lightweight image defogging system, comprising:

[0035] at least one processor;

[0036] at least one memory for storing at least one program;

[0037] When the at least one program is executed by the at least one processor, the at least one processor implements the lightweight image defogging method as described above.

[0038] The implementation of the present invention includes the following beneficial effects: the present invention obtains foggy images, extracts image features of the foggy images, inputs the image features into several different fog-free image prediction models respectively, obtains several initial fog-free images, the model parameters of the several fog-free image prediction models are different, generates weight maps for each initial fog-free image respectively according to the image features, performs weighted summation on the initial fog-free images according to all the weight maps, generates a final fog-free image, shares the extracted image features through multiple different fog-free image prediction models, and predicts the fog-free image using multiple different fog-free image prediction models, and performs weighted fusion on the fog-free images obtained by multiple different fog-free image prediction models by using weight map prediction, can adaptively handle various complex foggy environments, reduce the complexity of the model, save computing resources, can quickly and accurately remove haze in the image, and improve the visual effect of the image and its usability in the visual system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of the steps of a lightweight image defogging method provided by the present invention;

[0040] Figure 2 It is an execution flow chart of a lightweight image defogging method provided by the present invention;

[0041] Figure 3 It is a flowchart of steps for extracting image features of a foggy image provided by the present invention;

[0042] Figure 4It is a flowchart of steps for obtaining three initial fog-free images provided by the present invention;

[0043] Figure 5 It is a flowchart of the steps of training a fog-free image prediction model provided by the present invention;

[0044] Figure 6 It is a flow chart of the steps of updating the parameters of the initial model for predicting a fog-free image provided by the present invention;

[0045] Figure 7 It is a network structure diagram of a lightweight image defogging model provided by the present invention;

[0046] Figure 8 It is a network structure diagram of a receptive field adaptive feature extraction block provided by the present invention;

[0047] Fig. 9 It is a structural schematic diagram of a lightweight image defogging system provided by the present invention;

[0048] Fig.10 It is a structural schematic diagram of another lightweight image defogging system provided by the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0051] When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it is necessary to understand that the terms "first / S100", "second / S200", "third / S300", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. In the description of the present application, A and / or B represent that it can be A, B, or A and B. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0052] In order to better understand the present invention, the following technical terms are explained:

[0053] Multi-model collaborative prediction: Use the perception adaptive feature extraction block to extract model features, and integrate multiple lightweight haze-free image prediction models for collaborative prediction through adaptive estimation of weight maps, thus achieving adaptive processing of different haze intensities and types.

[0054] Image dehazing: Image dehazing is an image analysis and processing method that aims to meet the needs of specific scenarios, highlight image details and enhance image quality. In weather conditions such as fog or haze, there are a large number of tiny suspended particles in the outdoor air that refract and scatter light, causing a significant decrease in the clarity and contrast of images captured by outdoor image acquisition equipment, and even causing image color shift and a large loss of details.

[0055] It can be understood that the lightweight image defogging method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.

[0056] like Figure 1 and Figure 2 As shown, Figure 1It is a flowchart of the steps of a lightweight image dehazing method. Figure 2 The present invention provides a lightweight image defogging method, which includes the following steps:

[0057] S100: Acquire a foggy image and extract image features of the foggy image.

[0058] Specifically, after obtaining the foggy image, image features of several scales of the foggy image are extracted, and the image features of all scales are fused to obtain the image features of the foggy image. It is worth noting that the number of scales may be more than two.

[0059] S200, inputting image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of the at least three haze-free image prediction models are different.

[0060] Among them, at least three haze-free image prediction models have different model parameters.

[0061] The at least three different haze-free image prediction models include at least two haze-free image prediction models of a first type and at least one haze-free image prediction model of a second type.

[0062] At least two first-type haze-free image prediction models have the same model structure but different model parameters; and at least one second-type haze-free image prediction model has the same model structure but different model parameters.

[0063] The first type of haze-free image prediction model and the second type of haze-free image prediction model have different model structures. The first type of haze-free image prediction model includes a receptive field adaptive feature extraction block and a feature mapping layer, and the second type of haze-free image prediction model includes a receptive field adaptive feature extraction block, a feature mapping layer, and a pixel position addition layer.

[0064] Specifically, the image features are respectively input into each first type of fog-free image prediction model to obtain a first type of initial fog-free image output by each first type of fog-free image prediction model. The image features are input into at least one second type of fog-free image prediction model to obtain a second type of initial fog-free image output by each second type of fog-free image prediction model. It is worth noting that the image features are processed separately in each fog-free image prediction model to obtain the initial fog-free image outputted by it.

[0065] It is worth noting that, among the at least three different haze-free image prediction models, the ratio of the first type of haze-free image prediction model to the second type of haze-free image prediction model is not limited herein.

[0066] S300 , generating a weight map for each initial haze-free image according to image features.

[0067] Specifically, according to steps S100 and S200, a weight map is generated for each initial haze-free image based on the image features of the haze-containing image that are fused with image features at all scales.

[0068] If at least three initial haze-free images are generated in step S200 , at least three weight maps may be generated.

[0069] Two residual blocks are used to extract the features of the fused image, and the weight map mapping layer is used to map the extracted results into at least three weight maps. The features of every three channels of the weight map mapping layer are taken as a weight map. For example, if three initial haze-free images are generated, the features of the first three channels are taken as the first weight map, the features of the middle three channels are taken as the second weight map, and the features of the last three channels are taken as the third weight map.

[0070] In this step, the present invention obtains a weight map according to the image features, and uses the weight map to adaptively fuse different fog-free image prediction models, which can better adapt to a variety of different foggy scenes.

[0071] S400 , performing weighted summation on the initial haze-free image according to the weight map to generate a final haze-free image.

[0072] Specifically, finally, based on the initial haze-free image obtained by each haze-free image prediction model and the weight map corresponding to each initial haze-free image, all the initial haze-free images are integrated in a weighted summation manner to generate a final haze-free image.

[0073] The weighted summation method can be but is not limited to: multiplying the initial haze-free image obtained by each haze-free image prediction model and the weight map corresponding to each initial haze-free image pixel by pixel, obtaining the multiplication result, and adding the multiplication results to generate the final haze-free image.

[0074] In some embodiments, Figure 3 As shown, Figure 3 The present invention is a flowchart of steps for extracting image features of a foggy image. Step S100 extracts image features of a foggy image and specifically includes:

[0075] S110, extracting image features of the foggy image at several scales.

[0076] Specifically, a multi-scale feature extraction module is used to extract image features, and outputs of several layers in the middle of the multi-scale feature extraction module are selected and assigned to image features of several scales. The multi-scale feature extraction module can be, but is not limited to, an Efficient model.

[0077] For example, the first 8 layers of the Efficient model are used as the multi-scale feature extraction module, and the outputs of the 2nd, 4th and 7th layers of the Efficient model are selected as image features of three scales, which are denoted as F1, F2 and F3 respectively.

[0078] In some embodiments, an image pyramid model may also be included, and outputs of several layers in the downsampling process of the image pyramid model may be selected and assigned to image features of several scales.

[0079] S120: Fusing image features of all scales to obtain image features of the foggy image.

[0080] Specifically, in order to fuse image features of all scales, it is necessary to sample the features of each scale to the same resolution through sampling operations. In this embodiment, considering that it is ultimately necessary to predict a fog-free image with the same resolution as the foggy image, the resolution of image features of all scales is sampled to the same resolution as the foggy image. At the same time, in order to avoid introducing trainable parameters in the sampling process, bilinear interpolation is used to sample the resolution of image features of all scales to the same resolution as the foggy image. The number of channels of the sampled image features is converted into preset channels through a number of convolution kernels to obtain the converted image features, and the image features after channel conversion are fused in an additive manner to obtain the image features of the foggy image.

[0081] For example, for the above three scales of image features F1, F2 and F3, the resolutions of F1, F2 and F3 are 1 / 4, 1 / 8 and 1 / 16 of the resolution of the foggy image respectively. The resolutions of F1, F2 and F3 are sampled to the resolution of the foggy image by bilinear interpolation to obtain the sampled F1, F2 and F3. The sampled F1, F2 and F3 are input into three convolutional layers respectively. In each convolutional layer, the size of the convolution kernel is 1*1 and the output channel is 32. The three convolutional layers convert the number of channels of F1, F2 and F3 into 32 respectively. The converted F1, F2 and F3 are added together to obtain the fused image feature F f , which is the image feature of the foggy image.

[0082] In some embodiments, Figure 4 As shown, Figure 4 The invention is a flowchart of steps for obtaining three initial haze-free images. Step S200 inputs image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images, specifically including:

[0083] S210 , inputting image features into at least two first-type haze-free image prediction models to obtain a first-type initial haze-free image output by each of the first-type haze-free image prediction models.

[0084] The explanation of the first type of haze-free image prediction model is the same as the above explanation. At least two first type of haze-free image prediction models have different loss functions during training, so the parameters of each model are different. It is worth noting that since the model parameters of each first type of haze-free image prediction model are different, the output first type of initial haze-free image is also different.

[0085] Specifically, step S210 includes:

[0086] S211. Input the image features into at least two first-type haze-free image prediction models respectively, and perform first feature extraction on the image features through each first-type haze-free image prediction model to obtain a plurality of first extracted features; stack the plurality of first extracted features to obtain a first stacked feature map; and obtain a first-type initial haze-free image output by each first-type haze-free image prediction model based on the first stacked feature map and the image features.

[0087] Specifically, the image features are input into each first type of haze-free image prediction model respectively. The first type of haze-free image prediction model uses several cascaded convolutional layers to perform first feature extraction on the image features. The output of each convolutional layer is assigned to the first extracted feature to obtain several first extracted features. For example, four cascaded convolutional layers are used to extract the features of the image features. The output of each convolutional layer is used as the first extracted features X1, X2, X3 and X4 respectively. The convolution kernel size of the first two convolutional layers is 3*3, and the output channel is 32, which are used to extract local features. The convolution kernel size of the last two convolutional layers is 11*11, and the output channel is 32. Therefore, X1 and X2 contain more local information, and X3 and X4 contain more global information.

[0088] A plurality of first extracted features are integrated in a stacked manner to obtain a first stacked feature map with a plurality of channels. For example, the aforementioned X1, X2, X3 and X4 are integrated in a stacked manner to obtain a feature map with 128 channels.

[0089] The weight of each channel of the first stacked feature map is dynamically calculated according to the first extracted features, the first stacked feature map is input into the channel attention mechanism layer, and the weight of each channel is used to weight the first stacked feature map to obtain a weighted feature map. For example, the first extracted features X1, X2, X3 and X4 are substituted into formula (1) to obtain the weight α of all channels:

[0090] α=S igmoid(f c2 (f c1 (G A ([X1,X2,X3,X4])))) (1)

[0091] Among them, [·] indicates feature stacking along the channel dimension; G A (·) represents global average pooling; f c1 (·) and f c2 (·) are all fully connected layers, f c1 The output dimension of (·) is 16, and f c2 The output dimension of (·) is 128; S igmoid (·) represents the Sigmoid activation function, and α is a vector with a dimension of 128, representing the weights of the 128 channels of the first stacked feature map. Since α is a variable related to the first extracted feature, α is not static, but dynamically generated according to the first extracted feature. α can adaptively select channels that are beneficial to defogging, that is, give a relatively large weight to channels that are beneficial to defogging, which can improve the defogging effect.

[0092] After obtaining the weighted feature map, a convolution layer (channel transformation layer) is used to reduce the dimension of the weighted feature map to a preset dimension to obtain a feature map with reduced dimension, and a residual layer (residual operation) is used to add the feature map with reduced dimension and the aforementioned image features (input of the first haze-free image prediction model) to obtain a fused feature map. For example, a convolution layer with a convolution kernel size of 1*1 and an output channel of 32 is used as a channel transformation layer, and the channel transformation layer reduces the dimension of the weighted feature map to 32 to obtain a feature map with reduced dimension.

[0093] The fused feature map is mapped into an initial haze-free image of the first type using a feature mapping layer.

[0094] S220: Input the image features into at least one second-type haze-free image prediction model to obtain a second-type initial haze-free image output by each second-type haze-free image prediction model.

[0095] Among them, the second type of haze-free image prediction model includes a receptive field adaptive feature extraction block, a feature mapping layer and a pixel position addition layer.

[0096] Specifically, the processing process of image features by the receptive field adaptive feature extraction block of the second type of haze-free image prediction model is the same as the processing process of image features by the above-mentioned first type of haze-free image prediction model. The difference is that the feature mapping layer of the second type of haze-free image prediction model maps the fused feature map to the residual between the image features (the input of the second type of haze-free image prediction model) and the fused feature map.

[0097] The pixel position addition layer of the second type of fog-free image prediction model fuses the residual with the foggy image (obtained in step S100) to obtain a second type of initial fog-free image, wherein the fusion method can be but is not limited to pixel-by-pixel position addition.

[0098] In steps S210-S220, the receptive field adaptive feature extraction block of the present method has a dynamic receptive field, and can adaptively adjust the receptive field through the attention mechanism according to different inputs, thereby having better feature extraction capabilities.

[0099] like Figure 5 As shown, Figure 5 The flowchart is a step of training a haze-free image prediction model. In step S200, several different haze-free image prediction models are trained by the following method:

[0100] S230: Obtain a foggy sample image and a fog-free label image corresponding to the foggy sample image, and extract image sample features of the foggy sample image.

[0101] Specifically, an image defogging database is obtained, which contains a number of foggy sample images and their corresponding fog-free label images. In this embodiment, the ITS database is used, including 1,339 fog-free label images and their corresponding 13,990 foggy sample images. The foggy sample images are generated by simulation from the corresponding fog-free label images.

[0102] The process of extracting image sample features of the foggy sample image is the same as step S100 and its subordinate steps, except that the object is the foggy sample image, which will not be repeated here.

[0103] S240 , inputting the image sample features into at least three initial haze-free image prediction models respectively to obtain at least three initial haze-free sample images.

[0104] Among them, the initial model for haze-free image prediction is a model that has not been trained for dehazing.

[0105] Specifically, the process of inputting the image sample features into at least three initial haze-free image prediction models to obtain at least three initial haze-free sample images is the same as the process of the aforementioned steps S210-S220, except that the input is image sample features, which will not be repeated here.

[0106] S250 , generating a weighted sample map for each initial haze-free sample image according to the image sample features; performing weighted summation on the initial haze-free sample images according to all the weighted sample maps to generate a final haze-free sample image.

[0107] Specifically, the processing process of this step is the same as the processing process of S300-S400, except that this step processes the image sample features and the initial fog-free sample image, which will not be described in detail here.

[0108] S260, updating the parameters of the initial model for predicting the fog-free image according to the initial fog-free sample image, the fog-free label image and / or the final fog-free sample image until a preset condition is met, thereby obtaining at least three trained fog-free image prediction models.

[0109] Specifically, according to the description of steps S210-220, the trained fog-free image prediction model may have different parameters, such as each first type of fog-free image prediction model, or may have different structures, such as a first type of fog-free image prediction model and a second type of fog-free image prediction model. In this embodiment, the fog-free image prediction initial model may be trained by using different loss functions during the training process so that the parameters of the trained fog-free image prediction model are different, so that the model can better adapt to a more complex defogging environment.

[0110] like Figure 6 As shown, Figure 6 The invention is a flowchart of the steps of updating the parameters of the initial model for predicting a fog-free image, wherein at least three different fog-free image prediction models include at least two fog-free image prediction models of the first type and at least one fog-free image prediction model of the second type, wherein the at least two fog-free image prediction models of the first type include at least one initial fog-free image prediction model of a first loss and at least one initial fog-free image prediction model of a second loss, and the ratio of the initial fog-free image prediction model of the first loss and the initial fog-free image prediction model of the second loss is not limited, and step S260 specifically includes:

[0111] S261a: If the first instruction is received, generate a first loss function according to the fog-free sample image and the fog-free label image, and use the first loss function to update the parameters of the initial model for fog-free image prediction.

[0112] The first instruction is used to characterize the initial fog-free image prediction model of the first loss or the second type of fog-free image prediction model. For example, the first instruction is used to characterize the first fog-free image prediction model or the third fog-free image prediction model.

[0113] Specifically, for the first loss haze-free image prediction model and the second type haze-free image prediction model with different structures, the first loss function L1 is generated according to the haze-free sample image Y1 (the output of the initial model) and the haze-free label image Y2, as shown in formula (2):

[0114] L1(Y1,Y2)=‖Y1-Y2‖1 (2)

[0115] Then, based on the first loss function L1, the parameters of the initial model for predicting the haze-free image are updated using back propagation.

[0116] If multiple initial haze-free image prediction models with the first loss are included, multiple haze-free image prediction models can be obtained according to different indicators such as training accuracy.

[0117] S261b. If a second instruction is received, a first loss function is generated according to the fog-free sample image and the fog-free label image, a second loss function is generated according to the final fog-free sample image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function and the second loss function.

[0118] The second instruction is used to represent the initial haze-free image prediction model of the second loss. For example, the second instruction is used to represent the aforementioned second haze-free image prediction model.

[0119] Specifically, if the second instruction is received, the first loss function L1 is generated according to the fog-free sample image and the fog-free label image, and the second loss function L is generated according to the final fog-free sample image Y3. GAN , as shown in formula (3):

[0120] L GAN (Y3)=-D(Y3) (3)

[0121] Where D(·) represents the discriminator, which consists of a cascade of multiple convolutional layers, an average pooling layer, and a Sigmoid activation function. The number of convolutional layers can be set as needed.

[0122] Using the first loss function L1 and the second loss function L GAN The combination updates the parameters of the initial model for haze-free image prediction.

[0123] In some embodiments, if a first fog-free image prediction model, a second fog-free image prediction model, a third fog-free image prediction model…the Nth fog-free image prediction model are included, K1 initial models with the same structure as the first type of fog-free image prediction model can be selected, which use the aforementioned first loss function L1 as the loss function, and different preset conditions can be set to obtain K1 different fog-free image prediction models; K2 initial models with the same structure as the second type of fog-free image prediction model, which also use the aforementioned first loss function L1 as the loss function, and different preset conditions can be set to obtain K2 different fog-free image prediction models; the remaining N-K1-K2 initial models can have the same model structure as the first type of fog-free image prediction model, and use the aforementioned first loss function L1 and the second loss function L GANTogether as the loss function, different preset conditions can be set to obtain N-K1-K2 different haze-free image prediction models.

[0124] In some embodiments, the present invention provides different fog-free image prediction models including two first-type fog-free image prediction models and one second-type fog-free image prediction model, the two first-type fog-free image prediction models are respectively a first fog-free image prediction model and a second fog-free image prediction model, and one second-type fog-free image prediction model is a third fog-free image prediction model. Image features are separately input into the first fog-free image prediction model, the second fog-free image prediction model and the third fog-free image prediction model, and the above steps S210-S220 are processed to obtain a first initial fog-free image, a second initial fog-free image and a third initial fog-free image.

[0125] A first weight map of the first initial haze-free image, a second weight map of the second initial haze-free image, and a third weight map of the third initial haze-free image are generated according to the image features, and are processed according to step S400 to obtain a final haze-free image.

[0126] like Figure 7 As shown, Figure 7 is a network structure diagram of a lightweight image defogging model. In some embodiments, the present invention further provides a lightweight image defogging model, including a feature extraction module, a multi-model prediction module, a weight map generation module and a final prediction module: wherein,

[0127] A feature extraction module is used to obtain a foggy image and extract image features of the foggy image;

[0128] A multi-model prediction module is used to input image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of the at least three haze-free image prediction models are different;

[0129] A weight map generation module, used to generate a weight map of each initial haze-free image according to image features;

[0130] The final prediction module is used to perform weighted summation on the initial haze-free image according to the weight map to generate the final haze-free image.

[0131] Specifically, the feature extraction module is used to implement step S100, the multi-model prediction module is used to implement step S200, the weight map generation module is used to implement step S300, and the final prediction module is used to implement step S400. The specific process is the same as the aforementioned lightweight image dehazing method and will not be repeated here.

[0132] In some embodiments, Figure 7As shown, the feature extraction module includes a multi-scale feature extraction unit and a multi-scale feature fusion unit. The multi-scale feature fusion unit includes a bilinear interpolation layer, a convolution layer and a pixel-by-pixel position addition layer. The size of the convolution layer can be but is not limited to 1*1. Among them, the multi-scale feature extraction unit is used to extract image features of several scales of the foggy image; the multi-scale feature fusion unit is used to fuse image features of all scales to obtain image features of the foggy image. More specifically, the bilinear interpolation layer is used to sample the resolution of image features of all scales to the resolution of the foggy image; the convolution layer is used to convert the number of channels of the sampled image features to 32; the pixel-by-pixel position addition layer is used to add image features after channel conversion to obtain image features of the foggy image.

[0133] In some embodiments, the multi-model prediction module includes a first fog-free image prediction model, a second fog-free image prediction model, and a third fog-free image prediction model, wherein the first fog-free image prediction model, the second fog-free image prediction model, and the third fog-free image prediction model all include a receptive field adaptive feature extraction block and a feature mapping layer. Figure 8 As shown, Figure 8 The figure is a network structure diagram of a receptive field adaptive feature extraction block, which includes four cascaded convolutional layers, a channel attention mechanism layer, a channel transformation layer and a residual block. The functions of the receptive field adaptive feature extraction block and its sublayers are the same as those mentioned above and will not be repeated here.

[0134] The feature mapping layer may include a convolution layer with a convolution kernel size of 3*3 and an output channel of 3. The feature mapping layers of the first and second haze-free image prediction models map the output of the receptive field adaptive feature extraction block to the first initial haze-free image and the second initial haze-free image; the feature mapping layer of the third haze-free image prediction model maps the output of the receptive field adaptive feature extraction block to the residual between the output of the receptive field adaptive feature extraction block and the input (image feature) of the receptive field adaptive feature extraction block. The third haze-free image prediction model also includes a pixel position addition layer, which has the same function as described above and will not be repeated here.

[0135] In some embodiments, the weight map generation module includes two residual blocks and a weight map mapping layer, each residual block includes two cascaded convolutional layers and a residual operation layer, the convolution kernel size of the two convolutional layers is 3*3, and the output channel is 32; the weight map mapping layer includes a convolutional layer, the convolution kernel size of the convolutional layer is 3*3, the output channel is 9, the features of the first three channels are taken as the first weight map, the features of the middle three channels are taken as the second weight map, and the features of the last three channels are taken as the third weight map. If more weight maps need to be generated, the convolution kernel and output channel of the convolutional layer can be determined according to actual needs.

[0136] For the lightweight image defogging model proposed in the present invention, the multi-model prediction module and the weight map generation module share the feature extraction module. The multi-model prediction module is obtained by cascading small-size convolution kernels and large-size convolution kernels and combining the channel attention mechanism, and the defogging results of multiple sub-models are weightedly fused by weight prediction. It can not only handle various complex foggy environments, but also reduce the complexity of the model, thereby achieving lightweight and efficient image defogging.

[0137] The implementation of the present invention includes the following beneficial effects: the present invention shares the extracted image features through multiple different fog-free image prediction models, and uses multiple different fog-free image prediction models to predict fog-free images. The fog-free images obtained by multiple different fog-free image prediction models are weightedly fused by adopting a weight map prediction method, which can adaptively handle various complex foggy environments, reduce the complexity of the model, and save computing resources.

[0138] Each haze-free image prediction model of the present invention can reduce the complexity of the model by cascading a small-size convolution kernel and a large-size convolution kernel and combining it with a channel attention mechanism.

[0139] The present invention can be applied to intelligent monitoring or drone inspection. In the future, the model can be generalized to low-light image enhancement, image deraining, image super-resolution and other fields. It can be applied to intelligent video surveillance systems, automatic driving systems, drone cruise systems and other systems in severe weather.

[0140] For the lightweight image defogging model and lightweight image defogging method proposed in the present invention, the present invention uses the SOTS dataset to evaluate the defogging performance, and the evaluation results are shown in Table 1. Among them, the peak signal to noise ratio (PSNR) and the structural similarity (SSIM) are used to evaluate the similarity between the fog-free image estimated by each model and the fog-free label image, the model parameter index is used to evaluate the model size, and the multiply-accumulate operations (MACs) are used to reflect the complexity of the model performing a defogging operation. It can be seen that compared with the existing models, the All-in-one Dehazing Network (AOD-Net), the Gated Fusion Network (GFN), the Grid Dehaze Network (GDN), the Fusion Feature Attention Network (FFA-Net), the Multi-stream Fusion Network (MSFNet), the Knowledge Distilling Dehazing Network (KDDN), and the Multi-Scale Boosted Dehazing Network (MSBDN) with dense feature fusion are all existing dehazing models. From the data in the table, it can be seen that the present invention has a smaller parameter amount and lower complexity, and has a better dehazing effect.

[0141] Table 1

[0142]

[0143] In addition, the embodiment of the present application also discloses a computer program product or a computer program, and the computer program product or the computer program is stored in a computer-readable storage medium. The processor of the computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above method. Similarly, the contents in the above method embodiment are all applicable to the storage medium embodiment, and the functions specifically implemented by the storage medium embodiment are the same as those in the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method embodiment.

[0144] The present invention also provides a computer-readable storage medium, which stores a program executable by a processor. The program executable by the processor is used to implement the above method when executed by the processor.

[0145] It is understood that all or some steps and systems in the disclosed method above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0146] like Fig. 9 As shown, Fig. 9 The present invention is a structural schematic diagram of a lightweight image defogging system. The present invention also provides a lightweight image defogging system, including:

[0147] at least one processor;

[0148] at least one memory for storing at least one program;

[0149] When the at least one program is executed by the at least one processor, the at least one processor implements the lightweight image defogging method as described above.

[0150] Specifically, Fig.10 As shown, Fig.10It is a structural schematic diagram of another lightweight image defogging system. A lightweight image defogging system may also include an image acquisition device and a computer device connected to the image acquisition device, the computer device including the aforementioned at least one processor and at least one memory; wherein the image acquisition device is used to acquire foggy images and upload the foggy images to the computer device;

[0151] As for the image acquisition device, it is mainly realized by a camera or a camera, and it may specifically include at least one camera or a camera; and as for the computer device, it may be different types of electronic devices, including but not limited to desktop computers, laptops and other terminals.

[0152] It can be seen that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0153] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0154] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A lightweight image dehazing method, It is characterized in that The following steps are involved: Acquire a foggy image and extract image features of the foggy image; Inputting the image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of the at least three haze-free image prediction models are different; Generating a weight map for each initial haze-free image according to the image features; The initial haze-free image is weightedly summed according to the weight map to generate a final haze-free image.

2. The method according to claim 1, It is characterized in that The extracting the image features of the foggy image specifically includes: Extracting image features of the foggy image at several scales; Image features of all scales are fused to obtain image features of the foggy image.

3. The method according to claim 1, It is characterized in that The at least three different haze-free image prediction models include at least two haze-free image prediction models of the first type and at least one haze-free image prediction model of the second type; and the inputting the image features into the at least three different haze-free image prediction models to obtain at least three initial haze-free images specifically includes: Inputting the image features into at least two of the first-type haze-free image prediction models to obtain a first-type initial haze-free image output by each of the first-type haze-free image prediction models; at least two of the first-type haze-free image prediction models have different model parameters; The image features are input into at least one of the second-type haze-free image prediction models to obtain a second-type initial haze-free image output by each of the second-type haze-free image prediction models; the first-type initial haze-free image has a different model structure from the second-type haze-free image prediction model.

4. The method according to claim 3, It is characterized in that The step of inputting the image features into at least two of the first-type haze-free image prediction models to obtain the first-type initial haze-free image output by each of the first-type haze-free image prediction models specifically includes: The image features are respectively input into at least two of the first-type haze-free image prediction models, and first features are extracted from the image features by each of the first-type haze-free image prediction models to obtain a plurality of first extracted features; the plurality of first extracted features are stacked to obtain a first stacked feature map; and based on the first stacked feature map and the image features, an initial haze-free image of the first type output by each of the first-type haze-free image prediction models is obtained.

5. The method according to claim 3, It is characterized in that The step of inputting the image features into at least one second type of haze-free image prediction model to obtain a second type of initial haze-free image output by each second type of haze-free image prediction model specifically includes: The image features are input into at least one of the second-type haze-free image prediction models, and the residuals of the image features are separately extracted through each of the second-type haze-free image prediction models. Based on the residuals and the foggy image, an initial haze-free image of the second type output by each of the second-type haze-free image prediction models is obtained.

6. The method according to any one of claims 1 to 5, It is characterized in that At least three different haze-free image prediction models are trained using the following methods: Obtaining a foggy sample image and a fog-free label image corresponding to the foggy sample image, and extracting image sample features of the foggy sample image; Inputting the image sample features into at least three initial haze-free image prediction models respectively to obtain at least three initial haze-free sample images; Generating a weighted sample map for each initial haze-free sample image according to the image sample features; performing weighted summation on the initial haze-free sample images according to the weighted sample map to generate a final haze-free sample image; The parameters of the initial model for predicting the fog-free image are updated according to the initial fog-free sample image, the fog-free label image and / or the final fog-free sample image until preset requirements are met, thereby obtaining at least three trained fog-free image prediction models.

7. The method according to claim 6, It is characterized in that The at least three different fog-free image prediction models include at least two first-type fog-free image prediction models and at least one second-type fog-free image prediction model, and the at least two first-type fog-free image prediction models include at least one first-loss initial fog-free image prediction model and at least one second-loss initial fog-free image prediction model; the updating of the parameters of the initial fog-free image prediction model according to the initial fog-free sample image, the fog-free label image and / or the final fog-free sample image specifically includes: If a first instruction is received, a first loss function is generated according to the fog-free sample image and the fog-free label image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function; the first instruction is used to characterize the initial fog-free image prediction model or the second type of fog-free image prediction model that obtains the first loss; If a second instruction is received, a first loss function is generated based on the fog-free sample image and the fog-free label image, a second loss function is generated based on the final fog-free sample image, and the parameters of the initial model for fog-free image prediction are updated using the first loss function and the second loss function; the second instruction is used to characterize the initial fog-free image prediction model that obtains the second loss.

8. A lightweight image dehazing model, It is characterized in that It includes feature extraction module, multi-model prediction module, weight map generation module and final prediction module: The feature extraction module is used to obtain a foggy image and extract image features of the foggy image; The multi-model prediction module is used to input the image features into at least three different haze-free image prediction models respectively to obtain at least three initial haze-free images; the model parameters of at least three haze-free image prediction models are different; The weight map generation module is used to generate a weight map of each initial haze-free image according to the image features; The final prediction module is used to perform weighted summation on the initial haze-free image according to the weight map to generate a final haze-free image.

9. A computer-readable storage medium having a processor-executable program stored therein, It is characterized in that The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

10. A lightweight image dehazing system, It is characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

Images