Training method of image defogging model, image defogging method and device

Abstract

The disclosure provides a training method of an image defogging model, an image defogging method and device, and relates to the technical field of computer vision, in particular to the technical field of image processing. The specific implementation scheme of the training method of the image defogging model is as follows: an original fog image and a corresponding original clear image are obtained; feature extraction is performed on the original fog image to obtain image illumination information, image fog information and a defogging image of the original fog image; image reconstruction is performed based on the image illumination information, the image fog information and the defogging image to obtain a reconstructed fog image; a first loss between the defogging image and the original clear image and a second loss between the reconstructed fog image and the original fog image are calculated; and an image defogging model is trained based on the first loss and the second loss.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, in particular to the technical field of image processing, and more particularly to a training method of an image defogging model, an image defogging method and device. BACKGROUND

[0002] With the development of computer vision technology, the application range of computer vision is also expanding, such as object detection and segmentation, object tracking, etc., and obtaining high-quality image information can effectively improve the accuracy of computer vision technology. Fog often exists in real scenes, which can cause occlusion of objects in the scene, and the image obtained in the fog scene usually needs to be processed again to obtain a clear non-fog image. SUMMARY

[0003] The present disclosure provides a training method of an image defogging model, an image defogging method and device.

[0004] According to a first aspect of the present disclosure, a training method of an image defogging model is provided, comprising:

[0005] obtaining an original fog image and a corresponding original clear image;

[0006] performing feature extraction on the original fog image to obtain image illumination information, image fog information and a defogging image of the original fog image;

[0007] performing image reconstruction based on the image illumination information, the image fog information and the defogging image to obtain a reconstructed fog image;

[0008] calculating a first loss between the defogging image and the original clear image, and a second loss between the reconstructed fog image and the original fog image;

[0009] training an image defogging model based on the first loss and the second loss.

[0010] According to a second aspect of the present disclosure, an image defogging method is provided, comprising:

[0011] obtaining an image to be processed;

[0012] performing defogging processing on the image to be processed based on an image defogging model to obtain a clear image after defogging;

[0013] The image defogging model is a model trained according to the method of the first aspect.

[0014] According to a third aspect of the present disclosure, a training device of an image defogging model is provided, comprising:

[0015] The first obtaining module is configured to obtain an original fog image and a corresponding original clear image.

[0016] The feature extraction module is configured to perform feature extraction on the original fog image to obtain image illumination information, image fog information, and a defogged image of the original fog image.

[0017] The reconstruction module is configured to perform image reconstruction based on the image illumination information, the image fog information, and the defogged image to obtain a reconstructed fog image.

[0018] The loss calculation module is configured to calculate a first loss between the defogged image and the original clear image, and a second loss between the reconstructed fog image and the original fog image.

[0019] The training module is configured to train an image defogging model based on the first loss and the second loss.

[0020] According to a fourth aspect of the present disclosure, an image defogging apparatus is provided, comprising:

[0021] The second obtaining module is configured to obtain a to-be-processed image.

[0022] The processing module is configured to perform defogging processing on the to-be-processed image based on an image defogging model to obtain a defogged clear image.

[0023] The image defogging model is a model trained according to the method of the first aspect.

[0024] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0025] at least one processor; and

[0026] a memory connected with the at least one processor; wherein

[0027] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect or the method of the second aspect.

[0028] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the first aspect or the method of the second aspect.

[0029] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to the first aspect, or implements the method according to the second aspect.

[0030] The technical solution provided by the present disclosure can effectively improve the training effect of the image defogging model by calculating two kinds of losses as two kinds of supervision signals of the image defogging model, so that the precision of the trained image defogging model is higher, and the defogging effect of the image defogging model on the foggy image is better.

[0031] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation on the present disclosure. Among them:

[0033] Figure 1 is a flowchart of a training method of an image defogging model provided by an embodiment of the present disclosure;

[0034] Figure 2a is a flowchart of feature extraction and image reconstruction in a training method of an image defogging model provided by an embodiment of the present disclosure;

[0035] Figure 2b is a flowchart of channel-level feature enhancement in a training method of an image defogging model provided by an embodiment of the present disclosure;

[0036] Figure 2c is a flowchart of pixel-level feature enhancement in a training method of an image defogging model provided by an embodiment of the present disclosure;

[0037] Figure 3 is a flowchart of an image defogging method provided by an embodiment of the present disclosure;

[0038] Figure 4 is a structural diagram of a training device of an image defogging model provided by an embodiment of the present disclosure;

[0039] Figure 5 is a structural diagram of an image defogging device provided by an embodiment of the present disclosure;

[0040] Figure 6 is a block diagram of an electronic device for implementing the training method of the image defogging model or the image defogging method of the present embodiment. DETAILED DESCRIPTION

[0041] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and include various details intended to facilitate understanding of the present disclosure. Thus, it should be apparent to those skilled in the art that various modifications and changes can be made in the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, the description is not to be construed to limit the present disclosure to the specific examples contained herein. For the purposes of clarity and the expeditious presentation of the present disclosure, the description herein has omitted certain well-known functions and constructions.

[0042] The present embodiment provides a training method of an image defogging model.

[0043] Please refer to Figure 1 , Figure 1 is a flowchart of a training method of an image defogging model provided by the present embodiment. As shown in Figure 1 , the method comprises the following steps:

[0044] Step S101, obtaining an original fog image and a corresponding original clear image.

[0045] The original fog image and the original clear image correspond to each other, the original clear image is a defogging image or a fog-free image of the corresponding original fog image, and one original clear image and the corresponding original clear image form a fog-clear image pair. Optionally, in the present embodiment, multiple fog-clear image pairs can be obtained as training images of the image defogging model.

[0046] Step S102, performing feature extraction on the original fog image to obtain image illumination information, image fog information, and a defogging image of the original fog image.

[0047] In the present embodiment, the obtained original fog image is taken as an input of the image defogging model, and feature extraction or feature estimation is performed on the original fog image based on the image defogging model, for example, illumination information and fog information of the original fog image are estimated respectively, and then image illumination information, image fog information, and a defogging image are obtained, wherein the defogging image can be an image after the original fog image is extracted from the fog information.

[0048] Optionally, feature encoding and feature decoding can be performed on the original fog image based on the image defogging model to obtain image illumination information, image fog information, and a defogging image of the original fog image.

[0049] Step S103, performing image reconstruction based on the image illumination information, the image fog information, and the defogging image to obtain a reconstructed fog image.

[0050] In this step, the image light information, the image fog information and the defogging image extracted can be reconstructed based on an image defogging model to obtain a reconstructed fog image.

[0051] Optionally, the step S103 can further include:

[0052] obtaining a first product between the image fog information and the defogging image;

[0053] obtaining a second product between the image fog information and the image light information;

[0054] reconstructing the image based on the first product and the second product to obtain a reconstructed fog image.

[0055] It should be noted that the image light information, the image fog information and the defogging image can be characterized by eigenvalues, and the first product can be obtained by the eigenvalue of the image fog information and the eigenvalue of the defogging image, the second product can be obtained by the eigenvalue of the image fog information and the eigenvalue of the image light information, and the reconstructed fog image can be the sum of the first product and the second product.

[0056] Optionally, the reconstructed fog image can be reconstructed by the following formula:

[0057] I = J*T + (1-T)*A;

[0058] wherein, I is the reconstructed fog image, J is the defogging image, T is the image fog information, and A is the image light information. It should be noted that the parameters in the above formula can be obtained based on an image defogging model, for example, the image defogging model can obtain the image light information, the image fog information and the defogging image after feature encoding and feature decoding of the original fog image, and the above parameters can be characterized by model eigenvalues or model parameters. The corresponding model parameters obtained based on the network model can refer to the related principles of the network model, and the present disclosure will not be repeated.

[0059] In the embodiments of the present disclosure, the image light information, the image fog information and the defogging image obtained by feature extraction are reconstructed to obtain the reconstructed fog image, and the loss between the reconstructed fog image and the original fog image can be calculated, and the image defogging model is trained based on the loss to improve the accuracy of the image defogging model.

[0060] Step S104, calculating a first loss between the defogging image and the original clear image, and a second loss between the reconstructed fog image and the original fog image.

[0061] The dehazed image is an image obtained by performing feature extraction on the original hazy image by using an image dehazing model. For example, the dehazed image can be an image from which the hazy information is removed. In theory, the dehazed image is expected to be consistent with the original clear image. However, after the original hazy image is processed by using the image dehazing model, the dehazed image obtained by performing feature extraction is not completely the same as the original clear image, and there is a certain difference. Therefore, the first loss between the dehazed image and the original clear image is calculated.

[0062] In addition, in the embodiment of the present disclosure, the image light information, the image hazy information, and the dehazed image obtained by feature extraction are used for image reconstruction to obtain a reconstructed hazy image. In theory, the reconstructed hazy image is also expected to be consistent with the original hazy image. However, after feature extraction and image reconstruction based on the image dehazing model, the reconstructed hazy image obtained by reconstruction is not completely the same as the original hazy image, and there is a certain difference. Therefore, the second loss between the reconstructed hazy image and the original hazy image is calculated.

[0063] In step S105, the image dehazing model is trained based on the first loss and the second loss.

[0064] It can be understood that after the first loss and the second loss are calculated, the image dehazing model is trained based on the first loss and the second loss. That is, the image dehazing model is supervised and trained based on the two supervision signals that the dehazed image should be consistent with the original clear image and the reconstructed hazy image should be consistent with the original hazy image.

[0065] The training method of the image dehazing model provided by the embodiment of the present disclosure no longer only uses the loss between the dehazed image and the original clear image as a supervision signal to train the image dehazing model, but also uses the loss between the reconstructed hazy image and the original hazy image as another supervision signal to train the image dehazing model. The relationship and influence between different components of the original hazy image can be considered, the training effect of the image dehazing model is improved, the precision of the trained image dehazing model is higher, the dehazing effect of the hazy image is better, and a clearer image can be output.

[0066] Optionally, the image dehazing model includes a first network structure, and the step S102 can include:

[0067] The original hazy image is subjected to feature extraction based on the first network structure to obtain image light information, image hazy information, and a dehazed image of the original hazy image.

[0068] In the embodiments of the present disclosure, the original fog image can be extracted based on the first network structure, and the image illumination information, the image fog information and the defogging image of the original fog image can be obtained based on the first network structure. Alternatively, the defogging image of the original fog image can be obtained based on the first network structure, and the image illumination information and the image fog information of the original fog image can be obtained based on other network structures.

[0069] The image defogging model can include a plurality of network structures, and the plurality of network structures can be independent of each other. Alternatively, the image defogging model can include a plurality of network sub-models, the first network structure is one of the network sub-models, and the first network structure can independently implement feature encoding, feature decoding and model training operations. In the embodiments of the present disclosure, the original fog image is extracted by the first network structure in the image defogging model, so that the feature calculation of the feature extraction of the image defogging model is concentrated on one of the network structures, so as to improve the processing efficiency of the image defogging model.

[0070] Optionally, the image defogging model further includes a second network structure and a third network structure, and the step S102 can include:

[0071] The original fog image is extracted based on the first network structure, the second network structure and the third network structure respectively, and the defogging image output by the first network structure, the image fog information output by the second network structure and the image illumination information output by the third network structure are obtained.

[0072] That is, the original fog image is extracted by the first network structure to obtain the defogging image output by the first network structure, the original fog image is extracted by the second network structure to obtain the image fog information output by the second network structure, and the original fog image is extracted by the third network structure to obtain the image illumination information output by the third network structure. In this way, the original fog image is processed by three network structures respectively to obtain different image feature information.

[0073] The first network structure, the second network structure and the third network structure can be network structures of the same type, but the three network structures can use different model algorithms for operation.

[0074] In the embodiments of the present disclosure, the original fog image is extracted by different network structures, and the image illumination information, the image fog information and the defogging image are obtained by operation based on different network structures. The original fog image does not need to be extracted by a network structure for multiple times, and the feature extraction efficiency of the image defogging model can be effectively improved.

[0075] Optionally, the step S102 can further include:

[0076] performing feature encoding on the original fog image to obtain image encoding information;

[0077] performing feature enhancement on the image encoding information to obtain image feature enhancement information;

[0078] performing feature decoding on the image feature enhancement information to obtain image illumination information, image fog information and a defogging image of the original fog image.

[0079] In the embodiments of the present disclosure, the above steps can be performed based on an image defogging model. For example, the image defogging model is used to first perform feature encoding on the original fog image to obtain image encoding information; then perform feature enhancement on the image encoding information based on the image defogging model to obtain image feature enhancement information; and finally perform feature decoding on the image feature enhancement information based on the image defogging model to obtain image illumination information, image fog information and a defogging image of the original fog image.

[0080] Alternatively, in the case where the image defogging model includes a first network structure, a second network structure and a third network structure, the above steps can also be performed based on the above three network structures.

[0081] Optionally, the step S102 can include:

[0082] performing feature encoding on the original fog image based on a target network structure to obtain image encoding information, the target network structure being any one of the first network structure, the second network structure and the third network structure;

[0083] performing feature enhancement on the image encoding information based on the target network structure to obtain image feature enhancement information;

[0084] performing feature decoding on the image feature enhancement information based on the target network structure to obtain target image information output by the target network structure, the target image information being any one of the image illumination information, the image fog information and the defogging image, and the target image information corresponding to the target network structure.

[0085] For example, if the target network structure is the first network structure, the original fog image is sequentially subjected to feature encoding, feature enhancement and feature decoding based on the first network structure, and a defogging image output by the first network structure is obtained; if the target network structure is the second network structure, the original fog image is sequentially subjected to feature encoding, feature enhancement and feature decoding based on the second network structure, and image fog information output by the second network structure is obtained; if the target network structure is the third network structure, the original fog image is sequentially subjected to feature encoding, feature enhancement and feature decoding based on the third network structure, and image illumination information output by the third network structure is obtained.

[0086] In the embodiments of the present disclosure, the original fog image is subjected to feature encoding, feature enhancement and feature decoding based on different network structures respectively to obtain corresponding target image information, wherein the feature enhancement operation can enhance the required image features and weaken the unnecessary image features. For example, the feature enhancement operation of the third network structure can enhance the image illumination information and weaken the fog information, and thus the image illumination information is obtained. In this way, the feature extraction of the original fog image can be realized based on the feature enhancement operation of different network structures, so that different image information (i.e., image illumination information, image fog information and defogging image) of the original fog image is obtained, and the subsequent image reconstruction can be realized.

[0087] For example, please refer to Figure 2a After the original fog image is obtained, the original fog image is subjected to feature encoding, feature enhancement and feature decoding based on the first network structure to obtain a defogging image output by the first network structure, the original fog image is subjected to feature encoding, feature enhancement and feature decoding based on the second network structure to obtain image fog information output by the second network structure, and the original fog image is subjected to feature encoding, feature enhancement and feature decoding based on the third network structure to obtain image illumination information output by the third network structure. The defogging image, the image fog information and the image illumination information are subjected to image reconstruction to obtain a reconstructed fog image.

[0088] Optionally, the feature enhancement on the image coding information comprises:

[0089] The image coding information is subjected to channel-level feature enhancement and pixel-level feature enhancement respectively.

[0090] The channel-level feature enhancement can be feature enhancement based on channel parameters of the network structure, and the pixel-level feature enhancement can be feature enhancement based on pixel parameters of the image. In this way, the feature enhancement is performed from the network structure aspect and the image aspect respectively, so that the effect of image feature enhancement is improved, and the image features output by the network structure are ensured to be the required image features.

[0091] It should be noted that, in the case that the image defogging model comprises the first network structure, the second network structure and the third network structure, the three network structures can all be respectively subjected to the channel-level feature enhancement and the pixel-level feature enhancement operations, that is, the first network structure, the second network structure and the third network structure all need to be respectively subjected to the channel-level feature enhancement and the pixel-level feature enhancement.

[0092] Optionally, the channel-level feature enhancement on the image encoding information comprises:

[0093] performing a convolution operation and a deformation operation on the image encoding information to obtain first image features and second image features;

[0094] performing a matrix multiplication operation on the first image features and the second image features to obtain a first channel matrix;

[0095] performing a flexible maximum regression function operation on the first channel matrix to obtain a second channel matrix;

[0096] performing a convolution operation on the image encoding information to obtain third image features, and performing a matrix multiplication operation on the second channel matrix and the third image features to obtain first channel enhanced features;

[0097] performing a pixel-level addition operation on the first channel enhanced features and the third image features to obtain second channel enhanced features.

[0098] It should be noted that, in the case that the image defogging model comprises the first network structure, the second network structure and the third network structure, the above steps are specifically described by performing the above operations through a target network structure, wherein the target network structure is any one of the first network structure, the second network structure and the third network structure.

[0099] For example, refer to Figure 2bFor example, taking a feature F(c×h×w) in image encoding information as an example (wherein c represents the number of image channels, h represents the image height, and w represents the image width), the feature F(c×h×w) is subjected to convolution operation and reshape operation based on the target network structure, to obtain a first image feature Qc(c×(h*w)) and a second image feature Hc((h*w)×c). The first image feature and the second image feature are subjected to matrix multiplication operation, to obtain a first channel matrix Mc(c×c). Then, the first channel matrix is subjected to softmax operation, to obtain a second channel matrix Mc'(c×c), which is a new weight. In addition, a new feature (i.e., a third image feature) Fc'(c×h×w) is obtained by subjecting the feature F(c×h×w) to convolution operation. Then, the second channel matrix Mc'(c×c) and the third image feature Fc'(c×h×w) are subjected to matrix multiplication operation, to obtain an enhanced first channel feature Fh(c×h×w). Finally, a second channel enhanced feature Fc after channel enhancement is obtained by subjecting the enhanced first channel feature Fh(c×h×w) and the third image feature Fc'(c×h×w) to pixel-level addition.

[0100] It should be noted that the above is only an example of taking one feature F(c×h×w) in image encoding information as an example for description. Each feature in image encoding information can be subjected to channel-level feature enhancement based on the above manner, to further achieve channel-level feature enhancement of all features in image encoding information, and to obtain channel-level image feature enhancement information.

[0101] In addition, the first network structure, the second network structure, and the third network structure can be used to respectively subject each feature in image encoding information to channel-level feature enhancement based on the above steps, to perform multiplication operation through weight and feature matrix, and to further enhance the required channel-level feature and weaken the unrequired channel-level feature.

[0102] Optionally, the pixel-level feature enhancement on the image encoding information comprises:

[0103] The image encoding information is subjected to convolution operation and reshape operation, to obtain a third image feature and a fourth image feature;

[0104] The third image feature and the fourth image feature are subjected to matrix multiplication operation, to obtain a first pixel matrix;

[0105] The first pixel matrix is subjected to softmax operation, to obtain a second pixel matrix;

[0106] Convolution operation is performed on the image coding information to obtain a sixth image feature, and matrix multiplication is performed on the second pixel matrix and the sixth image feature to obtain a first pixel enhanced feature;

[0107] Pixel-level addition operation is performed on the first pixel enhanced feature and the sixth image feature to obtain a second pixel enhanced feature.

[0108] Specifically, referring to Figure 2c Still taking the feature F(c×h×w) in the image coding information as an example, the feature F(c×h×w) is subjected to convolution operation and reshape operation by a target network structure (the first network structure, the second network structure or the third network structure) to obtain a third image feature Qp(c×(h*w)) and a fourth image feature Hp((h*w)×c), matrix multiplication is performed on the two features to obtain a first pixel matrix Mp((h*w)×(h*w)), and then the first pixel matrix is subjected to softmax operation to obtain a second pixel matrix Mp’((h*w)×(h*w)), which is a new weight; in addition, the sixth image feature Fp’(c×h×w) is obtained by performing convolution operation on the feature F(c×h×w), matrix multiplication is performed on the second pixel matrix Mp’((h*w)×(h*w)) and the sixth image feature Fp’(c×h×w) to obtain a first pixel enhanced feature Fh(c×h×w), and finally pixel-level addition operation is performed on the first pixel enhanced feature Fh(c×h×w) and the sixth image feature Fp’(c×h×w) to obtain the final pixel enhanced second pixel enhanced feature Fp.

[0109] It should be noted that the above is only an example of one feature F(c×h×w) in the image coding information, and each feature in the image coding information can be subjected to pixel-level feature enhancement based on the above manner, so as to achieve pixel-level feature enhancement of all features in the image coding information and obtain pixel-level image feature enhancement information.

[0110] Alternatively, the first network structure, the second network structure and the third network structure can be used to respectively perform pixel-level feature enhancement on each feature in the image coding information, and multiplication operation is performed on the weight and the feature matrix, so as to enhance the required pixel-level feature and weaken the unnecessary pixel-level feature, so as to obtain the required image feature information.

[0111] Further, after the channel-level feature enhancement and the pixel-level feature enhancement are performed on the image coding information based on the above manner, the image feature enhancement information obtained includes:

[0112] The image feature enhancement information is obtained based on the second channel enhancement feature and the second pixel enhancement feature.

[0113] It can be understood that after the image coding information is respectively subjected to channel level feature enhancement and pixel level feature enhancement to obtain the second channel enhancement feature and the second pixel enhancement feature, the final image feature enhancement information is obtained based on the second channel enhancement feature and the second pixel enhancement feature. For example, the image feature enhancement information can be the sum of the second channel enhancement feature and the second pixel enhancement feature.

[0114] Optionally, the image feature enhancement information can be obtained by the following formula:

[0115] F = a * Fc + b * Fp;

[0116] wherein, F is the image feature enhancement information, Fc is the second channel enhancement feature, Fp is the second pixel enhancement feature, and a and b are constants, respectively representing the learnable parameters in model training.

[0117] Optionally, the values of a and b can be adjusted, so that the proportion of the second channel enhancement feature and the second pixel enhancement feature in the image feature enhancement information can be flexibly adjusted, and the finally obtained image feature enhancement information can be adjusted.

[0118] The scheme provided by the embodiments of the present disclosure trains the image defogging model through the loss between the defogged image and the original clear image and the loss between the reconstructed foggy image and the original foggy image, which are two supervision signals. Compared with the training method with only one supervision signal, the present disclosure can effectively improve the training effect of the image defogging model, so that the precision of the trained image defogging model is higher, the defogging effect of the foggy image is better, and a clearer image can be output.

[0119] The embodiments of the present disclosure further provide an image defogging method.

[0120] Please refer to Figure 3 , Figure 3 is a flowchart of an image defogging method provided by the embodiments of the present disclosure. As Figure 3 shown, the method comprises the following steps:

[0121] Step S301, obtaining a to-be-processed image;

[0122] Step S302, performing defogging processing on the to-be-processed image based on an image defogging model to obtain a defogged clear image.

[0123] The image defogging model is obtained according to the above Figure 1The model is trained by the training method. The image defogging method provided in the embodiments of the present disclosure can be applied to electronic devices such as mobile phones, tablet computers, and computers.

[0124] It can be understood that the to-be-processed image can refer to an image that needs to be defogged, for example, the to-be-processed image includes fog information; further, the to-be-processed image is defogged based on the image defogging model to obtain a clear image.

[0125] In the embodiments of the present disclosure, the image defogging model is obtained according to Figure 1 The model is trained by the training method. In Figure 1 In the training method, two supervision signals are used to train the image defogging model, which effectively improves the training effect of the model, so that the trained image defogging model has higher precision and better defogging effect on the image, and then the image defogging model trained based on Figure 1 The image defogging model trained by the training method is used to defog the to-be-processed image, which can have a better defogging effect, so that the output clear image has higher clarity.

[0126] The embodiments of the present disclosure also provide a training device of an image defogging model.

[0127] Please refer to Figure 4 , Figure 4 is a structural diagram of a training device of an image defogging model provided by the embodiments of the present disclosure. As Figure 4 shown, the training device of the image defogging model 400 includes:

[0128] The first acquisition module 401 is configured to acquire an original fog image and a corresponding original clear image.

[0129] The feature extraction module 402 is configured to perform feature extraction on the original fog image to obtain image illumination information, image fog information, and a defogging image of the original fog image.

[0130] The reconstruction module 403 is configured to perform image reconstruction based on the image illumination information, the image fog information, and the defogging image to obtain a reconstructed fog image.

[0131] The loss calculation module 404 is configured to calculate a first loss between the defogging image and the original clear image, and a second loss between the reconstructed fog image and the original fog image.

[0132] The training module 405 is configured to train the image defogging model based on the first loss and the second loss.

[0133] Optionally, the image defogging model comprises a first network structure, and the feature extraction module 402 is further configured to:

[0134] extracting features of the original fog image based on the first network structure to obtain image illumination information, image fog information, and a defogging image of the original fog image.

[0135] Optionally, the image defogging model further comprises a second network structure and a third network structure, and the feature extraction module 402 is further configured to:

[0136] extracting features of the original fog image based on the first network structure, the second network structure, and the third network structure respectively to obtain a defogging image output by the first network structure, image fog information output by the second network structure, and image illumination information output by the third network structure.

[0137] Optionally, the feature extraction module 402 further comprises:

[0138] a feature encoding unit configured to encode features of the original fog image to obtain image encoding information;

[0139] a feature enhancement unit configured to enhance features of the image encoding information to obtain image feature enhancement information;

[0140] a feature decoding unit configured to decode features of the image feature enhancement information to obtain image illumination information, image fog information, and a defogging image of the original fog image.

[0141] Optionally, the feature enhancement unit is further configured to:

[0142] performing channel-level feature enhancement and pixel-level feature enhancement on the image encoding information respectively.

[0143] Optionally, the feature enhancement unit is further configured to:

[0144] performing convolution operation and deformation operation on the image encoding information to obtain first image features and second image features;

[0145] performing matrix multiplication operation on the first image features and the second image features to obtain a first channel matrix;

[0146] performing flexible maximum regression function operation on the first channel matrix to obtain a second channel matrix;

[0147] performing convolution operation on the image encoding information to obtain third image features, and performing matrix multiplication operation on the second channel matrix and the third image features to obtain first channel enhancement features;

[0148] The first channel enhanced feature is subjected to pixel-level addition operation with the third image feature to obtain a second channel enhanced feature.

[0149] Optionally, the feature enhancement unit is further configured to:

[0150] The image coding information is subjected to convolution operation and morphing operation to obtain a third image feature and a fourth image feature;

[0151] The third image feature and the fourth image feature are subjected to matrix multiplication operation to obtain a first pixel matrix;

[0152] The first pixel matrix is subjected to flexible maximum regression function operation to obtain a second pixel matrix;

[0153] The image coding information is subjected to convolution operation to obtain a sixth image feature, and the second pixel matrix and the sixth image feature are subjected to matrix multiplication operation to obtain a first pixel enhanced feature;

[0154] The first pixel enhanced feature is subjected to pixel-level addition operation with the sixth image feature to obtain a second pixel enhanced feature.

[0155] Optionally, the feature enhancement unit is further configured to:

[0156] The image feature enhancement information is obtained based on the second channel enhanced feature and the second pixel enhanced feature.

[0157] Optionally, the reconstruction module 403 is further configured to:

[0158] A first product between the image fog information and the defogged image is obtained;

[0159] A second product between the image fog information and the image illumination information is obtained;

[0160] Image reconstruction is performed according to the first product and the second product to obtain a reconstructed fog image.

[0161] It should be noted that the training device 400 of the image defogging model provided by the embodiments of the present disclosure can implement the above Figure 1 all technical solutions in the training method of the image defogging model, and thus at least the above Figure 1 all technical effects of the method embodiments, which will not be repeated here.

[0162] The embodiments of the present disclosure further provide an image defogging device.

[0163] Please refer to Figure 5 , Figure 5 is a structural diagram of an image defogging device provided by the embodiments of the present disclosure. As shown inFigure 5 As shown in the figure, the image defogging device 500 comprises:

[0164] A second acquisition module 501 is configured to acquire a to-be-processed image.

[0165] A processing module 502 is configured to perform defogging processing on the to-be-processed image based on an image defogging model, to obtain a clear image after defogging.

[0166] The image defogging model is a model trained according to the method of the first aspect.

[0167] It should be noted that the image defogging device 500 provided by the embodiments of the present disclosure can achieve the above-mentioned Figure 3 all the technical solutions in the image defogging method embodiments, and thus at least the above-mentioned Figure 3 all the technical effects of the method embodiments can be achieved, which will not be repeated here.

[0168] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0169] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0170] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0171] As Figure 6 shown, the device 600 includes a computing unit 601 that can perform various suitable actions and processes in accordance with computer programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage unit 608. Various programs and data required by the device 600 for operation can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0172] A plurality of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0173] The computing unit 601 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the training method of the image defogging model or the image defogging method. For example, in some embodiments, the training method of the image defogging model or the image defogging method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the training method of the image defogging model or the image defogging method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the training method of the image defogging model or the image defogging method by any other appropriate means, such as by means of firmware.

[0174] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0175] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0178] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0179] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically happening over a communication network. This relationship between a client and a server is created by executing computer programs on the respective computers with the client and server programs interacting across a data communication which can be any form or medium of digital data communication. The server can be a cloud server, a server of a distributed system or a server incorporating blockchain.

[0180] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0181] The specific embodiments described above are not intended to be limiting. Various modifications, combinations, sub-combinations and alternatives can be apparent to one of ordinary skill in the art and can be made to the specific embodiments without departing from the spirit and scope of the disclosure. Such modifications, equivalents, and alternatives are intended to fall within the scope of the disclosure.

Claims

1. A method for training an image dehazing model, comprising: Acquire the original fog image and the corresponding original clear image; Feature extraction is performed on the original fog image to obtain the image illumination information, image fog information, and defogging image of the original fog image; Based on the image illumination information, the image fog information, and the defogging image, an image reconstruction is performed to obtain a reconstructed fog image; Calculate the first loss between the dehazed image and the original clear image, and the second loss between the reconstructed hazy image and the original hazy image; The image dehazing model is trained based on the first loss and the second loss; The step of extracting features from the original fog image to obtain image illumination information, image fog information, and a defogging image of the original fog image includes: The original fog image is feature-encoded to obtain image encoding information; The image encoding information is enhanced with features to obtain image feature enhancement information; The image feature enhancement information is decoded to obtain the image illumination information, image fog information, and defogging image of the original fog image; The feature enhancement of the image encoding information includes: The image encoding information is subjected to channel-level feature enhancement and pixel-level feature enhancement respectively; The step of reconstructing the image based on the image illumination information, the image fog information, and the dehazed image to obtain a reconstructed fog image includes: Obtain the first product between the image fog information and the defogging image; Obtain the second product between the image fog information and the image illumination information; The image is reconstructed based on the first product and the second product to obtain the reconstructed fog image.

2. The method according to claim 1, wherein, The image dehazing model includes a first network structure. The step of extracting features from the original fog image to obtain image illumination information, image fog information, and a dehazed image includes: Based on the first network structure, feature extraction is performed on the original fog image to obtain the image illumination information, image fog information, and defogging image of the original fog image.

3. The method according to claim 2, wherein, The image dehazing model further includes a second network structure and a third network structure. The step of extracting features from the original fog image to obtain image illumination information, image fog information, and a dehazed image includes: Feature extraction is performed on the original fog image based on the first network structure, the second network structure, and the third network structure, respectively, to obtain the defogging image output by the first network structure, the fog information output by the second network structure, and the illumination information output by the third network structure.

4. The method according to claim 1, wherein, The channel-level feature enhancement of the image encoding information includes: The image encoding information is subjected to convolution and deformation operations to obtain the first image feature and the second image feature; Perform matrix multiplication on the first image feature and the second image feature to obtain the first channel matrix; The first channel matrix is ​​subjected to a flexible maximum regression function operation to obtain the second channel matrix; The image encoding information is convolved to obtain the third image feature, and the second channel matrix and the third image feature are multiplied to obtain the first channel enhanced feature. The first channel enhancement feature is added to the third image feature at the pixel level to obtain the second channel enhancement feature.

5. The method according to claim 4, wherein, The pixel-level feature enhancement of the image encoding information includes: The image encoding information is subjected to convolution and deformation operations to obtain the third image feature and the fourth image feature; Perform matrix multiplication on the third image feature and the fourth image feature to obtain the first pixel matrix; The first pixel matrix is ​​subjected to a flexible maximum regression function operation to obtain the second pixel matrix; The image encoding information is convolved to obtain the sixth image feature, and the second pixel matrix and the sixth image feature are multiplied to obtain the first pixel enhancement feature. The first pixel enhancement feature is added to the sixth image feature at the pixel level to obtain the second pixel enhancement feature.

6. The method according to claim 5, wherein, The obtained image feature enhancement information includes: The image feature enhancement information is obtained based on the second channel enhancement feature and the second pixel enhancement feature.

7. An image dehazing method, comprising: Obtain the image to be processed; The image to be processed is dehazed based on the image dehazing model to obtain a clear image after dehazing; The image dehazing model is a model trained by the method according to any one of claims 1-6.

8. A training device for an image dehazing model, comprising: The first acquisition module is used to acquire the original fog image and the corresponding original clear image; The feature extraction module is used to extract features from the original fog image to obtain the image illumination information, image fog information, and defogging image of the original fog image; The reconstruction module is used to reconstruct the image based on the image illumination information, the image fog information, and the defogging image to obtain a reconstructed fog image. The loss calculation module is used to calculate the first loss between the dehazed image and the original clear image, and the second loss between the reconstructed hazy image and the original hazy image; The training module is used to train the image dehazing model based on the first loss and the second loss; The feature extraction module further includes: A feature encoding unit is used to perform feature encoding on the original fog image to obtain image encoding information; A feature enhancement unit is used to enhance the image coding information to obtain image feature enhancement information; The feature decoding unit is used to perform feature decoding on the image feature enhancement information to obtain the image illumination information, image fog information, and defogging image of the original fog image; The feature enhancement unit is further configured to: The image encoding information is subjected to channel-level feature enhancement and pixel-level feature enhancement respectively; The reconstruction module is further used for: Obtain the first product between the image fog information and the defogging image; Obtain the second product between the image fog information and the image illumination information; The image is reconstructed based on the first product and the second product to obtain the reconstructed fog image.

9. An image dehazing device, comprising: The second acquisition module is used to acquire the image to be processed; The processing module is used to perform dehazing processing on the image to be processed based on the image dehazing model to obtain a clear image after dehazing; The image dehazing model is a model trained by the method according to any one of claims 1-6.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6, or to perform the method of claim 7.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6, or to perform the method according to claim 7.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6, or implements the method according to claim 7.

Images