Image defogging model acquisition method, device and electronic device

Abstract

The present disclosure provides a method, device and electronic device for obtaining an image defogging model, and relates to the field of artificial intelligence technology such as image processing. The specific implementation is as follows: when obtaining the image defogging model, the foggy image in each image sample pair among multiple image sample pairs is input into the initial image defogging model, and the first loss function corresponding to each image sample pair is constructed according to the foggy image in each image sample pair and the output result corresponding to each image sample pair; the second loss function corresponding to each image sample pair is constructed according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair; and the first loss function and the second loss function corresponding to each image sample pair are combined to jointly update the network parameters of the initial image defogging model, so that the accuracy of the image defogging model can be improved, so that when the foggy image is defogged based on the image defogging model in the future, the accuracy of the processing result can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, in particular to the field of artificial intelligence technology such as image processing, and specifically to a method, device and electronic device for acquiring an image defogging model. Background Art

[0002] Image dehazing refers to a method of taking an image containing fog as input and performing dehazing processing on the image containing fog to obtain a clear image.

[0003] Fog often exists in real scenes, which can block objects in the scene and seriously affect the application of computer vision technology, such as object detection and segmentation, object tracking, etc. Therefore, how to defog foggy images is crucial. Summary of the invention

[0004] The present disclosure provides a method, device and electronic device for acquiring an image defogging model, which can acquire the image defogging model through training, so that the foggy image can be defogged based on the image defogging model, thereby improving the accuracy of the acquired clear image.

[0005] According to a first aspect of the present disclosure, a method for training an image defogging model is provided. The method for training an image defogging model may include:

[0006] A plurality of image sample pairs are obtained, wherein each image sample pair includes a fog image and a preset clear image corresponding to the fog image.

[0007] The fog image in each image sample pair is input into the initial image defogging model to obtain the output results corresponding to each image sample pair; wherein the output results include the illumination information image, the fog component image, and the predicted clear image corresponding to the fog image.

[0008] According to the fog image in each image sample pair and the output result corresponding to each image sample pair, a first loss function corresponding to each image sample pair is constructed, and according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair, a second loss function corresponding to each image sample pair is constructed.

[0009] According to the first loss function and the second loss function corresponding to each pair of image samples, the network parameters of the initial image dehazing model are updated.

[0010] According to a second aspect of the present disclosure, an image defogging method is provided, which may include:

[0011] Get the fog image to be processed.

[0012] The foggy image to be processed is input into an image defogging model to obtain a clear image corresponding to the foggy image to be processed.

[0013] Among them, the image dehazing model is trained by constructing a first loss function corresponding to each image sample pair based on the foggy image in each image sample pair and the output result corresponding to each image sample pair, and by training by constructing a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0014] According to a third aspect of the present disclosure, a training device for an image defogging model is provided. The training device for an image defogging model may include:

[0015] The acquisition unit is used to acquire a plurality of image sample pairs, wherein each image sample pair includes a fog image and a preset clear image corresponding to the fog image.

[0016] A processing unit is used to input the fog image in each image sample pair into the initial image defogging model to obtain the output results corresponding to each image sample pair; wherein the output results include the illumination information image, the fog component image, and the predicted clear image corresponding to the fog image.

[0017] A construction unit is used to construct a first loss function corresponding to each image sample pair based on the fog image in each image sample pair and the output result corresponding to each image sample pair, and to construct a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0018] An updating unit is used to update the network parameters of the initial image defogging model according to the first loss function and the second loss function corresponding to each pair of image samples.

[0019] According to a fourth aspect of the present disclosure, an image defogging device is provided, which may include:

[0020] The acquisition unit is used to acquire the fog image to be processed.

[0021] The processing unit is used to input the fog image to be processed into an image defogging model to obtain a clear image corresponding to the fog image to be processed.

[0022] Among them, the image dehazing model is trained by constructing a first loss function corresponding to each image sample pair based on the foggy image in each image sample pair and the output result corresponding to each image sample pair, and by training by constructing a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0023] According to a fifth aspect of the present disclosure, an electronic device is provided, which may include:

[0024] at least one processor; and

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the image defogging model described in the first aspect above, or execute the image defogging method described in the second aspect above.

[0027] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the training method of the image defogging model described in the first aspect above; or, to execute the image defogging method described in the second aspect above.

[0028] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program so that the electronic device executes the training method of the image defogging model described in the first aspect above; or, executes the image defogging method described in the second aspect above.

[0029] According to the technical solution of the present disclosure, an image defogging model can be acquired through training, so that the foggy image can be defogged based on the image defogging model, thereby improving the accuracy of the acquired clear image.

[0030] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0032] Figure 1 is a flowchart of a training method for an image defogging model provided according to the first embodiment of the present disclosure;

[0033] Figure 2 is a flowchart of a method for obtaining output results corresponding to each pair of image samples provided according to the second embodiment of the present disclosure;

[0034] Figure 3 is a schematic diagram of a network architecture of an initial image defogging model provided by an embodiment of the present disclosure;

[0035] Figure 4 is a structural diagram of a downsampling network provided by an embodiment of the present disclosure;

[0036] Figure 5 is a schematic diagram of the structure of an upsampling network provided by an embodiment of the present disclosure;

[0037] Figure 6 is a schematic diagram of a feature enhancement process provided by an embodiment of the present disclosure;

[0038] Figure 7 is a schematic diagram of a pixel-level feature processing provided by an embodiment of the present disclosure;

[0039] Figure 8 is a flowchart of a method for updating network parameters of an initial image defogging model provided according to a third embodiment of the present disclosure;

[0040] Fig. 9 is a flowchart of an image defogging method provided according to a fourth embodiment of the present disclosure;

[0041] Fig.10 is a structural schematic diagram of a training device for an image defogging model provided according to a fifth embodiment of the present disclosure;

[0042] Fig.11 is a structural schematic diagram of an image defogging device provided according to a sixth embodiment of the present disclosure;

[0043] Fig.12 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] In the embodiments of the present disclosure, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the text description of the present disclosure, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present disclosure, "first", "second", "third", "fourth", "fifth" and "sixth" are only used to distinguish the contents of different objects and have no other special meanings.

[0046] The technical solution provided by the embodiments of the present disclosure can be applied to image processing scenarios. Taking image defogging as an example, image defogging refers to a method of taking an image containing fog as input and performing defogging on the image containing fog to obtain a clear image.

[0047] When dehazing a foggy image, the prior art usually adopts a method based on data priors to statistically analyze the numerical distribution of the foggy image, obtain the distribution characteristics of the fog based on the numerical distribution of the foggy image, and then eliminate the fog components in the foggy image based on the distribution characteristics of the fog, so as to obtain a corresponding clear image.

[0048] However, when using data prior methods to dehaze foggy images, since data prior methods are more dependent on data, this method can only process prior foggy images in existing collected data, and its generalization is relatively limited.

[0049] In order to better defog foggy images, it is possible to consider training an image defogging model based on deep learning, so as to use the image defogging model to defog foggy images. In addition, in order to make the trained image defogging model more accurate, the component information in the foggy image, such as the illumination information image, the foggy component image, and the clear image, can be fully combined to jointly construct a loss function, so as to jointly train the image defogging model through the constructed loss function, thereby obtaining the image defogging model, so that the foggy image can be defogged based on the image defogging model in the future, thereby improving the accuracy of the obtained clear image.

[0050] Based on the above technical concept, the embodiment of the present disclosure provides a training method for an image defogging model. The following will describe the training method for the image defogging model provided by the present disclosure in detail through specific embodiments. It is understandable that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0051] Embodiment 1

[0052] Figure 1 is a flow chart of a training method for an image defogging model provided according to the first embodiment of the present disclosure. The training method for an image defogging model can be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. For example, see Figure 1 As shown, the training method of the image defogging model may include:

[0053] S101. Acquire multiple image sample pairs.

[0054] Each pair of image samples includes a foggy image and a preset clear image corresponding to the foggy image.

[0055] For example, the multiple image sample pairs can be multiple image sample pairs collected based on real scenes; they can also be multiple image sample pairs generated based on simulation operations; or some can be collected from real scenes and some can be generated based on simulation operations. The specific settings can be made according to actual needs.

[0056] For example, when obtaining multiple image sample pairs, multiple image sample pairs input by the user can be directly received, or multiple image sample pairs sent by other electronic devices can be received, or multiple image sample pairs can be obtained from local storage, or multiple image sample pairs can be obtained through other methods. The specific settings can be made according to actual needs. Here, the embodiments of the present disclosure do not make specific restrictions on the method of obtaining multiple image sample pairs.

[0057] After obtaining a plurality of image sample pairs, the plurality of image sample pairs can be used as image sample pairs for performing a training operation, and the fog image in each image sample pair in the plurality of image sample pairs is input into the initial image defogging model, that is, the following S102 is performed:

[0058] S102: Input the fog image in each image sample pair into the initial image defogging model to obtain the output result corresponding to each image sample pair.

[0059] The output results include the illumination information image corresponding to the fog image, the fog component image, and the predicted clear image.

[0060] It can be understood that, for the foggy image in each image sample pair, after it is input into the initial image defogging model, after the foggy image is defogged by the initial image defogging model, not only a clear image corresponding to the foggy image can be obtained, but also an illumination information image and a foggy component image corresponding to the foggy image can be obtained. In order to distinguish the preset clear image in the above image sample pair, the clear image output by the initial image defogging model can be recorded as a predicted clear image.

[0061] In this way, after the fog image in each image sample pair is input into the initial image dehazing model, the output result corresponding to each image sample pair will be obtained, and the output result corresponding to each image sample pair includes the illumination information image corresponding to the fog image, the fog component image, and the predicted clear image.

[0062] In the embodiment of the present disclosure, in order to make the trained image defogging model more accurate, the illumination information image, the fog component image, and the clear image in the fog image can be fully combined to jointly construct a loss function, so that the image defogging model can be jointly trained through the constructed loss function, that is, the following S103 and S104 are executed:

[0063] S103. Construct a first loss function corresponding to each image sample pair based on the fog image in each image sample pair and the output result corresponding to each image sample pair, and construct a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0064] It should be noted that the disclosed embodiment uses two supervisory signals when training the image defogging model. One supervisory signal is a supervisory signal based on the foggy image in the image sample pair and the output result corresponding to the image sample pair; the other supervisory signal is a supervisory signal based on the preset clear image in the image sample pair and the predicted clear image in the output result corresponding to the image sample pair. In this way, the initial image defogging model is trained by combining the two supervisory signals, which can improve the accuracy and robustness of the image defogging model.

[0065] Based on the supervisory signal composed of the foggy image in the image sample pair and the output result corresponding to the image sample pair, the first loss function corresponding to each image sample pair can be constructed according to the foggy image in each image sample pair and the output result corresponding to each image sample pair; and based on the supervisory signal composed of the preset clear image in the image sample pair and the predicted clear image in the output result corresponding to the image sample pair, the second loss function corresponding to each image sample pair can be constructed according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair. In this way, each image sample pair in the multiple image sample pairs will correspond to a second loss function and a second loss function.

[0066] After constructing the first loss function and the second loss function corresponding to each image sample pair, the following S104 can be executed:

[0067] S104: Update the network parameters of the initial image dehazing model according to the first loss function and the second loss function corresponding to each pair of image samples.

[0068] For example, when the network parameters of the initial image dehazing model are updated according to the first loss function and the second loss function corresponding to each pair of image samples, if the updated image dehazing model converges, the updated image dehazing model is directly determined as the final trained image dehazing model; if the updated image dehazing model does not converge, the above steps are performed again until the updated image dehazing model converges, and the image dehazing model at the time of convergence is determined as the final trained image dehazing model, thereby obtaining the final image dehazing model. The image dehazing model obtained in this way has better dehazing ability, thereby effectively improving the robustness of the image dehazing model.

[0069] It can be seen that in the disclosed embodiment, when acquiring the image defogging model, a plurality of image sample pairs including a foggy image and a preset clear image corresponding to the foggy image can be first acquired; the foggy image in each image sample pair is input into the initial image defogging model, and according to the foggy image in each image sample pair and the output result corresponding to each image sample pair, a first loss function corresponding to each image sample pair is constructed; according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair, a second loss function corresponding to each image sample pair is constructed; and then according to the first loss function and the second loss function corresponding to each image sample pair, the network parameters of the initial image defogging model are jointly updated, so that the accuracy of the image defogging model can be improved, so that when the foggy image is subsequently defogged based on the image defogging model, the accuracy of the processing result can be improved.

[0070] Based on the above Figure 1 In order to facilitate understanding of how to input the fog image in each image sample pair into the initial image defogging model in the above S102 to obtain the output result corresponding to each image sample pair, the following will be described. Figure 2 The second embodiment shown is described in detail.

[0071] Embodiment 2

[0072] Figure 2 is a flow chart of a method for obtaining output results corresponding to each image sample pair according to the second embodiment of the present disclosure. The method can also be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. For example, see Figure 2 As shown, the method for obtaining the output results corresponding to each pair of image samples may include:

[0073] S201, inputting the fog image in each image sample into the initial image defogging model, performing encoding processing through a downsampling network, and obtaining a feature map corresponding to each fog image.

[0074] The initial image dehazing model includes a downsampling network and three upsampling networks. For an example, see Figure 3 As shown, Figure 3 This is a schematic diagram of the network architecture of an initial image defogging model provided by an embodiment of the present disclosure. The downsampling network included in the initial image defogging model is used to encode the fog image in the image sample pair to obtain a feature map corresponding to the fog image; among the three upsampling networks, one upsampling network is used to decode the feature map to obtain a lighting information image; one upsampling network is used to decode the feature map to obtain a fog component image; and one upsampling network is used to decode the feature map to obtain a predicted clear image.

[0075] For an example of the downsampling network included in the initial image dehazing model, see Figure 4 As shown, Figure 4 is a schematic diagram of a downsampling network provided by an embodiment of the present disclosure, which can be Figure 4 The downsampling network shown encodes the fog image; for the upsampling network included in the initial image defogging model, see the example. Figure 5 As shown, Figure 5 is a schematic diagram of the structure of an upsampling network provided by an embodiment of the present disclosure, which can be Figure 5 The downsampling network shown decodes the feature map.

[0076] When the fog image in each image sample is input into the initial image defogging model, it will first be encoded through the downsampling network in the initial image defogging model to obtain the feature map corresponding to each fog image. After that, the feature map corresponding to each fog image will be decoded through the three upsampling networks respectively, that is, the following S202 is executed:

[0077] S202, respectively decoding the feature maps corresponding to the fog images through three upsampling networks to obtain output results corresponding to each image sample pair.

[0078] For example, when the feature maps corresponding to each fog image are decoded through three upsampling networks respectively, the feature maps corresponding to each fog image can be directly decoded through the three upsampling networks; or the feature maps can be enhanced first to obtain the target feature maps; and then the target feature maps are decoded through the three upsampling networks respectively to obtain the output results corresponding to each image sample pair. The specific settings can be made according to actual needs.

[0079] It can be understood that in the embodiment of the present disclosure, in addition to the decoding process, the feature map is enhanced so that the obtained target feature map has more obvious feature information than the feature map that has not been enhanced, which is more conducive to subsequent decoding processing.

[0080] For example, when the feature map is enhanced to obtain the target feature map, the channel features and pixel features in the feature map can be first determined; and the channel features and pixel features are enhanced respectively to obtain the target feature map.

[0081] It should be noted that in the embodiments of the present disclosure, when enhancing the channel features and pixel-level features in the feature map, the channel features in the feature map can be enhanced first, and then the pixel-level features of the feature map after the channel feature enhancement are enhanced; or the pixel-level features in the feature map can be enhanced first, and then the channel features of the feature map after the channel pixel-level enhancement are enhanced. The specific settings can be made according to actual needs.

[0082] Take the case of first enhancing the channel features in the feature map and then enhancing the pixel-level features of the feature map after the channel feature enhancement as an example. For an example, see Figure 6 As shown, Figure 6 is a schematic diagram of a feature enhancement process provided by an embodiment of the present disclosure. When the channel feature in the feature map is enhanced, the feature map can be recorded as F(C xH xW). The corresponding enhancement process includes:

[0083] (1) The feature map F(C x H x W) is transformed through the average pooling layer to obtain a feature map of size C x 1x 1;

[0084] (2) Perform a convolution operation on the converted feature map of size C x 1x1 through n convolution units to obtain a feature map H of size C x 1x 1;

[0085] (3) The feature map H is transformed through the sigmoid operation to obtain a weight map Q of size C x 1x1;

[0086] (4) Obtain Fh(C x H x W) = Q*F through characteristic multiplication operation;

[0087] (5) Through the feature addition operation, the feature map Fo(C x H x W) = Fh + F after channel feature enhancement is obtained.

[0088] In this way, through the above steps (1) to (5), the channel enhancement processing of the feature map can be achieved, thereby obtaining the feature map after channel feature enhancement processing.

[0089] After obtaining the feature map after channel feature enhancement processing, the pixel-level features of the feature map after channel feature enhancement processing can be enhanced. For an example, see Figure 7 As shown, Figure 7 is a schematic diagram of a pixel-level feature processing provided by an embodiment of the present disclosure. When enhancing the pixel-level feature, the feature map Fo(C x H x W) after the channel feature enhancement processing can be used as input. The corresponding enhancement processing process includes:

[0090] (1) Convolution operation is performed on Fo(C x H x W) through n convolution units to obtain a feature map H' of size 1x H x W. Among them, the output feature size of the last convolution layer of feature map H' is 1;

[0091] (2) The feature map H' is transformed through the sigmoid operation to obtain a weight map Q' with a size of 1x H x W;

[0092] (3) Through the characteristic multiplication operation, we obtain F'h(C x H x W) = Q'*Fo

[0093] (4) The enhanced feature F'o(C x H x W) = Fo h + Fo is obtained through feature addition operation.

[0094] In this way, through the above steps (1) to (4), the pixel-level features of the feature map after the channel feature enhancement processing can be enhanced to obtain the target feature map after the channel feature and pixel feature enhancement processing.

[0095] It can be seen that in the disclosed embodiment, when obtaining the output results corresponding to each image sample pair, the fog image in each image sample can be first input into the initial image defogging model, and encoded through the downsampling network to obtain the feature map corresponding to each fog image; then the feature map corresponding to each fog image is decoded through three upsampling networks respectively to obtain the illumination information image, fog component image, and clear image corresponding to each image sample pair. The influence of the illumination information image, fog component image, and clear image on the defogging operation is fully considered, so that the illumination information image, fog component image, and clear image in the fog image can be combined to jointly train the image defogging model, which can improve the accuracy of the image defogging model.

[0096] Based on any of the above embodiments, after the foggy images in each image sample pair are input into the initial image defogging model and the output results corresponding to each image sample pair are obtained, the network parameters of the initial image defogging model can be updated according to each image sample pair and the output results corresponding to each image sample pair. In order to facilitate understanding of how to update the network parameters of the initial image defogging model according to each image sample pair and the output results corresponding to each image sample pair in the embodiments disclosed herein, the following will be described. Figure 8 Embodiment 3 shown.

[0097] Embodiment 3

[0098] Figure 8 FIG. 1 is a flow chart of a method for updating the network parameters of the initial image defogging model according to the third embodiment of the present disclosure. The method can also be executed by software and / or hardware devices. For example, see Figure 8 As shown, the method for updating the network parameters of the initial image defogging model may include:

[0099] S801 . Reconstruct a reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair.

[0100] It can be understood that, given that the reconstruction methods of the reconstructed fog images corresponding to each image sample pair are similar, in order to avoid redundancy, the following will take the reconstruction of the reconstructed fog image corresponding to any one of the image sample pairs as an example to describe how to reconstruct the reconstructed fog images corresponding to each image sample pair.

[0101] For example, when reconstructing the corresponding reconstructed fog image according to the illumination information image, fog component image, and predicted clear image in the output results corresponding to the image samples, the corresponding reconstructed fog image can be reconstructed according to I=J*T+(1-T)*A to obtain the reconstructed fog image.

[0102] Among them, I represents the reconstructed fog image, J represents the predicted clear image, T represents the fog component image, and A represents the illumination information image.

[0103] After reconstructing the reconstructed fog image corresponding to each image sample pair, the first loss function corresponding to each image sample pair can be constructed according to the fog image in each image sample pair and its corresponding reconstructed fog image, that is, the following S802 is executed:

[0104] S802: Construct a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and its corresponding reconstructed fog image.

[0105] For example, the first loss function may be a cross entropy loss function, a difference loss function, or other loss functions, which may be set according to actual needs. The embodiments of the present disclosure do not impose any specific limitations thereto.

[0106] S803. Construct a second loss function corresponding to each image sample pair according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0107] For example, the second loss function may be a cross entropy loss function, a difference loss function, or other loss functions, which may be set according to actual needs. The embodiments of the present disclosure do not impose any specific limitations thereto.

[0108] It should be noted that, in the embodiments of the present disclosure, there is no sequence between the above S801-S802 and the above S803. S801-S802 may be executed first, and then S803; S803 may be executed first, and then S801-S802; S801-S802 and S803 may also be executed at the same time. The specific settings may be made according to actual needs. Here, the embodiments of the present disclosure do not make any specific restrictions on the execution order of S801-S802 and S803.

[0109] Combined with the description in S802 and S803 above, for each pair of image samples, a corresponding first loss function and a second loss function will be obtained, and then the network parameters of the initial image defogging model can be updated according to the first loss function and the second loss function corresponding to each image sample pair. For example, when updating the network parameters of the initial image defogging model according to the first loss function and the second loss function corresponding to each image sample pair, the first loss function and the second loss function corresponding to each image sample pair can be used to determine the loss function corresponding to each image sample pair, that is, execute the following S804:

[0110] S804: Determine the loss function corresponding to each image sample pair according to the first loss function and the second loss function corresponding to each image sample pair.

[0111] It can be understood that, given that the method for determining the loss function corresponding to each image sample pair is similar, in order to avoid redundancy, the determination of the loss function corresponding to any image sample pair will be taken as an example to describe how to determine the loss function corresponding to each image sample pair.

[0112] For example, when determining the loss function corresponding to the image sample pair based on the first loss function and the second loss function corresponding to the image sample pair, the first loss function and the second loss function can be directly summed, and the loss sum can be determined as the loss function corresponding to the image sample pair; or the first loss function and the second loss function can be weighted, and the processing result can be determined as the loss function corresponding to the image sample pair. The specific setting can be made according to actual needs.

[0113] After obtaining the loss function corresponding to each image sample pair, the following S805 can be executed:

[0114] S805: Update the network parameters of the initial image defogging model according to the loss function corresponding to each pair of image samples.

[0115] For example, when updating the network parameters of the initial image dehazing model according to the loss function corresponding to each image sample pair, considering that the multiple image sample pairs are multiple image sample pairs for performing one training operation, it is possible to first determine the average loss function corresponding to the multiple image sample pairs according to the loss function corresponding to each image sample pair; and update the network parameters of the initial image dehazing model according to the average loss function to train the initial image dehazing model.

[0116] It can be understood that after updating the network parameters of the initial image dehazing model according to the average loss function, if the updated image dehazing model converges, the updated image dehazing model is directly determined as the final trained image dehazing model; if the updated image dehazing model does not converge, the above steps are performed again until the updated image dehazing model converges, and the image dehazing model at the time of convergence is determined as the final trained image dehazing model, thereby obtaining the final image dehazing model. The image dehazing model obtained in this way has better dehazing ability, thereby effectively improving the robustness of the image dehazing model.

[0117] It can be seen that in the disclosed embodiment, when updating the network parameters of the initial image defogging model, a first loss function corresponding to each image sample pair can be constructed based on the foggy image in each image sample pair and its corresponding reconstructed foggy image; and a second loss function corresponding to each image sample pair can be constructed based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair; and then the network parameters of the initial image defogging model are jointly updated based on the first loss function and the second loss function corresponding to each image sample pair, so that the accuracy of the image defogging model can be improved, so that when the foggy image is subsequently defogged based on the image defogging model, the accuracy of the processing result can be improved.

[0118] After the image defogging model for defogging is obtained through training through any of the above embodiments, the image defogging model can be used to defog the foggy image, that is, the application process of the foggy image. For details, please refer to the following Fig. 9 Embodiment 4 shown.

[0119] Embodiment 4

[0120] Fig. 9 is a flowchart of an image defogging method provided according to the fourth embodiment of the present disclosure. The image defogging method can be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. For example, see Fig. 9 As shown, the image defogging method may include:

[0121] S901, obtaining a fog image to be processed.

[0122] For example, when obtaining the fog image to be processed, the fog image to be processed can be directly received from the user, or it can be received from other electronic devices, or it can be obtained from local storage, or it can be obtained through other methods. The specific settings can be made according to actual needs. Here, the embodiment of the present disclosure does not make any specific restrictions on the method for obtaining the fog image to be processed.

[0123] After obtaining the fog image to be processed, the fog image to be processed can be input into the image defogging model to obtain a clear image corresponding to the fog image to be processed, that is, the following S902 is executed:

[0124] S902: Input the foggy image to be processed into an image defogging model to obtain a clear image corresponding to the foggy image to be processed.

[0125] Among them, the image dehazing model is trained by constructing a first loss function corresponding to each image sample pair based on the foggy image in each image sample pair and the output result corresponding to each image sample pair, and by training a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0126] It should be noted that, in the embodiments of the present disclosure, the training method of the image defogging model can refer to the relevant description in any of the above embodiments, and the embodiments of the present disclosure will not be repeated here.

[0127] It can be understood that after inputting the fog image to be processed into the image defogging model, in addition to obtaining a clear image corresponding to the fog image to be processed, the illumination information image and fog component image corresponding to the fog image can also be obtained, which can provide more references for users.

[0128] It can be seen that in the disclosed embodiment, when defogging a foggy image, the foggy image to be processed can be obtained first, and the foggy image to be processed can be input into the image defogging model to obtain a clear image corresponding to the foggy image to be processed. In this way, the foggy image to be processed can be defogged by the image defogging model, which can improve the accuracy of the processing results.

[0129] Embodiment 5

[0130] Fig.10 is a structural diagram of a training device 100 for an image defogging model provided according to the fifth embodiment of the present disclosure, for example, see Fig.10 As shown, the training device 100 of the image defogging model may include:

[0131] The acquisition unit 1001 is used to acquire a plurality of image sample pairs, wherein each image sample pair includes a fog image and a preset clear image corresponding to the fog image.

[0132] The processing unit 1002 is used to input the fog image in each image sample pair into the initial image defogging model to obtain the output result corresponding to each image sample pair; wherein the output result includes the illumination information image corresponding to the fog image, the fog component image, and the predicted clear image.

[0133] The construction unit 1003 is used to construct a first loss function corresponding to each image sample pair based on the fog image in each image sample pair and the output result corresponding to each image sample pair, and to construct a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0134] The updating unit 1004 is used to update the network parameters of the initial image dehazing model according to the first loss function and the second loss function corresponding to each pair of image samples.

[0135] Optionally, the initial image dehazing model includes a downsampling network and three upsampling networks; the processing unit 1002 includes a first processing module and a second processing module.

[0136] The first processing module is used to input the fog image in each image sample into the initial image defogging model, perform encoding processing through the downsampling network, and obtain the feature map corresponding to each fog image.

[0137] The second processing module is used to decode the feature maps corresponding to each fog image through three upsampling networks respectively to obtain the output results corresponding to each image sample pair.

[0138] Optionally, the second processing module includes a first processing sub-module and a second processing sub-module.

[0139] The first processing submodule is used to perform enhancement processing on the feature map corresponding to each fog image to obtain a target feature map corresponding to each fog image.

[0140] The second processing submodule is used to decode the target feature map corresponding to each fog image through three upsampling networks respectively to obtain the output result corresponding to each image sample pair.

[0141] Optionally, the first processing submodule is specifically used to determine the channel features and pixel features in the feature map corresponding to each fog image; and enhance the channel features and pixel features in the feature map corresponding to each fog image respectively to obtain the target feature map corresponding to each fog image.

[0142] Optionally, the construction unit 1003 includes a first construction module and a second construction module;

[0143] The first construction module is used to reconstruct the reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair.

[0144] The second construction module is used to construct a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and its corresponding reconstructed fog image.

[0145] Optionally, the updating unit 1004 includes a first updating module and a second updating module.

[0146] The first updating module is used to determine the loss function corresponding to each image sample pair according to the first loss function and the second loss function corresponding to each image sample pair.

[0147] The second updating module is used to update the network parameters of the initial image defogging model according to the loss function corresponding to each pair of image samples.

[0148] Optionally, the second updating module includes a first updating sub-module and a second updating sub-module.

[0149] The first updating submodule is used to determine the average loss function corresponding to multiple image sample pairs according to the loss function corresponding to each image sample pair.

[0150] The second updating submodule is used to update the network parameters of the initial image dehazing model according to the average loss function.

[0151] The image defogging model training device 100 provided in the embodiments of the present disclosure can execute the technical solution of the image defogging model training method shown in any of the above embodiments. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the image defogging model training method. Please refer to the implementation principle and beneficial effects of the image defogging model training method, which will not be repeated here.

[0152] Embodiment 6

[0153] Fig.11 is a structural diagram of an image defogging device 110 provided according to the sixth embodiment of the present disclosure, for example, see Fig.11 As shown, the image defogging device 110 may include:

[0154] The acquisition unit 1101 is used to acquire the fog image to be processed.

[0155] The processing unit 1102 is used to input the fog image to be processed into the image defogging model to obtain a clear image corresponding to the fog image to be processed.

[0156] Among them, the image dehazing model is trained by constructing a first loss function corresponding to each image sample pair based on the foggy image in each image sample pair and the output result corresponding to each image sample pair, and by training a second loss function corresponding to each image sample pair based on the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair.

[0157] The image defogging device 110 provided in the embodiment of the present disclosure can execute the technical solution of the image defogging method shown in the above embodiment. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the image defogging method. Please refer to the implementation principle and beneficial effects of the image defogging method, and no further details will be given here.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0160] Fig.121 is a schematic block diagram of an electronic device 120 provided by an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Fig.12 As shown, the device 120 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 120 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0162] A number of components in the device 120 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 120 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0163] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the training method of the image defogging model or the image defogging method. For example, in some embodiments, the training method of the image defogging model or the image defogging method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 120 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the training method of the image defogging model or the image defogging method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the image defogging model training method or the image defogging method in any other appropriate manner (for example, by means of firmware).

[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for an image defogging model, comprising: Acquire a plurality of image sample pairs, wherein each image sample pair comprises a fog image and a preset clear image corresponding to the fog image; Inputting the fog image in each image sample pair into the initial image defogging model to obtain the output results corresponding to each image sample pair; wherein the output results include the illumination information image, the fog component image, and the predicted clear image corresponding to the fog image; According to the fog image in each image sample pair and the output result corresponding to each image sample pair, a first loss function corresponding to each image sample pair is constructed, and according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair, a second loss function corresponding to each image sample pair is constructed; Update the network parameters of the initial image dehazing model according to the first loss function and the second loss function corresponding to each pair of image samples; The constructing a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and the output result corresponding to each image sample pair includes: Reconstructing the reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair; According to the fog image in each image sample pair and its corresponding reconstructed fog image, a first loss function corresponding to each image sample pair is constructed.

2. The method according to claim 1, wherein: The initial image dehazing model includes a downsampling network and three upsampling networks; Inputting the fog image in each image sample into the initial image defogging model to obtain the output results corresponding to each image sample pair, including: Inputting the fog image in each image sample into the initial image defogging model, performing encoding processing through the downsampling network, and obtaining a feature map corresponding to each fog image; The feature maps corresponding to the fog images are decoded through three upsampling networks respectively to obtain output results corresponding to the image sample pairs.

3. The method according to claim 2, wherein: The decoding process is performed on the feature maps corresponding to the fog images by three upsampling networks respectively to obtain the output results corresponding to the image sample pairs, including: Performing enhancement processing on the feature maps corresponding to the respective fog images to obtain target feature maps corresponding to the respective fog images; The target feature maps corresponding to the fog images are decoded respectively through the three upsampling networks to obtain output results corresponding to the image sample pairs.

4. The method according to claim 3, wherein: The step of performing enhancement processing on the feature maps corresponding to the respective fog images to obtain target feature maps corresponding to the respective fog images includes: Determine the channel features and pixel features in the feature map corresponding to each fog image; The channel features and pixel features in the feature maps corresponding to the fog images are enhanced respectively to obtain the target feature maps corresponding to the fog images.

5. The method according to any one of claims 1 to 4, wherein: The updating of the network parameters of the initial image defogging model according to the first loss function and the second loss function corresponding to each pair of image samples includes: Determine the loss function corresponding to each image sample pair according to the first loss function and the second loss function corresponding to each image sample pair; According to the loss function corresponding to each pair of image samples, the network parameters of the initial image defogging model are updated.

6. The method according to claim 5, wherein: The updating of the network parameters of the initial image defogging model according to the loss functions corresponding to the image sample pairs includes: Determine an average loss function corresponding to the plurality of image sample pairs according to the loss function corresponding to each image sample pair; According to the average loss function, the network parameters of the initial image dehazing model are updated.

7. An image defogging method, comprising: Obtaining a fog image to be processed; Inputting the foggy image to be processed into an image defogging model to obtain a clear image corresponding to the foggy image to be processed; The image defogging model is obtained by training a first loss function corresponding to each image sample pair constructed according to the foggy image in each image sample pair and the output result corresponding to each image sample pair, and a second loss function corresponding to each image sample pair constructed according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair; The constructing a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and the output result corresponding to each image sample pair includes: Reconstructing the reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair; According to the fog image in each image sample pair and its corresponding reconstructed fog image, a first loss function corresponding to each image sample pair is constructed.

8. A training device for an image defogging model, comprising: An acquisition unit, configured to acquire a plurality of image sample pairs, wherein each image sample pair comprises a fog image and a preset clear image corresponding to the fog image; A processing unit, configured to input the fog image in each image sample pair into the initial image defogging model to obtain an output result corresponding to each image sample pair; wherein the output result includes an illumination information image, a fog component image, and a predicted clear image corresponding to the fog image; A construction unit, configured to construct a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and the output result corresponding to each image sample pair, and to construct a second loss function corresponding to each image sample pair according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair; An updating unit, configured to update the network parameters of the initial image dehazing model according to the first loss function and the second loss function corresponding to each pair of image samples; The construction unit comprises a first construction module and a second construction module; The first construction module is used to reconstruct the reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair; The second construction module is used to construct a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and its corresponding reconstructed fog image.

9. The device according to claim 8, wherein: The initial image defogging model includes a downsampling network and three upsampling networks; the processing unit includes a first processing module and a second processing module; The first processing module is used to input the fog image in each image sample into the initial image defogging model, perform encoding processing through the downsampling network, and obtain a feature map corresponding to each fog image; The second processing module is used to decode the feature maps corresponding to the fog images through three upsampling networks respectively to obtain output results corresponding to the image sample pairs.

10. The device according to claim 9, wherein: The second processing module includes a first processing submodule and a second processing submodule; The first processing submodule is used to perform enhancement processing on the feature maps corresponding to the fog images to obtain target feature maps corresponding to the fog images; The second processing submodule is used to decode the target feature maps corresponding to the fog images through the three upsampling networks respectively to obtain output results corresponding to the image sample pairs.

11. The device according to claim 10, The first processing submodule is specifically used to determine the channel features and pixel features in the feature map corresponding to each fog image; and enhance the channel features and pixel features in the feature map corresponding to each fog image respectively to obtain the target feature map corresponding to each fog image.

12. The device according to any one of claims 8 to 11, wherein: The updating unit includes a first updating module and a second updating module; The first updating module is used to determine the loss function corresponding to each image sample pair according to the first loss function and the second loss function corresponding to each image sample pair; The second updating module is used to update the network parameters of the initial image dehazing model according to the loss function corresponding to each pair of image samples.

13. The device according to claim 12, wherein: The second updating module includes a first updating submodule and a second updating submodule; The first updating submodule is used to determine the average loss function corresponding to the plurality of image sample pairs according to the loss function corresponding to each image sample pair; The second updating submodule is used to update the network parameters of the initial image defogging model according to the average loss function.

14. An image defogging device, comprising: An acquisition unit, used for acquiring a fog image to be processed; A processing unit, used for inputting the fog image to be processed into an image defogging model to obtain a clear image corresponding to the fog image to be processed; The image defogging model is obtained by training a first loss function corresponding to each image sample pair constructed according to the foggy image in each image sample pair and the output result corresponding to each image sample pair, and a second loss function corresponding to each image sample pair constructed according to the preset clear image in each image sample pair and the predicted clear image in the output result corresponding to each image sample pair; The constructing a first loss function corresponding to each image sample pair according to the fog image in each image sample pair and the output result corresponding to each image sample pair includes: Reconstructing the reconstructed fog image corresponding to each image sample pair according to the illumination information image, the fog component image, and the predicted clear image in the output results corresponding to each image sample pair; According to the fog image in each image sample pair and its corresponding reconstructed fog image, a first loss function corresponding to each image sample pair is constructed.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the image defogging model described in any one of claims 1-6, or execute the image defogging method described in claim 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the image defogging model training method described in any one of claims 1 to 6; or, to execute the image defogging method described in claim 7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the training method of the image defogging model described in any one of claims 1 to 6; or, implements the steps of the image defogging method described in claim 7.

Images