Image defogging method and device, electronic equipment and storage medium

Abstract

The invention discloses an image defogging method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target night image to be defogged; carrying out defogging processing on the target night image through a night defogging model to obtain a defogged target night fogless image; the training process of the night defogging model comprises the following steps: defogging a first night foggy image through an initial night defogging model to obtain a first night defogged image; performing depth estimation on the first night defogged image through a night image depth estimation network to obtain a first night depth image; and based on the difference between the first night depth image and the first daytime depth image, training the initial night defogging model and the night image depth estimation network to obtain a night defogging model. According to the method provided by the invention, the defogging processing of the night image is realized, the defogging effect of the night defogging model is relatively good, and the accuracy of the defogged target night fog-free image is relatively high.

Description

Technical Field

[0001] This application relates to the field of electronic information technology, and more specifically, to an image dehazing method, apparatus, electronic device, and storage medium. Background Technology

[0002] When light propagates through media such as fog and haze, the scattering effect of particles results in low-quality images captured by sensors, which significantly limits the usability of these images. Image dehazing refers to processing images affected by atmospheric interference factors such as fog, haze, rain, and mist using algorithms to restore the image to the clarity of the real scene. This technology has wide applications in many fields, such as autonomous driving and security monitoring.

[0003] In related technologies, a neural network model can be used to dehaze daytime images to obtain dehazed daytime images. However, this method is difficult to use for dehazing nighttime images. Summary of the Invention

[0004] In view of this, embodiments of this application propose an image dehazing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, embodiments of this application provide an image dehazing method, the method comprising: acquiring a target nighttime image to be dehazed; performing dehazing processing on the target nighttime image using a nighttime dehazing model to obtain a dehazed target nighttime fog-free image; wherein the training process of the nighttime dehazing model comprises: performing dehazing processing on a first nighttime foggy image using an initial nighttime dehazing model to obtain a first nighttime dehazed image; performing depth estimation on the first nighttime dehazed image using a nighttime image depth estimation network to obtain a first nighttime depth image; training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and a first daytime depth image, and obtaining the trained initial nighttime dehazing model as the nighttime dehazing model; wherein the first daytime depth image is the depth image corresponding to a first daytime fog-free image with image content consistent with the first nighttime foggy image.

[0006] Secondly, embodiments of this application provide an image dehazing device, comprising: an acquisition module for acquiring a target nighttime image to be dehazed; and a dehazing module for performing dehazing processing on the target nighttime image using a nighttime dehazing model to obtain a dehazed target nighttime fog-free image; wherein the training process of the nighttime dehazing model includes: performing dehazing processing on a first nighttime foggy image using an initial nighttime dehazing model to obtain a first nighttime dehazing image; performing depth estimation on the first nighttime dehazing image using a nighttime image depth estimation network to obtain a first nighttime depth image; training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and a first daytime depth image, and obtaining the trained initial nighttime dehazing model as the nighttime dehazing model; wherein the first daytime depth image is the depth image corresponding to a first daytime fog-free image with image content consistent with the first nighttime foggy image.

[0007] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the above-described method.

[0009] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the method described above.

[0010] This application provides an image dehazing method, apparatus, electronic device, and storage medium. In this application, a nighttime dehazing model is used to dehaze a target nighttime image. During the training process of the nighttime dehazing model, the difference between the depth of a first nighttime dehazed image (indicated by a first nighttime depth image) and the depth of a first daytime fog-free image (indicated by a first daytime depth image) with consistent image content is used to train the initial nighttime dehazing model. This increases the nighttime image depth estimation network's ability to recognize the depth of nighttime scenes, indirectly improving the nighttime dehazing model's understanding of the depth of nighttime scenes. Consequently, the nighttime dehazing model has better dehazing capabilities for nighttime scenes, resulting in a more accurate and realistic target nighttime fog-free image output by the nighttime dehazing model, better matching the real scene corresponding to the target nighttime image. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram is shown illustrating the application scenarios applicable to the embodiments of this application;

[0013] Figure 2 A flowchart illustrating a training method for a nighttime defogging model according to an embodiment of this application is shown;

[0014] Figure 3 Shown Figure 2 A flowchart of step S130 in one embodiment is shown in the corresponding example;

[0015] Figure 4 Shown Figure 2 The flowchart of step S130 in another embodiment is shown in the corresponding embodiment;

[0016] Figure 5 Shown Figure 4 A flowchart of step S1322 in one embodiment is shown in the corresponding example;

[0017] Figure 6 Shown Figure 5 A flowchart of step S13223 in one embodiment is shown in the corresponding example;

[0018] Figure 7 Shown Figure 2 The flowchart of step S130 in another embodiment is shown in the corresponding embodiment;

[0019] Figure 8 Shown Figure 2 The flowchart of step S130 in another embodiment is shown in the corresponding embodiment;

[0020] Figure 9 This illustration shows a branch process in the training process of an initial nighttime defogging model according to an embodiment of this application;

[0021] Figure 10 Shown Figure 9 A schematic diagram of the data processing process in branch two of the training process of the nighttime defogging model;

[0022] Figure 11 A flowchart of an image dehazing method according to an embodiment of this application is shown;

[0023] Figure 12This illustration shows a block diagram of a defogging device according to an embodiment of this application;

[0024] Figure 13 A structural block diagram of an electronic device for performing a graph dehazing method according to an embodiment of this application is shown. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0026] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the application. It should be noted that "multiple" as used herein refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0028] Please refer to Figure 1 The diagram illustrates an application scenario applicable to the embodiments of this application. This application scenario includes a terminal 110, a server 120, and a database 130.

[0029] Terminal 110 can be, for example, a smartphone, tablet, e-book reader, music player, wearable device, smart home device, in-vehicle terminal, etc. Terminal 110 has an image dehazing client installed. This client can perform dehazing processing on a target nighttime image to obtain a fog-free nighttime image. Alternatively, the client can send an image dehazing request so that server 120 can perform dehazing processing on the target nighttime image based on the request to obtain a fog-free nighttime image.

[0030] The server 120 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0031] The remote database 130 can be an independent storage device, a cluster of multiple storage devices, a distributed system, or a cloud storage device that provides cloud services. The remote database 130 can store various images and data required during the training process.

[0032] The server 120 can directly obtain sample data such as the first night foggy image and the first day fog-free image from the remote database 130 (referring to various data including training the night defogging model). Then, it determines the first night depth image based on the first night foggy image and the first day depth image based on the first day fog-free image. After that, the server 120 trains the initial night defogging model and the night image depth estimation network based on the difference between the first night depth image and the first day depth image, and obtains the trained initial night defogging model as the night defogging model and caches the night defogging model.

[0033] Then, terminal 110 can generate and send an image dehazing request to server 120 (through the image dehazing client). Server 120 performs dehazing processing on the target night image based on the image dehazing request to obtain the target night fog-free image. Finally, server 120 returns the target night fog-free image to terminal 110 so that terminal 110 can display the target night fog-free image.

[0034] In some implementations, the server 120 itself can respond to the image dehazing operation triggered locally by the user, perform dehazing processing on the target night image, and obtain the target night image without fog.

[0035] Alternatively, the terminal 110 itself can respond to the image dehazing operation triggered locally by the user, perform dehazing processing on the target night image, and obtain a target night image without fog.

[0036] Of course, when the terminal 110 implements the image dehazing process of this application, after the server 120 obtains the night dehazing model, it can send the night dehazing model to the terminal 110, and the terminal 110 stores the night dehazing model so that the terminal can implement image dehazing.

[0037] It is easy to understand that the terminal 110 can also obtain sample data such as the first night foggy image and the first day fog-free image from the remote database 130 through the server 120, and determine the first night depth image based on the first night foggy image, and determine the first day depth image based on the first day fog-free image. Then, the terminal 110 trains the initial night defogging model and the night image depth estimation network based on the difference between the first night depth image and the first day depth image, and obtains the trained initial night defogging model as the night defogging model. Finally, the terminal 110 caches the night defogging model.

[0038] In some other implementations, the remote database 130 may be a storage module integrated within the server 120 to store data such as images.

[0039] To more clearly explain the solution of this application, the following embodiments all use electronic devices as the execution subject of the image dehazing method.

[0040] Please see Figure 2 , Figure 2 The flowchart illustrates a training method for a nighttime defogging model according to an embodiment of this application. This method is applied to an electronic device, which may be... Figure 1 The terminal 110 or server 120 in the middle, the method may include:

[0041] S110. The foggy image of the first night is processed by the initial nighttime defogging model to obtain the defogging image of the first nighttime.

[0042] In this application, the initial nighttime dehazing model refers to the nighttime dehazing model to be trained, and the first nighttime foggy image refers to a foggy image at night, which can be a real image obtained by taking a picture of a foggy scene at night.

[0043] Initial nighttime dehazing models can be based on network structures such as convolutional networks, deep neural networks, and encoders to achieve image dehazing.

[0044] First, the foggy image of the first night is input into the initial night defogging model, which performs defogging processing to obtain the defogging image, which is used as the first night defogging image.

[0045] In this application, both the defogging and fogging processes follow the atmospheric degradation model of nighttime images, which can be expressed as:

[0046] I(x) = R(x) + g(x)

[0047] R(x)=J(x)t(x)+A(x)(1-t(x))

[0048]

[0049] Where x represents the position of a pixel in the image, R(x) is the image affected by haze, g(x) is the glow image affected by nighttime light sources, I(x) is the observed foggy nighttime image, J(x) is the scene radiance, which can be understood as the image after defogging, t(x) is the transmission image of scene to camera transmittance, and A(x) is the global atmospheric light composition, which is a known quantity. a (x) is a component of the point light source. Let be the atmospheric point spread function. The transmitted image t(x) reflects the fog density and can be represented by the scene depth d(x) and the scattering coefficient β: t(x) = e^(-π / 2) * π / 2 ... -βd(x) .

[0050] Therefore, in this application, the initial nighttime dehazing model may include a nighttime image dehazing network (G). NDeh aze The nighttime image dehazing network consists of two parts: an initial nighttime dehazing model and an atmospheric degradation model (i.e., the aforementioned nighttime image atmospheric degradation model). The initial nighttime dehazing model can follow a recurrent generative adversarial network framework. The nighttime image dehazing network comprises a backbone network, a transmission image estimation subnetwork, a scattering coefficient estimation subnetwork, and a glow image estimation network. This invention does not limit the network structure used for the backbone network and the three subnetworks. At this point, the nighttime foggy image I is input into the nighttime image dehazing network (G). NDeh eze The process involves processing the data to obtain the transmitted image t, the scattering coefficient β, and the glow image g. This process can be expressed as: (t, β, g) = G NDeh aze (I).

[0051] Accordingly, S110 may include:

[0052] A1. The first nighttime foggy image is preprocessed by using the initial nighttime defogging model to obtain the first transmission image and the first glow image;

[0053] A2. Using the initial nighttime defogging model, based on the first transmitted image and the first glow image, the first nighttime foggy image is defogging and reprocessed to obtain the first nighttime defogging image.

[0054] That is, the first nighttime foggy image is preprocessed by the nighttime image dehazing network to obtain the first transmission image and the first glow image. Of course, the nighttime image dehazing network also outputs a third piece of information - the first scattering coefficient image. Then, the first nighttime foggy image is dehazed again by the atmospheric degradation model based on the first transmission image and the first glow image (the first scattering coefficient image can be omitted for dehazing reprocessing) to obtain the first nighttime dehazed image.

[0055] Specifically, the first night's foggy image INH Input nighttime image dehazing network G NDeh aze This allows us to obtain the nighttime image dehazing network G. NDeh aze Estimated first transmitted image First scattering coefficient image and the first glow image Then, based on the atmospheric degradation model, the estimated first nighttime dehazing image can be calculated. At this point, the above defogging process can be represented as:

[0056]

[0057] In this application, the nighttime image dehazing network (G) NDeh aze The dehazing process of the night image dehazing network (G) is actually based on generating a fog-free dehazed image of the first night based on the foggy image of the first night. Therefore, the night image dehazing network (G) NDeh aze It is actually a generative network (e.g., it could be a generative adversarial network (GAN)).

[0058] S120. The depth of the first nighttime dehazed image is estimated by using a nighttime image depth estimation network to obtain the first nighttime depth image.

[0059] A nighttime image depth estimation network is a network used to estimate the depth values ​​of nighttime images. The nighttime image depth estimation network is denoted as G. NDepth Therefore, the estimated first nighttime dehazed image can be obtained. The nighttime image depth estimation network is input to obtain the first nighttime depth image corresponding to the first nighttime image.

[0060] In this application, the nighttime image depth estimation network G NDepth The depth estimation process is actually based on generating a depth image from the first nighttime dehazed image—the first nighttime depth image. Therefore, the nighttime image depth estimation network G... NDepth It is actually a generative network (e.g., a convolutional neural network).

[0061] S130. Based on the difference between the first nighttime depth image and the first daytime depth image, train the initial nighttime dehazing model and the nighttime image depth estimation network, and obtain the trained initial nighttime dehazing model as the nighttime dehazing model.

[0062] The first daytime depth image is the depth image corresponding to the first daytime fog-free image, which has the same image content as the first nighttime foggy image. That is, the first daytime fog-free image and the first nighttime foggy image have the same image content, the only difference being that the first daytime fog-free image is daytime and fog-free, while the first nighttime foggy image is nighttime and foggy. All other content of the first daytime fog-free image and the first nighttime foggy image is the same.

[0063] In this application, a pre-trained depth estimation network or the depth estimation algorithm in OpenCV (referred to in this application as the daytime depth estimation module G) can be used. DDepth ), for the foggy image I of the first night NH First daytime fog-free image I (paired) DC By performing depth estimation, we can obtain the depth image of the first daytime image corresponding to the first daytime fog-free image.

[0064] Through the first night depth image Indicates the depth value of a certain content at night, based on the first daytime depth image. The depth value of the same content during the day is indicated, and then the loss is determined by the difference between the first night depth image and the first day depth image. The initial night dehazing model is trained by the loss, and the trained initial night dehazing model is obtained as the night dehazing model used in the dehazing process.

[0065] As mentioned above, the initial nighttime dehazing model can include two parts: a nighttime image dehazing network and an atmospheric degradation model. The atmospheric degradation model is actually an operational rule and does not include trainable parameters. Therefore, in this embodiment, the nighttime image depth estimation network and the nighttime image dehazing network are actually trained by the loss determined by the difference between the first nighttime depth image and the first daytime depth image, and the trained nighttime image dehazing network and atmospheric degradation model are obtained as the nighttime dehazing model used in the dehazing process.

[0066] Optionally, in this application, the first depth loss can be determined based on the difference between the first nighttime depth image and the first daytime depth image. For example, the difference between the first nighttime depth image and the first daytime depth image can be determined, and mean squared error loss and absolute value error loss can be calculated, with the calculated results used as the first depth loss. For example, when the L1 norm of the difference between the first nighttime depth image and the first daytime depth image is determined as the first depth loss, the calculation process for the first depth loss is as follows:

[0067] In this embodiment, during the training of the nighttime dehazing model, the difference between the depth of the first nighttime dehazing image (indicated by the first nighttime depth image) and the depth of the first daytime fog-free image (indicated by the first daytime depth image) with consistent image content is used to train the initial nighttime dehazing model. This increases the nighttime image depth estimation network's ability to recognize the depth of nighttime scenes, indirectly improving the nighttime dehazing model's understanding of the depth of nighttime scenes, thereby enabling the nighttime dehazing model to have better dehazing capabilities for nighttime scenes.

[0068] In some embodiments, such as Figure 3 As shown, S130 may include:

[0069] S1311. Using an atmospheric degradation model, fogging is applied to the first night's defogging image to obtain the second night's foggy image.

[0070] The initial nighttime dehazing model can include a nighttime image dehazing network and an atmospheric degradation model. In addition to dehazing capability (achieved through the nighttime image dehazing network), the initial nighttime dehazing model also has fogging capability (through the atmospheric degradation model). Therefore, the atmospheric degradation model in the initial nighttime dehazing model can be used to fog the first nighttime dehazing image after dehazing to obtain a synthesized second nighttime foggy image.

[0071] In some implementations, as described above, the nighttime image dehazing network in the initial nighttime dehazing model, after dehazing the first nighttime foggy image, also outputs a first scattering coefficient image. and the first glow image It also outputs the first transmitted image. At this point, S1311 includes: using an atmospheric degradation model, based on the first nighttime depth image, the first scattering coefficient image, and the first glow image, performing fogging processing on the first nighttime defogging image to obtain the second nighttime foggy image.

[0072] Specifically, it could be using the first nighttime image depth image. First scattering coefficient image and the first glow image The atmospheric degradation model in the initial nighttime defogging model was used to analyze the first nighttime defogging image. A fogging process was performed to obtain a composite second nighttime foggy image. At this point, the above fogging process can be represented as:

[0073]

[0074] S1312. Based on the differences between the foggy images of the first night and the foggy images of the second night, and the differences between the depth images of the first night and the depth images of the first day, the initial night defogging model and the night image depth estimation network are trained, and the trained initial night defogging model is obtained as the night defogging model.

[0075] The first night foggy image is a real night foggy image, while the second night foggy image is a synthesized image after defogging and fogging processes. Therefore, the loss can be determined based on the difference between the first night foggy image and the second night foggy image, as well as based on the difference between the first night depth image and the first day depth image. The initial night defogging model and the night image depth estimation network can then be trained based on these two losses.

[0076] Optionally, the differences between the foggy images from the first night and the foggy images from the second night can be determined, and mean squared error loss and absolute value error loss can be calculated. The results are used as the first cycle consistency loss. For example, when determining the L1 norm of the difference between the foggy images from the first night and the foggy images from the second night as the first cycle consistency loss, the calculation process for the first cycle consistency loss is as follows:

[0077] Simultaneously, a first depth loss can be determined based on the difference between the first nighttime depth image and the first daytime depth image. Then, the first depth loss and the first cycle consistency loss are summed and weighted, and the result is used as the total loss. The initial nighttime dehazing model (mainly training the nighttime image dehazing network within it, but not the atmospheric degradation model, because the atmospheric degradation model is an operational rule and does not include trainable parameters) and the nighttime image depth estimation network are trained using the total loss.

[0078] In this embodiment, the difference between the foggy images of the first night and the foggy images of the second night is used to indicate the consistency of the defogging and fogging processes of the initial night defogging model. Thus, when the initial night defogging model is trained using the difference between the foggy images of the first night and the foggy images of the second night, the initial night defogging model can learn the defogging process better, resulting in a better defogging ability and higher defogging accuracy of the trained night defogging model.

[0079] In some embodiments, such as Figure 4 As shown, S130 includes:

[0080] S1321. Depth estimation is performed on the nighttime fog-free image using a nighttime image depth estimation network to obtain a second nighttime depth image.

[0081] Among them, nighttime fog-free images refer to nighttime images without fog, and nighttime fog-free images I NC Input nighttime image depth estimation network G NDepth Second nighttime image obtained

[0082] It is worth mentioning that the nighttime fog-free image in this application can be an image with content inconsistent with the first nighttime foggy image in S110 mentioned above. This reduces the difficulty of obtaining the nighttime fog-free image and the first nighttime foggy image (because the nighttime foggy image and the nighttime fog-free image actually need to be determined based on the same scene at the same time, and the content of the nighttime foggy image and the nighttime fog-free image should be kept as consistent as possible. However, it is difficult for the same scene at the same time to have both foggy and fog-free situations. Therefore, the nighttime foggy image and the nighttime fog-free image can only be determined for the same scene at different times. When determined at different times, it is difficult to ensure that the content is consistent). This improves the training efficiency of the initial nighttime defogging model.

[0083] S1322. Based on the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, train the initial nighttime dehazing model and the nighttime image depth estimation network, and obtain the trained initial nighttime dehazing model as the nighttime dehazing model.

[0084] The second daytime depth image is the depth image corresponding to the second daytime fog-free image, which has the same image content as the nighttime fog-free image. That is, the nighttime fog-free image and the second daytime fog-free image have the same image content, the only difference being that the second daytime fog-free image is daytime, while the nighttime fog-free image is nighttime. All other content of the nighttime fog-free image and the second daytime fog-free image is the same.

[0085] The aforementioned daytime depth estimation module G can be used DDepth For the input nighttime fog-free image I NC Paired second daytime fog-free image I DC Depth estimation was performed to obtain the second daytime depth image.

[0086] Then, based on the differences between the first nighttime depth image and the first daytime depth image, as well as the differences between the second nighttime depth image and the second daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network can be trained.

[0087] Optionally, the mean squared error loss and absolute value error loss can be calculated based on the difference between the second nighttime depth image and the second daytime depth image, and the calculated results can be used as the second depth loss. For example, when determining the L1 norm of the difference between the second nighttime depth image and the second daytime depth image as the second depth loss, the calculation process for the second depth loss is as follows:

[0088] Accordingly, a first depth loss can be determined based on the difference between the first nighttime depth image and the first daytime depth image. Then, the first depth loss and the second depth loss are summed and weighted, and the result is used as the total loss. The initial nighttime dehazing model and the nighttime image depth estimation network are trained using the total loss.

[0089] In this embodiment, the difference between the second nighttime depth image and the second daytime depth image, as well as the difference between the first nighttime depth image and the first daytime depth image, are further combined to train the initial nighttime dehazing model. This further enhances the nighttime image depth estimation network's ability to recognize the depth of nighttime scenes and improves the nighttime dehazing model's understanding of the depth of nighttime scenes, thereby enabling the nighttime dehazing model to have better dehazing capabilities for nighttime scenes.

[0090] In some embodiments, such as Figure 5 As shown, S1322 also includes:

[0091] S13221. Using an atmospheric degradation model, based on randomly sampled second glow image, second scattering coefficient image, and second nighttime depth image, fogging is applied to the nighttime fog-free image to obtain a third nighttime foggy image.

[0092] The initial nighttime dehazing model can include a nighttime image dehazing network and an atmospheric degradation model, thereby generating a second scattering coefficient image through random sampling. Second glow image Using the second nighttime image depth image Second scattering coefficient image Second glow image Based on the atmospheric degradation model in the initial nighttime defogging model, the input nighttime fog-free image I... NC Fog was added to obtain the composite third nighttime foggy image. At this point, the above fogging process can be represented as:

[0093]

[0094] S13222. The third night foggy image is preprocessed by using the initial night defogging model to obtain the third glow image.

[0095] In other words, the night image dehazing network G in the initial night dehazing model NDeh aze Based on the third nighttime foggy image, a nighttime image dehazing network G is obtained through dehazing preprocessing. NDeh aze The third glow image output Similarly, the nighttime image dehazing network G... NDeh aze It also output a second transmitted image. and the third scattering coefficient image

[0096] S13223. Based on the differences between the second glow image and the third glow image, the differences between the first night depth image and the first day depth image, and the differences between the second night depth image and the second day depth image, the initial night dehazing model and the night image depth estimation network are trained, and the trained initial night dehazing model is obtained as the night dehazing model.

[0097] The mean squared error loss and absolute value error loss can be calculated based on the difference between the second and third glow images, and the results are used as the first contrast loss. For example, when determining the squared loss of the difference between the second and third glow images as the first contrast loss, the calculation process for the first contrast loss is as follows:

[0098] Accordingly, the first depth loss can be determined based on the difference between the first night depth image and the first day depth image, and the second depth loss can be determined based on the difference between the second night depth image and the second day depth image. Then, the first contrast loss, the first depth loss, and the second depth loss are summed and weighted, and the result is used as the total loss. The initial night dehazing model and the night image depth estimation network are trained using the total loss.

[0099] In this embodiment, the initial nighttime dehazing model is further trained by combining the differences between the second glow image and the third glow image, the differences between the second nighttime depth image and the second daytime depth image, and the differences between the first nighttime depth image and the first daytime depth image. By combining the differences between the second glow image and the third glow image, the influence of nighttime imaging fog and light on imaging can be decoupled, thereby improving the dehazing capability of the trained nighttime dehazing model for nighttime scenes, enabling the trained nighttime dehazing model to perform accurate dehazing processing on nighttime scene images.

[0100] In addition, the first contrast loss is determined based on the difference between the second glow image and the third glow image, rather than the difference between the two glow coefficients. This makes the first contrast loss more accurate in indicating the impact of fog and light on nighttime imaging, thus making the trained nighttime defogging model have a higher defogging capability in nighttime scenes.

[0101] In some embodiments, as described above, after the initial nighttime dehazing model performs dehazing preprocessing on the third nighttime foggy image, it also outputs a third scattering coefficient image. Accordingly, S13223 further includes: training the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the second and third scattering coefficient images, the differences between the second and third glow images, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, and obtaining the trained initial nighttime dehazing model as the nighttime dehazing model.

[0102] The mean squared error loss and absolute value error loss can be calculated based on the difference between the second and third scattering coefficient images, and the results are used as the second contrast loss. For example, when determining the squared loss of the difference between the second and third scattering coefficient images as the second contrast loss, the calculation process for the second contrast loss is as follows:

[0103] Accordingly, a first contrast loss can be determined based on the difference between the second glow image and the third glow image, a first depth loss can be determined based on the difference between the first night depth image and the first day depth image, and a second depth loss can be determined based on the difference between the second night depth image and the second day depth image. Then, the first contrast loss, the second contrast loss, the first depth loss, and the second depth loss are summed and weighted, and the result is used as the total loss. The initial night dehazing model and the night image depth estimation network are trained using the total loss.

[0104] In this embodiment, the initial nighttime dehazing model is further trained by combining the differences between the second and third scattering coefficient images, the differences between the second and third glow images, the differences between the second nighttime depth image and the second daytime depth image, and the differences between the first nighttime depth image and the first daytime depth image. By using the differences between the second and third scattering coefficient images, the influence of atmospheric composition on nighttime imaging can be decoupled, thereby improving the dehazing capability of the trained nighttime dehazing model for nighttime scenes, enabling the trained nighttime dehazing model to perform accurate dehazing processing on nighttime scene images.

[0105] In addition, the second contrast loss is determined based on the difference between the second and third scattering coefficient images, rather than the difference between the two scattering coefficients. This makes the second contrast loss more accurate in indicating the influence of atmospheric composition on nighttime imaging, thus resulting in a higher defogging capability for nighttime scenes in the trained nighttime defogging model.

[0106] In some embodiments, as described above, after the initial nighttime dehazing model performs dehazing preprocessing on the third nighttime foggy image, it also outputs a second transmitted image; correspondingly, as... Figure 6 As shown, S13223 also includes:

[0107] S132231. Using the initial nighttime defogging model, based on the third glow image and the second transmission image, the third nighttime foggy image is defogging and reprocessed to obtain the second nighttime defogging image.

[0108] Specifically, a second image transmission is used. and the third glow image Based on the atmospheric degradation model in the initial nighttime defogging model, the foggy images of the third night were analyzed. By performing defogging, an estimated second nighttime defogging image can be obtained. At this point, the above defogging process can be represented as:

[0109]

[0110] S132232. Based on the differences between the nighttime fog-free image and the second nighttime defogging image, the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, the initial nighttime defogging model and the nighttime image depth estimation network are trained, and the trained initial nighttime defogging model is obtained as the nighttime defogging model.

[0111] The nighttime fog-free image is a genuine nighttime fog-free image, while the second nighttime dehazed image is a synthesized nighttime fog-free image obtained through fogging-dehazing processing. At this point, based on the differences between the nighttime fog-free image and the second nighttime dehazed image, mean squared error loss and absolute value error loss can be calculated, and the results are used as the second cycle consistency loss. For example, when determining the L1 norm of the difference between the nighttime fog-free image and the second nighttime dehazed image as the second cycle consistency loss, the calculation process for the second cycle consistency loss is as follows:

[0112] Accordingly, a first contrast loss can be determined based on the difference between the second glow image and the third glow image, a first depth loss can be determined based on the difference between the first night depth image and the first day depth image, and a second depth loss can be determined based on the difference between the second night depth image and the second day depth image. Then, the second cycle consistency loss, the first contrast loss, the first depth loss, and the second depth loss are summed and weighted, and the result is used as the total loss. The initial night dehazing model and the night image depth estimation network are trained using the total loss.

[0113] In this embodiment, the initial nighttime defogging model is further trained by combining the differences between the nighttime fog-free image and the second nighttime defogging image, the differences between the second glow image and the third glow image, the differences between the second nighttime depth image and the second daytime depth image, and the differences between the first nighttime depth image and the first daytime depth image. The differences between the nighttime fog-free image and the second nighttime defogging image indicate the consistency of the fogging and defogging processes of the initial nighttime defogging model. Therefore, when the initial nighttime defogging model is trained by further combining the differences between the nighttime fog-free image and the second nighttime defogging image, the initial nighttime defogging model can better learn the defogging process, resulting in a better defogging ability and higher defogging accuracy of the trained nighttime defogging model.

[0114] In some embodiments, such as Figure 7 As shown, S130 also includes:

[0115] S1331. The discriminator distinguishes the nighttime fog-free image and the first nighttime defogging image respectively, and obtains the first discrimination result corresponding to the nighttime fog-free image and the second discrimination result corresponding to the first nighttime defogging image.

[0116] The discriminator can be used to distinguish between the nighttime fog-free image and the first nighttime dehazed image to identify the possibility that the nighttime fog-free image and the first nighttime dehazed image are real images or synthetic images.

[0117] That is, to use the nighttime fog-free image I NC Input discriminator D NDeh aze The first discrimination result D is obtained by performing discrimination. NDeh aze (INC), the first nighttime dehazing image Input discriminator D NDeh aze Perform the discrimination to obtain the second discrimination result.

[0118] S1332. Based on the first discrimination result and the second discrimination result, determine the adversarial loss.

[0119] The first adversarial loss L can be determined based on the first and second discrimination results. GAN (D NDeh aze ) and second-countermeasure loss L GAN (G NDeh aze Then, sum the first adversarial loss and the second adversarial loss to obtain the adversarial loss (L). GAN (D NDeh aze )+L GAN (G NDeh aze )).

[0120] For example, the process for determining the first adversarial loss can be as follows:

[0121]

[0122] Accordingly, the process for determining the second adversarial loss is as follows:

[0123]

[0124] S1333. Based on the difference between the first nighttime depth image and the first daytime depth image and the adversarial loss, train the initial nighttime dehazing model and the nighttime image depth estimation network, and obtain the trained initial nighttime dehazing model as the nighttime dehazing model.

[0125] The first depth loss can be determined based on the difference between the first night depth image and the first day depth image. Then, the first depth loss and the adversarial loss are summed or weighted and summed to obtain the total loss. The initial night dehazing model and the night image depth estimation network are trained using the total loss.

[0126] Of course, a discriminator is also present in this embodiment. Therefore, when training the initial nighttime dehazing model and the nighttime image depth estimation network, the discriminator is also trained simultaneously to increase the discriminator's discrimination ability. However, after training, the discriminator is discarded and only the trained initial nighttime dehazing model is used as the nighttime dehazing model.

[0127] In this embodiment, the initial night dehazing model is also trained by combining an adversarial loss that indicates the generative ability of the initial night dehazing model. This allows the initial night dehazing model to better learn the image reconstruction and generation capabilities, so as to gradually generate more accurate dehazed images during the training process. This results in a better dehazing ability and higher dehazing accuracy for the trained night dehazing model.

[0128] In some implementations, such as Figure 8 As shown, S130 includes:

[0129] S1341. Using an atmospheric degradation model, the first nighttime dehazed image is fogged to obtain a second nighttime foggy image; a nighttime image depth estimation network is used to estimate the depth of the nighttime fog-free image to obtain a second nighttime depth image; using an atmospheric degradation model, based on randomly sampled second glow image, second scattering coefficient image, and second nighttime depth image, the nighttime fog-free image is fogged to obtain a third nighttime foggy image; using an initial nighttime dehazing model, the third nighttime foggy image is dehazed to obtain a third glow image, third scattering coefficient image, and second transmission image; using an initial nighttime dehazing model, based on the third glow image and second transmission image, the third nighttime foggy image is dehazed again to obtain a second nighttime dehazed image; a discriminator is used to distinguish between the nighttime fog-free image and the first nighttime dehazed image, obtaining a first discrimination result corresponding to the nighttime fog-free image and a second discrimination result corresponding to the first nighttime dehazed image; based on the first discrimination result and the second discrimination result, the adversarial loss is determined.

[0130] The description of S1341 refers to the aforementioned second night fog image, second night depth image, third night fog image, third glow image, third scattering coefficient image, second transmission image, second night defogging image, and the process of determining the adversarial loss, and will not be repeated here.

[0131] S1342. Based on the differences between the first nighttime depth image and the first daytime depth image, the differences between the first nighttime foggy image and the second nighttime foggy image, the differences between the second nighttime depth image and the second daytime depth image, the differences between the second glow image and the third glow image, the differences between the second scattering coefficient image and the third scattering coefficient image, the differences between the nighttime fog-free image and the second nighttime dehazed image, and the adversarial loss, the initial nighttime dehazed model and the nighttime image depth estimation network are trained, and the trained initial nighttime dehazed model is obtained as the nighttime dehazed model.

[0132] Specifically, the following steps can be taken: First depth loss is determined based on the difference between the first nighttime depth image and the first daytime depth image; first cycle consistency loss is determined based on the difference between the first nighttime foggy image and the second nighttime foggy image; second depth loss is determined based on the difference between the second nighttime depth image and the second daytime depth image; second cycle consistency loss is determined based on the difference between the nighttime fog-free image and the second nighttime dehazed image; first contrast loss is determined based on the difference between the second glow image and the third glow image; second contrast loss is determined based on the difference between the second scattering coefficient image and the third scattering coefficient image; the sum of the first depth loss and the second depth loss is calculated as the first sum loss, and the sum of the first cycle consistency loss and the second cycle consistency loss is calculated as the second sum loss. Then, the initial nighttime dehazing model and the nighttime image depth estimation network are trained based on the weighted sum of the adversarial loss, the first sum loss, the second sum loss, the first contrast loss, and the second contrast loss.

[0133] The determination process of the first depth loss, the second depth loss, the first cycle consistency loss, the second cycle consistency loss, the first contrast loss, the second contrast loss, and the adversarial loss is as described in the foregoing embodiments and will not be repeated here.

[0134] The total loss can be calculated by weighting the adversarial loss, the first sum loss, the second sum loss, the first contrastive loss, and the second contrastive loss. total Represented as:

[0135]

[0136] Where, λ GAN , λ depth , λ cyc , λ β , λ g It is the weight that balances the various losses.

[0137] Of course, it is not difficult to understand that this application involves a first depth loss, a second depth loss, a first cycle consistency loss, a second cycle consistency loss, a first contrast loss, a second contrast loss, and an adversarial loss. When obtaining the first depth loss, at least one loss can be selected from the five losses: the second depth loss, the first cycle consistency loss, the second cycle consistency loss, the first contrast loss, the second contrast loss, and the adversarial loss. The selected loss is then weighted and summed with the first depth loss to obtain the total loss. The initial nighttime dehazing model and the nighttime image depth estimation network are trained using the total loss (of course, when adversarial loss is included, a discriminator also needs to be trained). The trained initial nighttime dehazing model is then obtained as the nighttime dehazing model. The various combinations of different losses are not elaborated in this application.

[0138] For example, the training process of the initial nighttime defogging model is as follows: Figures 9-10 As shown. The initial nighttime dehazing model includes a nighttime image dehazing network and an atmospheric degradation model.

[0139] First, refer to Figure 9 This involves the defogging-fogging process:

[0140] A nighttime image dehazing network is used to preprocess the first nighttime foggy image to obtain a first transmission image, a first glow image, and a first scattering coefficient image. Then, an atmospheric degradation model is used to further dehaze the first nighttime foggy image based on the first transmission image and the first glow image to obtain a first nighttime dehazed image. A nighttime depth estimation network is used to determine the first nighttime depth image corresponding to the first nighttime foggy image. Finally, an atmospheric degradation model is used to generate a second nighttime foggy image based on the first nighttime depth image, the first nighttime dehazed image, the first glow image, and the first scattering coefficient image.

[0141] Meanwhile, the daytime depth estimation module performs depth estimation based on the first daytime fog-free image paired with the first nighttime foggy image (if the content is the same, it is a pair) to obtain the first daytime depth image corresponding to the first daytime fog-free image.

[0142] Accordingly, the first depth loss can be determined based on the first daytime depth image and the first nighttime depth image, the adversarial loss can be determined based on the nighttime fog-free image and the first nighttime defogging image, and the first cycle consistency loss can be determined based on the first nighttime foggy image and the second nighttime foggy image.

[0143] Continue, refer to Figure 10 This involves the fogging and defogging process:

[0144] A nighttime depth estimation network is used to estimate the depth of the fog-free nighttime image, resulting in a second nighttime depth image. Then, an atmospheric degradation model is used to apply fog to the fog-free nighttime image based on randomly generated second glow images, second scattering coefficient images, and the second nighttime depth image, resulting in a third nighttime foggy image. Next, a nighttime image defogging network is used to preprocess the third nighttime foggy image to obtain a third glow image, a third scattering coefficient image, and a second transmission image. Finally, an atmospheric degradation model is used to further defog the third nighttime foggy image based on the third glow image and the second transmission image, resulting in a second nighttime defogging image.

[0145] Meanwhile, the daytime depth estimation module performs depth estimation based on the second daytime fog-free image paired with the nighttime fog-free image (pairing is considered if the content is the same), and obtains the second daytime depth image corresponding to the second daytime fog-free image.

[0146] Accordingly, a second depth loss is determined based on the second nighttime depth image and the second daytime depth image, a second cycle consistency loss is determined based on the nighttime fog-free image and the second nighttime defogging image, a first contrast loss is determined based on the second glow image and the third glow image, and a second contrast loss is determined based on the second scattering coefficient image and the third scattering coefficient image.

[0147] Finally, the total loss is determined by the first depth loss, the second depth loss, the first cycle consistency loss, the second cycle consistency loss, the first contrast loss, the second contrast loss, and the adversarial loss. The initial nighttime defogging model and the nighttime depth estimation network are trained using the total loss (of course, when determining the adversarial loss, the discriminator is used, and the discriminator also needs to be trained simultaneously). The trained initial nighttime defogging model is then obtained as the nighttime defogging model used for defogging processing.

[0148] In this embodiment, the initial nighttime dehazing model is trained by combining the first depth loss, the second depth loss, the first cycle consistency loss, the second cycle consistency loss, the first contrast loss, the second contrast loss, and the adversarial loss. This results in a nighttime dehazing model with high dehazing capability and good dehazing effect for nighttime images.

[0149] See also Figure 11 , Figure 11 This application illustrates a flowchart of an image dehazing method according to an embodiment of the present application. This method is applied to an electronic device, which may be... Figure 1 The terminal 110 or server 120 in the middle, the method may include:

[0150] S210. Obtain the target nighttime image to be dehazed.

[0151] The target nighttime image to be defogging refers to an image with fog at night. It can be an image taken by a camera or an image synthesized by a neural network model. This application does not limit the scope of the image.

[0152] S220. The target nighttime image is dehazed using a nighttime dehazing model to obtain a dehazed, fog-free nighttime image of the target.

[0153] The target nighttime image is directly input into the nighttime dehazing model for dehazing processing, and the resulting dehazed nighttime image is used as the target fog-free nighttime image. The training process of the nighttime dehazing model is as described in the previous embodiments and will not be repeated here.

[0154] As mentioned above, the nighttime dehazing model is trained from the initial nighttime dehazing model. The nighttime dehazing model includes a nighttime image dehazing network and an atmospheric degradation model. Therefore, the trained nighttime dehazing model includes a nighttime image dehazing network and an atmospheric degradation model. Accordingly, S220 includes: performing dehazing preprocessing on the target nighttime image through the nighttime image dehazing network in the nighttime dehazing model to obtain a target transmission image, a target glow image, and a target scattering coefficient image; then, performing dehazing reprocessing on the target nighttime image based on the target transmission image, target glow image, and target scattering coefficient image through the atmospheric degradation model in the nighttime dehazing model to obtain a target nighttime fog-free image.

[0155] At this point, the aforementioned process can be briefly described as follows: I of the target nighttime image T-NH Input the trained nighttime image dehazing network G NDehaze The estimated target transmission image is obtained. Target scattering coefficient image and target glow image Subsequently, based on the atmospheric degradation model, an estimated fog-free nighttime image of the target can be calculated. The defogging process can be represented as:

[0156]

[0157] In this embodiment, a nighttime dehazing model is used to dehaze the target nighttime image. During the training process of the nighttime dehazing model, the difference between the depth of a first nighttime dehazed image with consistent image content (indicated by the first nighttime depth image) and the depth of a first daytime fog-free image (indicated by the first daytime depth image) is used to train the initial nighttime dehazing model. This increases the nighttime image depth estimation network's ability to recognize the depth of nighttime scenes, indirectly improving the nighttime dehazing model's understanding of the depth of nighttime scenes. As a result, the nighttime dehazing model has a better dehazing ability for nighttime scenes, which in turn makes the target nighttime fog-free image output by the nighttime dehazing model more accurate and more consistent with the real scene corresponding to the target nighttime image.

[0158] Please see Figure 12 , Figure 12 This illustration shows a block diagram of an image dehazing apparatus according to an embodiment of this application. The image dehazing apparatus 800 includes:

[0159] The acquisition module 810 is used to acquire the target nighttime image to be dehazed;

[0160] The dehazing module 820 is used to dehaze the target night image using a night dehazing model to obtain a dehazed target night image without fog.

[0161] The training process for the nighttime defogging model includes:

[0162] The foggy image of the first night is dehazed by using the initial nighttime dehazing model to obtain the dehazed image of the first nighttime.

[0163] The first nighttime dehazed image is obtained by using a nighttime image depth estimation network to estimate the depth of the first nighttime image;

[0164] Based on the difference between the first nighttime depth image and the first daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network are trained, and the trained initial nighttime dehazing model is obtained as the nighttime dehazing model; wherein, the first daytime depth image is the depth image corresponding to the first daytime fog-free image with the same image content as the first nighttime foggy image.

[0165] The training process of the aforementioned nighttime defogging model is implemented through a training module.

[0166] Optionally, the initial nighttime dehazing model includes an atmospheric degradation model; the training module is also used to apply fogging to the first nighttime dehazing image using the atmospheric degradation model to obtain a second nighttime foggy image; based on the differences between the first nighttime foggy image and the second nighttime foggy image, and the differences between the first nighttime depth image and the first daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network are trained.

[0167] Optionally, after the initial nighttime dehazing model dehazes the first nighttime foggy image, it also outputs a first scattering coefficient image and a first glow image; the training module is also used to perform fogging processing on the first nighttime dehazing image based on the first nighttime depth image, the first scattering coefficient image and the first glow image using an atmospheric degradation model to obtain a second nighttime foggy image.

[0168] Optionally, the training module is also used to perform dehazing preprocessing on the first night foggy image using the initial night dehazing model to obtain a first transmission image and a first glow image; and to perform dehazing reprocessing on the first night foggy image based on the first transmission image and the first glow image using the initial night dehazing model to obtain a first night dehazing image.

[0169] Optionally, the training module is also used to perform depth estimation on the nighttime fog-free image through the nighttime image depth estimation network to obtain a second nighttime depth image; and to train the initial nighttime defogging model and the nighttime image depth estimation network based on the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image; wherein, the second daytime depth image is the depth image corresponding to the second daytime fog-free image with the same image content as the nighttime fog-free image.

[0170] Optionally, the training module is also used to perform fogging processing on the fog-free nighttime image based on the randomly sampled second glow image, second scattering coefficient image, and second nighttime depth image using an atmospheric degradation model to obtain a third nighttime foggy image; to perform defogging preprocessing on the third nighttime foggy image using an initial nighttime defogging model to obtain a third glow image; and to train the initial nighttime defogging model and the nighttime image depth estimation network based on the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image.

[0171] Optionally, after the initial nighttime dehazing model performs dehazing preprocessing on the foggy images of the third night, it also outputs a third scattering coefficient image; the training module is also used to train the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the second and third scattering coefficient images, the differences between the second and third glow images, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image.

[0172] Optionally, after the initial nighttime dehazing model performs dehazing preprocessing on the third nighttime foggy image, it also outputs a second transmission image. The training module is further used to perform dehazing reprocessing on the third nighttime foggy image based on the third glow image and the second transmission image using the initial nighttime dehazing model to obtain the second nighttime dehazing image. The initial nighttime dehazing model and the nighttime image depth estimation network are trained based on the differences between the nighttime fog-free image and the second nighttime dehazing image, the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image.

[0173] Optionally, the training module is further configured to use a discriminator to distinguish between the nighttime fog-free image and the first nighttime dehazed image, respectively, to obtain a first discrimination result corresponding to the nighttime fog-free image and a second discrimination result corresponding to the first nighttime dehazed image; based on the first discrimination result and the second discrimination result, determine the adversarial loss; and based on the difference between the first nighttime depth image and the first daytime depth image and the adversarial loss, train the initial nighttime dehazed model and the nighttime image depth estimation network.

[0174] Optionally, the training module is also used to perform fogging processing on the first nighttime defogging image using an atmospheric degradation model to obtain a second nighttime foggy image; to perform depth estimation on the nighttime fog-free image using a nighttime image depth estimation network to obtain a second nighttime depth image; to perform fogging processing on the nighttime fog-free image using an atmospheric degradation model based on randomly sampled second glow image, second scattering coefficient image, and second nighttime depth image to obtain a third nighttime foggy image; to perform defogging preprocessing on the third nighttime foggy image using an initial nighttime defogging model to obtain a third glow image, third scattering coefficient image, and second transmission image; and to perform defogging reprocessing on the third nighttime foggy image using the initial nighttime defogging model based on the third glow image and second transmission image to obtain a second nighttime defogging image. The discriminator distinguishes between the nighttime fog-free image and the first nighttime dehazed image, obtaining a first discrimination result for the nighttime fog-free image and a second discrimination result for the first nighttime dehazed image. Based on the first and second discrimination results, an adversarial loss is determined. Based on the differences between the first nighttime depth image and the first daytime depth image, the differences between the first nighttime foggy image and the second nighttime foggy image, the differences between the second nighttime depth image and the second daytime depth image, the differences between the second glow image and the third glow image, the differences between the second scattering coefficient image and the third scattering coefficient image, the differences between the nighttime fog-free image and the second nighttime dehazed image, and the adversarial loss, the initial nighttime dehazed model and the nighttime image depth estimation network are trained.

[0175] Optionally, the training module is further configured to: determine a first depth loss based on the difference between the first nighttime depth image and the first daytime depth image; determine a first cycle consistency loss based on the difference between the first nighttime foggy image and the second nighttime foggy image; determine a second depth loss based on the difference between the second nighttime depth image and the second daytime depth image; determine a second cycle consistency loss based on the difference between the nighttime fog-free image and the second nighttime dehazed image; determine a first contrast loss based on the difference between the second glow image and the third glow image; determine a second contrast loss based on the difference between the second scattering coefficient image and the third scattering coefficient image; calculate the sum of the first depth loss and the second depth loss as the first sum loss, and calculate the sum of the first cycle consistency loss and the second cycle consistency loss as the second sum loss; and train the initial nighttime dehazing model and the nighttime image depth estimation network based on the weighted sum of the adversarial loss, the first sum loss, the second sum loss, the first contrast loss, and the second contrast loss.

[0176] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0177] Figure 13 A structural block diagram of an electronic device for performing an image dehazing method according to an embodiment of this application is shown. The electronic device may be... Figure 1 The terminal (110) or server (120), etc., should be noted. Figure 13 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0178] like Figure 13 As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. Various programs and data required for system operation are also stored in RAM 1203. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. An Input / Output (I / O) interface 1205 is also connected to bus 1204.

[0179] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0180] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0181] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0182] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0184] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0185] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the electronic device to perform the methods of any of the above embodiments.

[0186] In the embodiments of this application, the terms "module" or "unit" refer to a part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0187] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0188] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0189] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image dehazing method, characterized in that, The method includes: Acquire the nighttime image of the target to be dehazed; The target nighttime image is dehazed using a nighttime dehazing model to obtain a dehazed, fog-free nighttime image of the target; The training process of the nighttime defogging model includes: The foggy image of the first night is dehazed by using the initial nighttime dehazing model to obtain the dehazed image of the first nighttime. The first nighttime dehazed image is depth-estimated using a nighttime image depth estimation network to obtain a first nighttime depth image; Based on the difference between the first nighttime depth image and the first daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network are trained, and the trained initial nighttime dehazing model is obtained as the nighttime dehazing model; wherein, the first daytime depth image is the depth image corresponding to the first daytime fog-free image that has the same image content as the first nighttime foggy image.

2. The method according to claim 1, characterized in that, The step of training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and the first daytime depth image includes: By using an atmospheric degradation model, the first nighttime defogging image is processed to add fog, resulting in a second nighttime foggy image; Based on the differences between the first nighttime foggy image and the second nighttime foggy image, and the differences between the first nighttime depth image and the first daytime depth image, the initial nighttime defogging model and the nighttime image depth estimation network are trained.

3. The method according to claim 2, characterized in that, After the initial nighttime defogging model performs defogging processing on the first nighttime foggy image, it also outputs a first scattering coefficient image and a first glow image. The process of applying fogging to the first nighttime defogging image using the atmospheric degradation model to obtain a second nighttime foggy image includes: Using the atmospheric degradation model, based on the first nighttime depth image, the first scattering coefficient image, and the first glow image, the first nighttime defogging image is fogged to obtain the second nighttime foggy image.

4. The method according to claim 1, characterized in that, The process of dehazing the first nighttime foggy image using an initial nighttime dehazing model to obtain a first nighttime dehazed image includes: The first nighttime foggy image is preprocessed using the initial nighttime defogging model to obtain the first transmission image and the first glow image. Using the initial nighttime defogging model, based on the first transmitted image and the first glow image, the first nighttime foggy image is defogging and reprocessed to obtain the first nighttime defogging image.

5. The method according to claim 1, characterized in that, The step of training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and the first daytime depth image includes: The nighttime image depth estimation network is used to estimate the depth of the fog-free nighttime image to obtain a second nighttime depth image; Based on the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network are trained; wherein, the second daytime depth image is the depth image corresponding to the second daytime fog-free image whose image content is consistent with the nighttime fog-free image.

6. The method according to claim 5, characterized in that, The step of training the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, includes: Using an atmospheric degradation model, based on randomly sampled second glow images, second scattering coefficient images, and second nighttime depth images, fogging is applied to the fog-free nighttime images to obtain a third foggy nighttime image. The third nighttime foggy image is preprocessed using the initial nighttime defogging model to obtain the third glow image; The initial nighttime dehazing model and the nighttime image depth estimation network are trained based on the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image.

7. The method according to claim 6, characterized in that, After performing dehazing preprocessing on the third night foggy image, the initial night dehazing model also outputs a third scattering coefficient image; The training of the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image includes: The initial nighttime dehazing model and the nighttime image depth estimation network are trained based on the differences between the second and third scattering coefficient images, the second and third glow images, the first nighttime depth image and the first daytime depth image, and the second nighttime depth image and the second daytime depth image.

8. The method according to claim 6, characterized in that, After performing dehazing preprocessing on the third night foggy image, the initial night dehazing model also outputs a second transmitted image; The training of the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image includes: Using the initial nighttime defogging model, based on the third glow image and the second transmission image, the third nighttime foggy image is defogging and reprocessed to obtain the second nighttime defogging image; Based on the differences between the nighttime fog-free image and the second nighttime defogging image, the differences between the second glow image and the third glow image, the differences between the first nighttime depth image and the first daytime depth image, and the differences between the second nighttime depth image and the second daytime depth image, the initial nighttime defogging model and the nighttime image depth estimation network are trained.

9. The method according to claim 1, characterized in that, The step of training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and the first daytime depth image includes: The discriminator distinguishes the nighttime fog-free image and the first nighttime defogging image respectively, and obtains a first discrimination result corresponding to the nighttime fog-free image and a second discrimination result corresponding to the first nighttime defogging image; Based on the first and second discrimination results, the adversarial loss is determined; Based on the difference between the first nighttime depth image and the first daytime depth image, and the adversarial loss, the initial nighttime dehazing model and the nighttime image depth estimation network are trained.

10. The method according to claim 1, characterized in that, The step of training the initial nighttime dehazing model and the nighttime image depth estimation network based on the difference between the first nighttime depth image and the first daytime depth image includes: By using an atmospheric degradation model, the first nighttime defogging image is processed to add fog, resulting in a second nighttime foggy image; The nighttime image depth estimation network is used to estimate the depth of the fog-free nighttime image to obtain a second nighttime depth image; Using the atmospheric degradation model, based on the randomly sampled second glow image, second scattering coefficient image, and second nighttime depth image, fogging is applied to the fog-free nighttime image to obtain a third foggy nighttime image. The third nighttime foggy image is preprocessed using the initial nighttime defogging model to obtain a third glow image, a third scattering coefficient image, and a second transmission image. Using the initial nighttime defogging model, based on the third glow image and the second transmission image, the third nighttime foggy image is defogging and reprocessed to obtain the second nighttime defogging image; The discriminator distinguishes the nighttime fog-free image and the first nighttime defogging image respectively, and obtains a first discrimination result corresponding to the nighttime fog-free image and a second discrimination result corresponding to the first nighttime defogging image; Based on the first and second discrimination results, the adversarial loss is determined; Based on the differences between the first nighttime depth image and the first daytime depth image, the differences between the first nighttime foggy image and the second nighttime foggy image, the differences between the second nighttime depth image and the second daytime depth image, the differences between the second glow image and the third glow image, the differences between the second scattering coefficient image and the third scattering coefficient image, the differences between the nighttime fog-free image and the second nighttime dehazed image, and the adversarial loss, the initial nighttime dehazing model and the nighttime image depth estimation network are trained.

11. The method according to claim 10, characterized in that, The training of the initial nighttime dehazing model and the nighttime image depth estimation network based on the differences between the first nighttime depth image and the first daytime depth image, the differences between the first nighttime foggy image and the second nighttime foggy image, the differences between the second nighttime depth image and the second daytime depth image, the differences between the second glow image and the third glow image, the differences between the second scattering coefficient image and the third scattering coefficient image, the differences between the nighttime fog-free image and the second nighttime dehazing image, and the adversarial loss, includes: The first depth loss is determined based on the difference between the first nighttime depth image and the first daytime depth image; The first cycle consistency loss is determined based on the difference between the first night foggy image and the second night foggy image; The second depth loss is determined based on the difference between the second nighttime depth image and the second daytime depth image; The second cycle consistency loss is determined based on the difference between the nighttime fog-free image and the second nighttime defogging image; Based on the difference between the second glow image and the third glow image, a first contrast loss is determined; The second contrast loss is determined based on the difference between the second scattering coefficient image and the third scattering coefficient image; Calculate the sum of the first depth loss and the second depth loss as the first sum loss, and calculate the sum of the first cycle consistency loss and the second cycle consistency loss as the second sum loss; The initial nighttime dehazing model and the nighttime image depth estimation network are trained based on the weighted sum of the adversarial loss, the first sum loss, the second sum loss, the first contrast loss, and the second contrast loss.

12. An image dehazing device, characterized in that, The device includes: The acquisition module is used to acquire the target nighttime image to be dehazed; The dehazing module is used to dehaze the target night image using a nighttime dehazing model to obtain a dehazed target nighttime image without fog. The training process of the nighttime defogging model includes: The foggy image of the first night is dehazed by using the initial nighttime dehazing model to obtain the dehazed image of the first nighttime. The first nighttime dehazed image is depth-estimated using a nighttime image depth estimation network to obtain a first nighttime depth image; Based on the difference between the first nighttime depth image and the first daytime depth image, the initial nighttime dehazing model and the nighttime image depth estimation network are trained, and the trained initial nighttime dehazing model is obtained as the nighttime dehazing model; wherein, the first daytime depth image is the depth image corresponding to the first daytime fog-free image that has the same image content as the first nighttime foggy image.

13. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-11.

15. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method of any one of claims 1-11.

Images