Method for training an image enhancement network, and method for enhancing images

Abstract

A method for training an image enhancement network includes determining an optical flow vector 502 based on a first image dataset, generating 504 a second dataset by reducing quality of the first image dataset, and training 506 the image enhancement network in a first training stage based on the first and second datasets. A third image dataset is created 508 based on the second dataset and the optical flow vector. A fourth image dataset based on the first dataset and the optical flow vector is generated 508. A warped first output dataset is generated 512 based on the first output dataset and the optical flow vector. The method further includes 514 training the image enhancement network in a second training stage based on the third image dataset, the warped first output dataset, and the fourth image dataset. Noise may be added to the first image dataset.

Description

[0001] Various embodiments relate to methods for training a neural network for enhancing images, as well as methods for enhancing images. More particularly, but not exclusively, the trained neural network may be used to enhance low-light traffic images for advanced driver assistance applications or autonomous driving.

[0002] Advanced Driver Assistance Systems (ADAS) often rely on cameras to detect and recognize road traffic participants such as pedestrians, obstacles, traffic signs etc. There will be information loss and unexpected noise if the camera images are acquired in a dimly lit setting. The performance of vision-based algorithms will be hampered by the use of such poor-quality images. As such, the ability to enhance low light images is advantageous for vision-based applications. Traditional computer vision algorithms, such as Histogram equalization, Contrast Limited Adaptive Histogram Equalization (CLAHE) etc., apply the image enhancement globally. Such an approach leads to undesirable local illumination and amplifies noise. On the other hand, image enhancement methods based on dehazing apply noise suppression as a post processing step, which can lead to blurring of details. There are deep learning networks that may perform image enhancements, for example, the Multi Branch Low Light Image Enhancement Network (MBLLEN) and zeros reference-based deep learning networks like the ZeroDCE. Although these networks perform better than the older techniques, their improved performance is mostly limited to static single images, as their performance degrades when there is dynamic movement of objects in the scene that produce artefacts and flickering. In other words, they fail to produce quality results in low light videos. SUMMARY

[0003] According to various embodiments, there is provided a computer-implemented method for training an image enhancement network. The method includes determining an optical flow vector based on a first image dataset, generating a second image dataset by reducing image quality of the first image dataset, and training the image enhancement network in a first training stage based on the first image dataset and the second image dataset. The image enhancement network generates a first output dataset in the first training stage. The method further includes generating a third image dataset based on the second image dataset and the optical flow vector, generating a fourth image dataset based on the first image dataset and the optical flow vector. The method further includes generating a warped first output dataset based on the first output dataset and the optical flow vector. The method further includes training the image enhancement network in a second training stage based on the third image dataset, the fourth image dataset, and the warped first output dataset.

[0004] According to various embodiments, there is provided a computer-implemented method for enhancing images. The method includes providing an input image to an image enhancement network trained according to the above-described method, to result in an enhanced output image.

[0005] According to various embodiments, there is provided a computer program. When the computer program is executed by a computer, it causes the computer to carry out the abovedescribed method for enhancing images.

[0006] According to various embodiments, there is provided a data carrier signal. The data carrier signal carries the above-described computer program.

[0007] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0009] FIGS. 1A to IB illustrate the process of generating a training dataset used in a first training stage according to various embodiments.

[0010] FIG. 2 shows a first training stage for training an image enhancement network according to various embodiments.

[0011] FIGS. 3 A to 3C illustrate the process of generating a training dataset according to various embodiments.

[0012] FIG. 4 shows a second training stage for training the image enhancement network according to various embodiments.

[0013] FIG. 5 shows a flow diagram of a method for training an image enhancement network according to various embodiments.

[0014] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0015] It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0016] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0017] According to various embodiments, an image enhancement network is provided. The image enhancement network, also referred herein as “the neural network”, is configured to enhance low light videos, in other words, sequence of images captured under dim lighting conditions. The neural network may brighten the low light image frames in the video, while minimizing flickering between the image frames.

[0018] FIGS. 1A to IB illustrate the process of generating a training dataset according to various embodiments. The training dataset is used to train the image enhancement network in a first training stage.

[0019] FIG. 1A shows a process 110 for determining an optical flow vector 112. A first image dataset 102 may be fed into an optical flow network 104. The first image dataset 102 may include a well-illuminated video. In other words, the first image dataset 102 may include a sequence of image frames that were captured under good lighting conditions. The optical flow network 104 may be configured to estimate optical flow vector 112, denoted herein as f(t), of the first image dataset 102. The optical flow network 104 may determine the optical flow vector 112 for every two sequential image frames. The optical flow vector 112 may indicate motion between image frames of the video. The optical flow network 104 may determine optical flow vectors 112 for at least one of, horizontal and vertical movements, between the image frames.

[0020] The first image dataset 102 may be obtained from a publicly available video database. An example of a first image dataset may be the Scene Flow Dataset which may be accessed at https: / / lmb.informatik.uni-freiburg.de / resources / datasets / SceneFlowDatasets.en.html.

[0021] An example of a suitable neural network for the optical flow network 104 is the Recurrent All Pairs Field Transforms (RAFT) network disclosed “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow” by Teed et. al., which can be found at https: / / link.springer.com / chapter / 10.1007 / 978-3-030-58536-5_24.

[0022] FIG. IB shows a process 120 for lowering the image quality of the first image dataset 102. The process 102 may include darkening 124 the first image dataset 102, and then adding noise 126 to the darkened first image dataset 102. The process of darkening 124 may include applying gamma correction and linear scaling to the first image dataset 102 using Equation (1): x = / 3 x (a x y)Y (1) where y denotes gamma correction, while a and / ? denote linear scaling factors. In an example, y may be sampled in a uniform distribution 11(2,3.5) while a and (3 may be sampled from 11(0.9,1) and 11(0.5,1) respectively. The process of adding noise 126 may include simulating noise using Gaussian and Poisson noise. The process 120 may further include applying random cropping, horizontal flipping of ± 1 to 50 pixels, and rotation of ± 1 to 270°, to the first image dataset 102. The resulting dataset, which includes low light video, is referred herein as the second image dataset 122.

[0023] FIG. 2 shows a first training stage 200 for training an image enhancement network 214 according to various embodiments. The first training stage 200 may train the image enhancement network 214 to recover normal light images with the help of corresponding well-illuminated ground truth images. In the training process, the poorly-lit images, i.e. the second image dataset 122 may be provided to the image enhancement network 214 as an input. The image enhancement network 214 may output a first output dataset 202 based on the second image dataset 122. The image enhancement network 214 may determine a loss, LI, in a loss computation step 216. LI may be determined based on comparing the first output dataset 202 against the well-illuminated ground truth images that correspond to the second image dataset 122, i.e. the first image dataset 102. LI may be determined based on a difference between the first output dataset 202 and the first image dataset 102.

[0024] FIGS. 3A to 3C illustrate the process of generating training datasets according to various embodiments. The training datasets are used to train the image enhancement network 214 in a second training stage of training.

[0025] FIG. 3A shows a process 130 for distorting the second image dataset 122, according to various embodiments. The process 130 may including warping 134 the second image dataset 122 based on the optical flow vector 112 determined from the process 110. The objective of the process 130 is to obtain a third image dataset 132 that is a low-light, noisy and warped version of the first image dataset 102, i.e. the ground truth. The warping 134 of the second image dataset 122 using the optical flow vector 112 replicates the effect of vehicle motion that causes flickering between video frames.

[0026] FIG. 3B shows a process 140 for distorting the first image dataset 102, according to various embodiments. The process 140 may including warping 134 the first image dataset 102 based on the optical flow vector 112. The objective of the process 140 is to obtain a fourth image dataset 142 that is a warped version of the first image dataset 102.

[0027] FIG. 3C shows a process 150 for distorting the first output dataset 202, according to various embodiments. The process 150 may include warping 134 the first output dataset 202 based on the optical flow vector 112. The objective of the process 150 is to obtain a warped first output dataset 302 that is a warped version of the brightened images produced by the image enhancement network 214.

[0028] FIG. 4 shows a second training stage 400 for training the image enhancement network 214 according to various embodiments. The second training stage 400 may include providing the third image dataset 132 to the image enhancement network 214. The image enhancement network 214 may generate a second output dataset 402 based on the third image dataset 132. The image enhancement network 214 may determine a loss L2 based on the warped first output dataset 302 and the fourth image dataset 142, in a loss computation step 416. The loss L2 may be determined based on comparing the second output dataset 402 to the warped first output dataset 302. This comparison may indicate the ability of the image enhancement network 214 to be stable against flickering in the input images. The loss L2 may be determined further based on comparing the second output dataset 402 to the fourth image dataset 142. The loss L2 may include a sum of (a) the difference between the second output dataset 402 and the warped first output dataset 302, and (b) the difference between the second output dataset 402 and the fourth image dataset 142.This comparison may indicate the accuracy of the image enhancement network 214, in compensating for flickering effect in the input images.

[0029] In each epoch of training the image enhancement network 214, the first training stage 200 and the second training stage 400 may be carried out, and the losses LI and L2 may be computed. The losses LI and L2 are then backpropagated so that the weights of the image enhancement network 214 may be updated based on LI and L2. The weights of the image enhancement network 214 may be updated so as to minimize each of LI and L2.

[0030] In other words, the first training stage 200 and the second training stage 400 may each include computing their respective loss functions LI and L2, within a single epoch. A single backpropagation may be carried out to minimize both LI and L2 in each epoch. A plurality of epochs, for example, at least 20 epochs, for example, 50 epochs, may be carried out to train the image enhancement network 214.

[0031] The first training stage 200 may train the image enhancement network 214 to recover normal light images with the help of corresponding well-illuminated ground truth images, i.e. the first image dataset. The second training stage 400 may train the image enhancement network 214 to recover normal light, flickering images with the help of corresponding well-illuminated, flickering images (i.e., the fourth image dataset) being used as the ground truth.

[0032] According to various embodiments, the image enhancement network 214 may include a Convolutional Neural Network (CNN). An example of a suitable neural network for the image enhancement network 214 is the UNET 3+ network, which is disclosed in “UNET 3+: A Full-scale Connected UNET for Medical Image Segmentation” by Huang et.al.

[0033] As an example of how the image enhancement network 214 may be trained, in each of the first and second training stages, the respective training dataset may be provided to the image enhancement network 214 and the following standard steps for training a general neural network may be followed.

[0034] Before training the image enhancement network 214, the weights of the image enhancement network 214 may be randomly initialized to numbers between 0.01 and 0.1, while the biases may be randomly initialized to numbers between 0.1 and 0.9.

[0035] Subsequently, observations of the training dataset may be loaded into the input layer of the image enhancement network 214 and the output value(s) is generated by forward-propagation of the input values of the input layers. Afterwards a loss function is used to calculate loss with the output value(s) for each of the first training stage 200 and the second training stage 400. The loss function, i.e. LI and L2 may include mean square error.

[0036] The weights and biases are subsequently updated by an AdamOptimizer with a learning rate of 1 * 1 O'4. The other parameters of the AdamOptimizer are set to default values, for example:

[0037] betal = 0.9

[0038] beta 2 = 0.999

[0039] eps = le-08

[0040] weight decay = 0

[0041] The steps described above should be repeated with the next set of observations until all the observations are used for training. This represents the first training epoch. This was repeated until 50 epochs were done.

[0042] According to various embodiments, each epoch may include both the first training stage 200 and the second training stage 400. The first training stage 200 may be carried out before the second training stage 400, within the epoch. The respective losses LI and L2 may be backpropagated concurrently in the same epoch.

[0043] FIG. 5 shows a flow diagram of a computer-implemented method 500 for training an image enhancement network 214 according to various embodiments. The method 500 may include determining an optical flow vector 112 based on a first image dataset 102, in step 502. The step 502 may include the process 110. The method 500 may further include generating a second image dataset 122 by reducing image quality of the first image dataset 102, in step 604. The step 504 may include the process 120. The method 500 may further include training the image enhancement network 214 in a first training stage 200 based on the first image dataset 102 and the second image dataset 122, in step 506. The image enhancement network 214 may generate a first output dataset 202 in the first training stage 200. The method 500 may further include generating a third image dataset 132 based on the second image dataset 122 and the optical flow vector 112 in step 508. The step 508 may include the process 130. The method 500 may further include generating a fourth image dataset 142 based on the first image 102 dataset and the optical flow vector 112, in step 510. The step 510 may include the process 140. The method may further include generating a warped first output dataset 302 based on the first output dataset 202 and the optical flow vector 112, in step 512. The step 512 may include the process 150. The method 500 may further include training the image enhancement network 214 in a second training stage 400 based on the third image dataset 132, the fourth image dataset 142, and the warped first output dataset 302, in step 514. The method 500 may result in an image enhancement network 214 that is capable of stabilizing a video sequence that has blurriness due to motion effects, as well as improving the brightness of the video sequence while suppressing noise.

[0044] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, generating the second image dataset 122 may include darkening the first image dataset 102. This may simulate an image captured under poor lighting conditions or at night.

[0045] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, generating the second image dataset 122 may further include adding noise to the first image dataset 102. This may simulate the noise effect in images captured under low light conditions.

[0046] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, determining the optical flow vector 112 may include inputting the first image dataset 102 to an optical flow neural network configured to determine the optical flow vector 112. The optical flow vector 112 may be used to represent motion between video frames. Using the optical flow neural network may result in high accuracy and efficiency in determining the optical flow vector.

[0047] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, generating the third image dataset 132 may include warping the second image dataset 122 using the optical flow vector 112, such that the third image dataset 132 exhibit rapid changes of brightness and contrast between video frames.

[0048] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, generating the fourth image dataset 142 may include warping the first image dataset 102 using the optical flow vector 112, such that the fourth image dataset 142 exhibit rapid changes of brightness and contrast between video frames.

[0049] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, training the image enhancement network 214 in the first training stage 200 may include inputting the second image dataset 122 to the image enhancement network 214 to thereby generate the first output dataset 202.

[0050] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, training the image enhancement network 214 in the first training stage 200 may further include computing a first loss parameter based on the first output dataset 202 and the first image dataset 102.

[0051] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, training the image enhancement network 214 in the second training stage 400 may include inputting the third image dataset 132 to the image enhancement network 214 to thereby generate the second output dataset 402.

[0052] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, training the image enhancement network 214 in the second training stage 400 may further include computing a second loss parameter based on the warped first output dataset 302, the fourth image dataset 142 and the second output dataset 402.

[0053] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, computing the second loss parameter includes determining a difference between the second output dataset 402 and the fourth image dataset 142. Computing the second loss parameter may further include determining a difference between the second output dataset 402 and the warped first output dataset 302.

[0054] According to an embodiment which may be combined with any of the above-described embodiment or with any below described further embodiment, computing the second loss parameter includes adding the difference between the second output dataset 402 and the fourth image dataset 142, to the difference between the second output dataset 402 and the warped first output dataset 302.

[0055] According to various embodiments, a computer-implemented method for enhancing images may include providing an input image to an image enhancement network 214 trained according to the method 500, to result in an enhanced output image. The input image may include a video sequence captured at least partially under dim lighting conditions. The video sequence may also be captured by a moving vehicle, and hence exhibit blurriness or flickering due to the motion. The method for enhancing images may generate an enhance output image that is stabilized and well-illuminated. The enhanced output image may be used for computer vision applications such as object detection, classification, segmentation etc., and result in higher accuracies due to the improved image quality.

[0056] According to various embodiments, a computer program may include instructions, which, when the computer program is executed by a computer, causes the computer to carry out the abovedescribed method of enhancing images. The computer program may be carried by a data carrier signal.

[0057] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0058] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0059] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0060] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A computer-implemented method (500) for training an image enhancement network(214), the method (500) comprising:determining an optical flow vector (112) based on a first image dataset (102);generating a second image dataset (122) by reducing image quality of the first image dataset (102);training the image enhancement network (214) in a first training stage (200) based on the first image dataset (102) and the second image dataset (122), wherein the image enhancement network (214) generates a first output dataset (202) in the first training stage (200);generating a third image dataset (132) based on the second image dataset (122) and the optical flow vector (112);generating a fourth image dataset (142) based on the first image dataset (102) and the optical flow vector (112);generating a warped first output dataset (302) based on the first output dataset (202) and the optical flow vector (112); andtraining the image enhancement network (214) in a second training stage (400) based on the third image dataset (132), the fourth image dataset (142) and the warped first output dataset (302), wherein the image enhancement network (214) generates a second output dataset (402) in the second training stage (400).

2. The method (500) of claim 1, wherein generating the second image dataset (122) comprises darkening the first image dataset (102).

3. The method (500) of claim 2, wherein generating the second image dataset (122) further comprises adding noise to the first image dataset (102).

4. The method (500) of any preceding claim, wherein determining the optical flow vector(112) comprises inputting the first image dataset (102) to an optical flow neural network configured to determine the optical flow vector (112).

5. The method (500) of any preceding claim, wherein generating the third image dataset (132) comprises warping the second image dataset (122) using the optical flow vector (112), such that the third image dataset (132) exhibit rapid changes of brightness and contrast between video frames.

6. The method (500) of any preceding claim, wherein generating the fourth image dataset (142) comprises warping the first image dataset (102) using the optical flow vector (112), such that the fourth image dataset (142) exhibit rapid changes of brightness and contrast between video frames.

7. The method (500) of any preceding claim, wherein training the image enhancementnetwork (214) in the first training stage (200) comprises inputting the second image dataset (122) to the image enhancement network (214) to thereby generate the first output dataset (202).

8. The method (500) of claim 7, wherein training the image enhancement network (214) in the first training stage (200) further comprises computing a first loss parameter based on the first output dataset (202) and the first image dataset (102).

9. The method (500) of any preceding claim, wherein training the image enhancement network (214) in the second training stage (400) comprises inputting the third image dataset (132) to the image enhancement network (214) to thereby generate the second output dataset (402).

10. The method (500) of claim 9, wherein training the image enhancement network (214) in the second training stage (400) further comprises computing a second loss parameter based on the warped first output dataset (302), the fourth image dataset (142) and the second output dataset.

11. The method (500) of claim 10, wherein computing the second loss parameter comprises determining a difference between the second output dataset (402) and the fourth image dataset (142), and further comprises determining a difference between the second output dataset (402) and the warped first output dataset (302).

12. The method (500) of claim 12, wherein computing the second loss parameter further comprises adding the difference between the second output dataset (402) and the fourth image dataset (142), to the difference between the second output dataset (402) and the warped first output dataset (302).

13. A computer-implemented method for enhancing images, the method comprising: providing an input image to an image enhancement network (214) trained according to the method (500) of any preceding claim, to result in an enhanced output image.

14. A computer program comprising instructions, which, when the computer program is executed by a computer, causes the computer to carry out the method of claim 13.

15. A data carrier signal carrying the computer program of claim 14.

Images