Training data collection method and system of neural network model for recovering low-illumination image

Abstract

The invention relates to the technical field of image processing, and provides a training data collection method and system of a neural network model for recovering a low-illumination image. The method comprises the following steps: randomly selecting a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illumination images; the second image set is a set of normal illumination images; constructing a mapping relation from the color histogram of the second image to the color histogram of the first image; adjusting the intensity of each pixel in the second image based on the mapping relation to obtain a third image; wherein the third image is an image which corresponds to the second image and has the illumination characteristic of the first image. Therefore, under the condition that various low-illumination environments do not need to be reproduced on site, a large number of diversified training data sets are generated, and remarkable physical obstacles and technical obstacles needed by directly collecting paired normal-illumination images and low-illumination images are avoided.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for collecting training data for a neural network model for restoring low-light images. Background Art

[0002] The training of the neural network model requires the collection of Dion's training data. The type of training data provided determines the type of neural network that can be trained. When training a convolutional neural network to restore low-light images, it is often difficult to collect training data.

[0003] The training data for the neural network model for restoring low-light images is usually paired images, and the paired images used for training (images of the same scene with different illuminations) must have the same content, and there must not be any displacement difference between the paired images. In addition, during the process of taking paired images, the ambient illumination needs to be constantly changed, so it is very difficult to collect training data for the neural network model for restoring low-light images.

[0004] At present, the method to solve the above problem is mainly to use gamma transform on the normal illumination picture to obtain low illumination pictures, and then obtain a pair of training pictures that can be used for neural network model training. This processing method makes the illumination feature of the low illumination picture single. Even if the random value method is used to randomly select the illumination adjustment parameters in the gamma transform, the generated low illumination image is still very single in illumination feature.

[0005] Therefore, there is an urgent need to provide a technical solution to the above-mentioned deficiencies in the prior art. Summary of the invention

[0006] The purpose of the present application is to provide a method and system for collecting training data for a neural network model for restoring low-light images, so as to solve or alleviate the problems existing in the above-mentioned prior art.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] The present application provides a training data collection method for a neural network model for restoring low-illuminance images, including: step S101, randomly selecting a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illuminance images; and the second image set is a set of normal-illuminance images; step S102, constructing a mapping relationship from a color histogram of the second image to a color histogram of the first image; step S103, adjusting the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0009] Preferably, in step S102, according to the formula:

[0010]

[0011] Constructing a mapping relationship Q(m)=1 from the color histogram of the second image to the color histogram of the first image; wherein, is the sum of the probabilities that the intensity of all pixels in each single color channel of the first image is less than or equal to 1; is the sum of the probabilities that the intensities of all pixels in the color channel of the second image corresponding to the first image are less than or equal to m; m, l=[0, 255], where m and l are both positive integers.

[0012] Preferably, step S102 includes: respectively calculating the color histogram of each color channel in the first image and the second image; respectively calculating the probability density of pixel intensity in each color channel in the first image and the second image based on the color histogram of each color channel in the first image and the second image; and respectively calculating the sum of the probabilities that the pixel intensity in each color channel of the first image and the second image is less than or equal to a preset pixel intensity based on the probability density of the pixel intensity in each color channel in the first image and the second image.

[0013] Preferably, according to the formula:

[0014]

[0015] Calculate the color histogram of the target image; wherein the target image includes: the first image and the second image; H(i) is the number of pixels with pixel intensity i in the color histogram of a single color channel of the target image; W and H are the width and height of the target image respectively; X M(x，y) represents the pixel value of the target image in the color channel M at the coordinate (x, y); M = {R, G, B}, R, G, B are red, green, and blue respectively; i = [0, 255], i is a positive integer; is the indicator function, when X M(x，y) =i, Otherwise 0.

[0016] Preferably, according to the formula:

[0017]

[0018] Calculate the probability density of pixel intensity in each single color channel of the target image; wherein P(i) is the probability density of pixel intensity i in each single color channel of the target image; H(i) is the number of pixels with pixel intensity i in the color histogram of the single color channel of the target image; N is the total number of pixels in the target image; i=[0, 255], i is a positive integer.

[0019] Preferably, according to the formula:

[0020]

[0021] Calculate the sum of the probabilities that the intensity of all pixels in each single color channel of the target image is less than or equal to the preset pixel intensity; wherein, is the sum of the probabilities that the intensities of all pixels in the color channel M of the target image are less than or equal to i; P(i) is the probability density of the pixel intensity i in each single color channel of the target image; i, j are both pixel intensities in the target image, j = [0, i]; i = [0, 255], i, j are both positive integers; M = {R, G, B}, R, G, B are red, green, blue, respectively.

[0022] Preferably, in step S103, based on the mapping relationship Q(m)=l, each pixel intensity m in the second image is adjusted to the pixel intensity l in the first image to obtain the third image.

[0023] An embodiment of the present application also provides a training data collection system for a neural network model for restoring low-illuminance images, comprising: an image acquisition unit, configured to randomly select a first image and a second image from a first image set and a second image set, respectively; wherein the first image set is a set of low-illuminance images; and the second image set is a set of normal-illuminance images; a mapping construction unit, configured to construct a mapping relationship from a color histogram of the second image to a color histogram of the first image; an illumination matching unit, configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0024] Beneficial effects:

[0025] The training data collection method for a neural network model for restoring low-illuminance images provided in an embodiment of the present application comprises the following steps: first, randomly selecting a low-illuminance image (first image) and a normal-illuminance image (second image) from a set of low-illuminance images (first image set) and a set of normal-illuminance images (second image set), respectively; then, constructing a mapping relationship from a color histogram of the normal-illuminance image (second image) to a color histogram of the first image (low-illuminance image); finally, adjusting the intensity of each pixel in the normal-illuminance image (second image) based on the constructed mapping relationship, and obtaining a third image corresponding to the normal-illuminance image (second image) and having the illumination characteristics of the low-illuminance image (first image). Thereby, the third image and the second image form a set of training data for a neural network model for restoring low-light images. Furthermore, through a large number of low-light images, images with low-light features corresponding to normal-light images are formed, which together with the images in the second image set constitute the training data for the neural network model for restoring low-light images, so that the formed pairs of training images only differ in illumination level without any displacement, angle change or scene difference, and have precise consistency. Moreover, the number of training data formed is large and there is no need to collect paired images through a camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0027] in:

[0028] Figure 1 A flowchart of a method for collecting training data for a neural network model for restoring low-light images provided according to some embodiments of the present application;

[0029] Figure 2 A schematic diagram of an image having illumination characteristics of a low illumination image generated from a normal illumination image according to some embodiments of the present application;

[0030] Figure 3 A schematic diagram of an image having illumination characteristics of a low illumination image generated from another normal illumination image provided according to some embodiments of the present application;

[0031] Figure 4 A schematic diagram of the structure of a training data collection system for a neural network model for restoring low-light images provided according to some embodiments of the present application. DETAILED DESCRIPTION

[0032] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments. Each example is provided by way of explanation of the present application and does not limit the present application. In fact, it will be clear to those skilled in the art that modifications and variations may be made in the present application without departing from the scope or spirit of the present application. For example, a feature shown or described as a part of an embodiment may be used in another embodiment to produce yet another embodiment. Therefore, it is desired that the present application includes such modifications and variations within the scope of the appended claims and their equivalents.

[0033] In the current machine learning field, there is a lot of training data, including images with normal illumination and low illumination. However, when performing low illumination image restoration, there are very few public paired data sets. The existing famous LOL data set still has the problem of single illumination feature. When training neural networks with training data with single illumination feature, the robustness (stability) of the neural network is very poor due to the very single illumination feature, which greatly reduces the generalization of the neural network and completely fails to reflect the generalization advantage of artificial intelligence over automation.

[0034] In particular, collecting pairs of normal illumination images and low illumination images is crucial for training neural network models that restore low illumination images. However, these pairs of images need to be captured in the same scene, under normal illumination and low illumination conditions, respectively, to ensure that the training network can learn the mapping from low illumination conditions to normal illumination conditions. Ideally, the paired images should only differ in illumination level, while all other conditions (such as camera parameters, scene composition, etc.) should remain unchanged to ensure that the neural network model can focus on learning the impact of illumination changes rather than other interference factors, thereby improving the accuracy of the neural network model. However, in actual operation, it is very difficult to achieve this high degree of control, especially in natural environments, because slight changes in lighting conditions may also cause differences between images, and it is required that there should be no displacement, angle change or scene difference between the two paired images. It is extremely difficult to ensure precise consistency between images in these two illumination states.

[0035] Based on this, this application proposes a method for collecting training data for a neural network model for restoring low-light images, which solves the problem of collecting high-quality paired data sets when restoring low-light images under a supervised learning framework. Figures 1 to 3 As shown, the training data collection method of the neural network model for restoring low-light images includes:

[0036] Step S101: randomly select a first image and a second image from a first image set and a second image set respectively.

[0037] In the present application, the first image set is a set of multiple low-light images, and the second image set is a set of multiple normal-light images. The low-light image set, i.e., the first image set, is:

[0038] A={a1,a2,…,a s , …a p}

[0039] The normal illumination image set, i.e., the second image set, is:

[0040] B={b1,b2,…,b d , …a u}

[0041] Among them, s∈Uniform(1,p), d∈Uniform(1,u); s, d, p, u are all positive integers.

[0042] Randomly select a low-light image X=a from the low-light image set A s , randomly select a normal illumination image Y=b from the normal illumination image set B d That is, X is a randomly selected image from the background illumination image set A, s is the index in the background illumination image set A; Y is a randomly selected image from the normal illumination image set B, d is the index in the normal illumination image set B.

[0043] Step S102: construct a mapping relationship between the color histogram of the second image and the color histogram of the first image.

[0044] In this application, the color histogram matching technology of normal illumination images and low illumination images is used to produce the paired images required for neural network model training. That is, through the existing large number of normal illumination images and low illumination image resources, the histogram matching technology is applied to adjust the histogram distribution of the normal illumination image, and the illumination characteristics of the low illumination image are used to generate visually consistent paired images under different illumination conditions.

[0045] First, the color histogram of each color channel in the low-light image (first image) and the normal-light image (second image) is calculated respectively. Specifically, according to the formula:

[0046]

[0047] Calculate the color histogram of the target image (first image and second image). Where H(i) is the number of pixels with pixel intensity i in the color histogram of a single color channel of the target image; W and H are the width and height of the target image respectively; X M(x，y)is the pixel value of the target image in the color channel M at the coordinate (x, y); M = {R, G, B}, R, G, B are red, green, and blue respectively; i = [0, 255], i is a positive integer. is the indicator function, when X M(x，y) =i, Otherwise 0.

[0048] Then, based on the color histogram of each color channel in the low-light image (first image) and the normal-light image (second image), the probability density of the pixel intensity in each color channel in the low-light image (first image) and the normal-light image (second image) is calculated respectively. Specifically, according to the formula:

[0049]

[0050] Calculate the probability density of pixel intensity in each single color channel of the target image (first image and second image). Where P(i) is the probability density of pixel intensity i in each single color channel of the target image; H(i) is the number of pixels with pixel intensity i in the color histogram of the single color channel of the target image; N is the total number of pixels in the target image, i = [0, 255], i is a positive integer.

[0051] Next, according to the probability density of the pixel intensity in each color channel in the low-light image (first image) and the normal-light image (second image), the sum of the probabilities that the pixel intensity in each color channel of the low-light image (first image) and the normal-light image (second image) is less than or equal to the set pixel intensity is calculated. Specifically, according to the formula:

[0052]

[0053] Calculate the sum of the probabilities that the intensity of all pixels in each single color channel of the target image (the first image and the second image) is less than or equal to the preset pixel intensity. is the sum of the probabilities that the intensities of all pixels in the color channel M of the target image are less than or equal to i; P(i) is the probability density of the pixel intensity i in each single color channel of the target image; i, j are both pixel intensities in the target image, j = [0, i]; i = [0, 255], i, j are both positive integers; M = {R, G, B}, R, G, B are red, green, and blue, respectively.

[0054] Finally, according to the formula:

[0055]

[0056] Constructing a mapping relationship Q(m)=1 from the color histogram of the second image to the color histogram of the first image;

[0057] in, is the sum of the probabilities that the intensity of all pixels in each single color channel of the first image is less than or equal to l; is the sum of the probabilities that the intensities of all pixels in the color channel corresponding to the first image in the second image are less than or equal to m; m, l = [0, 255], where m and l are both positive integers.

[0058] Step S103: adjusting the intensity of each pixel in the second image based on the mapping relationship to obtain a third image.

[0059] After completing the construction of the mapping relationship of the color histogram from the normal illumination image to the low illumination image, based on the mapping relationship Q(m)=l, each pixel intensity m in the second image is adjusted to the pixel intensity l in the first image to obtain a third image. The third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0060] Usually, low-light images tend to have a histogram distribution close to 0, so the calculated CDF function will be steeper near 0, while the normal-light image will be slower near 0. When the mapping relationship Q(m)=1 is applied, assuming that in a normal-light photo, you want to map a pixel with a pixel intensity of 50 (i.e., the pixel intensity m in the second image is 50), the calculated CDF may be 0.4, i.e. However, in low-light images Greater than or equal to 0.4, the pixel intensity set included is {5, 6, 7...255}, that is, the probability density of the pixel intensity in the color channel of the first image being greater than or equal to 5 l={5,6,7...255}. At this time, the minimum value 5 of this set is taken, that is, l=5, that is, the pixel intensity of 50 in the image with normal illumination is mapped to 5.

[0061] Therefore, by applying histogram matching, the histogram distribution of the normal illumination image is adjusted to match the characteristics of the low illumination image, so that visually consistent image pairs can be generated under different illumination conditions; through histogram matching, the normal illumination image is effectively converted to simulate the low illumination condition, thereby generating a large and diverse training data set without the need to reproduce various low illumination environments in the field, avoiding the significant physical and technical barriers required to directly collect pairs of normal illumination images and low illumination images.

[0062] The training data set produced by the method of the present application not only achieves a significant increase in quantity, but also shows richness in scene diversity and illumination changes, providing a more comprehensive learning opportunity for the training of supervised neural network models for restoring low-light images, and significantly improving the generalization ability of the model when performing low-light image restoration.

[0063] The present application also provides a training data collection system for a neural network model for restoring low-light images. Figure 4 As shown, the collection system includes:

[0064] The image acquisition unit 401 is configured to randomly select a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illuminance images; and the second image set is a set of normal-illuminance images.

[0065] The mapping construction unit 402 is configured to construct a mapping relationship from the color histogram of the second image to the color histogram of the first image.

[0066] The illumination matching unit 403 is configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image, wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0067] The training data collection system for the neural network model for restoring low-light images provided in the embodiments of the present application can implement the steps and processes of the training data collection method for the neural network model for restoring low-light images in any of the above embodiments, and achieve the same technical effects, which will not be described one by one here.

[0068] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for collecting training data for a neural network model for restoring low-light images, characterized in that: include: Step S101, randomly selecting a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illuminance images; and the second image set is a set of normal-illuminance images; Step S102: constructing a mapping relationship between the color histogram of the second image and the color histogram of the first image; Step S103: adjusting the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

2. The method for collecting training data for a neural network model for restoring low-light images according to claim 1, characterized in that: In step S102, According to the formula: Constructing a mapping relationship Q(m)=1 from the color histogram of the second image to the color histogram of the first image; in, is the sum of the probabilities that the intensity of all pixels in each single color channel of the first image is less than or equal to 1; is the sum of the probabilities that the intensities of all pixels in the color channel of the second image corresponding to the first image are less than or equal to m; m, l=[0, 255], where m and l are both positive integers.

3. The method for collecting training data for a neural network model for restoring low-light images according to claim 2, characterized in that: Step S102 includes: Calculating the color histogram of each color channel in the first image and the second image respectively; Based on the color histograms of each color channel in the first image and the second image, respectively calculate the probability density of pixel intensity in each color channel in the first image and the second image; According to the probability density of the pixel intensity in each color channel in the first image and the second image, the sum of the probabilities that the pixel intensity in each color channel of the first image and the second image is less than or equal to the preset pixel intensity is calculated respectively.

4. The method for collecting training data for a neural network model for restoring low-light images according to claim 3, characterized in that: According to the formula: Calculate the color histogram of the target image; The target image includes: the first image and the second image; H(i) is the number of pixels with pixel intensity i in the color histogram of a single color channel of the target image; W and H are the width and height of the target image respectively; X M(x,y) represents the pixel value of the target image at the coordinate (x, y) in the color channel M; M = {R, G, B}, R, G, B are red, green, and blue respectively; i = [0, 255], i is a positive integer; is the indicator function, when X M(x,y) =i, Otherwise 0.

5. The method for collecting training data for a neural network model for restoring low-light images according to claim 4, characterized in that: According to the formula: Calculating the probability density of pixel intensity in each single color channel of the target image; Wherein, P(i) is the probability density of pixel intensity i in each single color channel in the target image; H(i) is the number of pixels with pixel intensity i in the color histogram of a single color channel of the target image; M is the total number of pixels in the target image; i=[0, 255], i is a positive integer.

6. The method for collecting training data for a neural network model for restoring low-light images according to claim 5, characterized in that: According to the formula: Calculating the sum of probabilities that the intensity of all pixels in each single color channel of the target image is less than or equal to a preset pixel intensity; in, is the sum of the probabilities that the intensities of all pixels in the color channel M of the target image are less than or equal to i; P(i) is the probability density of the pixel intensity i in each single color channel of the target image; i, j are both pixel intensities in the target image, j = [0, i]; i = [0, 255], i, j are both positive integers; M = {R, G, B}, R, G, B are red, green, blue, respectively.

7. The method for collecting training data for a neural network model for restoring low-light images according to claim 1, characterized in that: In step S103, Based on the mapping relationship Q(m)=1, each pixel intensity m in the second image is adjusted to the pixel intensity l in the first image to obtain the third image.

8. A training data collection system for a neural network model for restoring low-light images, characterized in that: include: An image acquisition unit, configured to randomly select a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illuminance images; and the second image set is a set of normal-illuminance images; a mapping construction unit, configured to construct a mapping relationship from the color histogram of the second image to the color histogram of the first image; An illumination matching unit is configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

Images