Training data collection method and system for a neural network model for restoring low-light images

Abstract

The application relates to the technical field of image processing, and provides a training data collection method and system of a neural network model for restoring a low-illumination image. The method comprises the following steps: randomly selecting a first image and a second image from a first image set and a second image set respectively; the first image set is a set of low-illumination images; the second image set is a set of normal-illumination images; a mapping relationship from a color histogram of the second image to a color histogram of the first image is constructed; the intensity of each pixel in the second image is adjusted based on the mapping relationship, and a third image is obtained; the third image is an image corresponding to the second image and having the illumination characteristics of the first image. Therefore, without the need of reproducing various low-illumination environments in the field, a large amount of diversified training data sets can be generated, and significant physical and technical obstacles that need to be faced when directly collecting pairs of normal-illumination images and low-illumination images are avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a training data collection method and system of a neural network model for recovering low-illumination images. BACKGROUND

[0002] Training of a neural network model requires collection of training data, and what kind of training data provided determines the function of the neural network that can be trained. When training a convolutional neural network for recovering low-illumination images, it is often difficult to collect training data.

[0003] For training of a neural network model for recovering low-illumination images, the training data are usually paired images, and the content of the paired images (images of the same scene under different illuminations) used for training is required to be consistent, and there should be no displacement difference between the paired images. In addition, during shooting of the paired images, the environmental illumination needs to be changed constantly, so it is very difficult to collect training data for training of a neural network model for recovering low-illumination images.

[0004] Currently, the main method to solve the above problems is to use gamma transformation on normal-illumination images to obtain low-illumination images, and then obtain a pair of training images that can be used for training of a neural network model. This processing method makes the illumination characteristics of the low-illumination images single, and even if a random selection method is used to randomly select the illumination adjustment parameter in the gamma transformation, the generated low-illumination images are still very single in illumination characteristics.

[0005] Therefore, there is an urgent need to provide a technical solution to the above technical problems of the prior art. SUMMARY

[0006] The present application aims to provide a training data collection method and system of a neural network model for recovering low-illumination images to solve or alleviate the above problems in the prior art.

[0007] In order to achieve the above-mentioned purpose, the present application provides the following technical solution:

[0008] The present application provides a training data collection method of a neural network model for recovering low-illumination images, comprising: step S101, randomly selecting a first image and a second image from a first image set and a second image set, respectively; wherein the first image set is a set of low-illumination images; the second image set is a set of normal-illumination images; step S102, constructing a mapping relationship from a color histogram of the second image to a color histogram of the first image; step S103, adjusting the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0009] Preferably, in step S102, the mapping relationship Q(m)=l of the color histogram of the second image to the color histogram of the first image is constructed according to the formula:

[0010]

[0011] the color histogram of the second image to the color histogram of the first image is constructed according to the formula: is the sum of probabilities of all pixel intensities less than or equal to l in each single color channel of the first image; is the sum of probabilities of all pixel intensities less than or equal to m in the color channel corresponding to the first image in the second image; m, l = [0, 255], m, l are positive integers.

[0012] Preferably, step S102 comprises: calculating the color histogram of each color channel in the first image and the second image respectively; calculating the probability density of pixel intensity in each color channel in the first image and the second image based on the color histogram of each color channel in the first image and the second image; and calculating the sum of probabilities of pixel intensity less than or equal to a preset pixel intensity in each color channel of the first image and the second image according to the probability density of pixel intensity in each color channel of the first image and the second image.

[0013] Preferably, the mapping relationship Q(m)=l of the color histogram of the second image to the color histogram of the first image is constructed according to the formula:

[0014]

[0015] calculating the color histogram of the target image; wherein the target image comprises the first image and the second image; H(i) is the number of pixels of pixel intensity i in the color histogram of a single color channel of the target image; W, H are the width and height of the target image respectively; X M(x，y) represents the pixel value of the target image at coordinates (x, y) in the color channel M; M = {R, G, B}, R, G, B are red, green and blue respectively; i = [0, 255], i is a positive integer; is an indicator function, when X M(x，y) = i, otherwise 0.

[0016] Preferably, the mapping relationship Q(m)=l of the color histogram of the second image to the color histogram of the first image is constructed according to the formula:

[0017]

[0018] calculating the probability density of the pixel intensity in each single color channel of the target image; wherein, P(i) is the probability density of the pixel intensity i in each single color channel of the target image; H(i) is the number of pixels of the pixel intensity i in the color histogram of the single color channel of the target image; N is the total number of pixels in the target image; i = [0, 255], i is a positive integer.

[0019] Preferably, according to the formula:

[0020]

[0021] calculating the sum of the probabilities of all pixel intensities less than or equal to a preset pixel intensity in each single color channel of the target image; wherein, is the sum of the probabilities of all pixel intensities less than or equal to i in the color channel M of the target image; P(i) is the probability density of the pixel intensity i in each single color channel of the target image; i, j are both pixel intensities in the target image, j = [0, i]; i = [0, 255], i, j are both positive integers; M = {R, G, B}, R, G, B are red, green and blue respectively.

[0022] Preferably, in step S103, based on the mapping relationship Q(m) = l, each pixel intensity m in the second image is adjusted to a pixel intensity l in the first image to obtain the third image.

[0023] The embodiment of the present application also provides a training data collection system of a neural network model for restoring a low-illumination image, comprising: an image acquisition unit configured to randomly select a first image and a second image from a first image set and a second image set respectively; wherein, the first image set is a set of low-illumination images; the second image set is a set of normal-illumination images; a mapping construction unit configured to construct a mapping relationship from a color histogram of the second image to a color histogram of the first image; an illumination matching unit configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image; wherein, the third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0024] Beneficial effects:

[0025] The training data collection method of the neural network model for recovering low-illumination images provided by the embodiments of the present application first randomly selects a low-illumination image (first image) and a normal-illumination image (second image) from a set of low-illumination images (first image set) and a set of normal-illumination images (second image set) respectively, then constructs a mapping relationship from the color histogram of the normal-illumination image (second image) to the color histogram of the first image (low-illumination image), and finally adjusts the intensity of each pixel in the normal-illumination image (second image) based on the constructed mapping relationship to obtain a third image corresponding to the normal-illumination image (second image) and having the illumination characteristics of the low-illumination image (first image). The third image and the second image form a set of training data of the neural network model for recovering low-illumination images, and then a large number of low-illumination images form images with low-illumination characteristics corresponding to normal-illumination images, which form training data of the neural network model for recovering low-illumination images together with the images in the second image set, so that the formed pairs of training images only differ in illumination level, without any displacement, angle change or scene difference, have precise consistency, and moreover, the number of training data formed is large and no longer needs to collect paired images through a camera. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The present application should not be limited by the accompanying drawings.

[0027] Wherein:

[0028] Figure 1 A flowchart of a training data collection method of a neural network model for recovering low-illumination images according to some embodiments of the present application is provided.

[0029] Figure 2 An image diagram of a normal-illumination image generating an image with the illumination characteristics of a low-illumination image according to some embodiments of the present application is provided.

[0030] Figure 3 Another image diagram of a normal-illumination image generating an image with the illumination characteristics of a low-illumination image according to some embodiments of the present application is provided.

[0031] Figure 4 A structural diagram of a training data collection system of a neural network model for recovering low-illumination images according to some embodiments of the present application is provided. DETAILED DESCRIPTION

[0032] The present application will be described in detail below with reference to the attached drawings and embodiments. Various examples are provided by way of explanation of the present application but not to limit the present application. It will be apparent to those skilled in the art that modifications and variations can be made in the present application without departing from the scope or spirit of the present application. For example, features shown or described as part of one embodiment can be used in another embodiment to create yet another embodiment. It is, therefore, desired that the present application contain all such modifications and variations as fall within the scope of the appended claims and their equivalents.

[0033] In the current machine learning field, there are a lot of training data, including normal illumination images and low illumination images. However, when performing low illumination image restoration, there are very few public paired data sets, and the existing famous LOL data set still has the problem of single illumination feature. The training data with single illumination feature has very poor robustness (stability) of the neural network, greatly reduces the generalization of the neural network, and completely fails to reflect the generalization advantage of artificial intelligence compared with automation.

[0034] In particular, collecting pairs of normal illumination images and low illumination images is crucial for training the neural network model for restoring low illumination images. However, these paired images need to be captured under the same scene, respectively under normal illumination and low illumination conditions, to ensure that the training network can learn the mapping from low illumination conditions to normal illumination conditions. Ideally, the paired images should only differ in illumination level, and all other conditions (such as camera parameters, scene composition, etc.) should remain unchanged to ensure that the neural network model can focus on learning the impact of illumination changes rather than other interference factors, improving the accuracy of the neural network model. However, in actual operation, it is very difficult to achieve such a high level of control, especially in a natural environment, because small changes in lighting conditions can also cause differences between images, and it is extremely difficult to ensure that the two images under these two illumination states have precise consistency without any displacement, angle change or scene difference between the two images.

[0035] Based on this, the present application proposes a training data collection method for a neural network model for restoring low illumination images, which solves the problem of collecting high-quality paired data sets when performing low illumination image restoration under a supervised learning framework. As shown in Figures 1 to 3 The training data collection method for the neural network model for restoring low illumination images includes:

[0036] Step S101, randomly selecting a first image and a second image from the first image set and the second image set, respectively.

[0037] In the present application, the first image set is a set containing multiple low-illumination images, and the second image set is a set containing multiple normal-illumination images. The low-illumination image set, i.e., the first image set, is:

[0038] A = {a1, a2,..., a s ,..., a p}

[0039] The normal-illumination image set, i.e., the second image set, is:

[0040] B = {b1, b2,..., b d ,..., a u}

[0041] where s ∈ Uniform(1, p), d ∈ Uniform(1, u); s, d, p, u are positive integers.

[0042] A low-illumination image X = a s is randomly selected from the low-illumination image set A, and a normal-illumination image Y = b d is randomly selected from the normal-illumination image set B. That is, X is a randomly selected image from the low-illumination image set A, and s is the index in the low-illumination image set A; Y is a randomly selected image from the normal-illumination image set B, and d is the index in the normal-illumination image set B.

[0043] Step S102, a mapping relationship of the color histogram of the second image to the color histogram of the first image is constructed.

[0044] In the present application, the color histogram matching technology of normal-illumination images and low-illumination images is used to produce paired images required for training the neural network model. That is, by using a large number of existing normal-illumination images and low-illumination image resources, the histogram matching technology is applied to adjust the histogram distribution of the normal-illumination images, and paired images visually consistent under different illumination conditions are generated according to the illumination characteristics of the low-illumination images.

[0045] First, the color histograms of each color channel in the low-illumination image (first image) and the normal-illumination image (second image) are calculated, respectively. Specifically, according to the formula:

[0046]

[0047] The color histogram of the target image (first image and second image) is calculated. Where H(i) is the number of pixels with intensity i in the color histogram of a single color channel of the target image; W and H are the width and height of the target image, respectively; X M(x，y)is the pixel value of the target image in the image in color channel M at coordinate (x, y); M = {R, G, B}, R, G, and B are red, green, and blue respectively; i = [0, 255], i is a positive integer. is an indicator function, when X M(x，y) = i, 0 otherwise.

[0048] Then, based on the color histogram of each color channel in the low-illumination image (the first image) and the normal-illumination image (the second image), the probability density of the pixel intensity in each color channel in the low-illumination image (the first image) and the normal-illumination image (the second image) is calculated respectively. Specifically, according to the formula:

[0049]

[0050] The probability density of the pixel intensity in each single color channel of the target image (the first image and the second image) is calculated. Wherein, P(i) is the probability density of the pixel intensity i in each single color channel of the target image; H(i) is the number of pixels with intensity i in the color histogram of the single color channel of the target image; N is the total number of pixels in the target image, i = [0, 255], i is a positive integer.

[0051] Next, based on the probability density of the pixel intensity in each color channel in the low-illumination image (the first image) and the normal-illumination image (the second image), the sum of the probabilities that the pixel intensity in each color channel of the low-illumination image (the first image) and the normal-illumination image (the second image) is less than or equal to a preset pixel intensity is calculated respectively. Specifically, according to the formula:

[0052]

[0053] The sum of the probabilities that all pixel intensities in each single color channel of the target image (the first image and the second image) are less than or equal to a preset pixel intensity is calculated. Wherein, is the sum of the probabilities that all pixel intensities in the color channel M of the target image are less than or equal to i; P(i) is the probability density of the pixel intensity i in each single color channel of the target image; i, j are both pixel intensities in the target image, j = [0, i]; i = [0, 255], i, j are both positive integers; M = {R, G, B}, R, G, and B are red, green, and blue respectively.

[0054] Finally, according to the formula:

[0055]

[0056] The mapping relationship Q(m) = l of the color histogram of the second image to the color histogram of the first image is constructed;

[0057] Wherein, is the sum of probabilities of all pixel intensities in each single color channel of the first image being less than or equal to l; is the sum of probabilities of all pixel intensities in the color channel corresponding to the first image in the second image being less than or equal to m; m, l = [0, 255], m, l are positive integers.

[0058] In step S103, the intensity of each pixel in the second image is adjusted based on the mapping relationship to obtain a third image.

[0059] After the mapping relationship of the color histogram from the normal-illumination image to the low-illumination image is constructed, each pixel intensity m in the second image is adjusted to the pixel intensity l in the first image based on the mapping relationship Q(m) = l to obtain a third image. The third image is the image corresponding to the second image and having the illumination characteristics of the first image.

[0060] Generally, the histogram distribution of the low-illumination image is closer to 0, so the calculated CDF function is steeper near 0, and the histogram distribution of the normal-illumination image is slower near 0. When the mapping relationship Q(m) = l is applied, it is assumed that the pixel intensity of 50 in the normal-illumination image (i.e., the pixel intensity m = 50 in the second image) is to be mapped. The calculated CDF is 0.4, i.e., However, in the low-illumination image is greater than or equal to 0.4, and the included pixel intensity set is {5, 6, 7,..., 255}, i.e., the probability density of the pixel intensity greater than or equal to 5 in the color channel of the first image l = {5, 6, 7,..., 255}, at this time, the minimum value 5 of the set is taken, i.e., l = 5, i.e., the pixel intensity of 50 in the normal-illumination image is mapped to 5.

[0061] Therefore, by applying histogram matching to adjust the histogram distribution of the normal-illumination image to match the characteristics of the low-illumination image, it is possible to generate visually consistent image pairs under different illumination conditions. Through histogram matching, the normal-illumination image is effectively converted to simulate low-illumination conditions, thereby generating a large number of diversified training data sets without the need to reproduce various low-illumination environments in the field, thereby avoiding the significant physical and technical obstacles faced by directly collecting pairs of normal-illumination images and low-illumination images.

[0062] The training data set produced by the method of the present application not only significantly increases in quantity, but also shows richness in scene diversity and illumination changes, providing more comprehensive learning opportunities for the training of supervised neural network models for restoring low-illumination images, and significantly improving the generalization ability of the model when restoring low-illumination images.

[0063] The embodiment of the present application further provides a training data collection system of a neural network model for recovering a low-illumination image, as shown in the following table. Figure 4 The collection system comprises:

[0064] An image acquisition unit 401 is configured to randomly select a first image and a second image from a first image set and a second image set respectively; wherein the first image set is a set of low-illumination images; and the second image set is a set of normal-illumination images.

[0065] A mapping construction unit 402 is configured to construct a mapping relationship from a color histogram of the second image to a color histogram of the first image.

[0066] An illumination matching unit 403 is configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image. The third image is an image corresponding to the second image and having the illumination characteristics of the first image.

[0067] The training data collection system of the neural network model for recovering a low-illumination image provided by the embodiment of the present application can realize the steps and processes of the training data collection method of the neural network model for recovering a low-illumination image of any one of the above embodiments, and achieve the same technical effects, which will not be described one by one here.

[0068] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for collecting training data for a neural network model for recovering low-light images, characterized in that, include: Step S101: Randomly select a first image from the first image set of the low-light image set, and randomly select a second image from the second image set of the normal-light image set; Step S102: Construct a mapping relationship between the color histogram of the second image and the color histogram of the first image; wherein, according to the formula: ； Construct a mapping relationship between the color histogram of the second image and the color histogram of the first image. In the formula, For all pixels in each single color channel of the first image, the intensity is less than or equal to The sum of probabilities; The second image contains all pixels with intensities less than or equal to the color channel corresponding to the first image. The sum of probabilities; , All are positive integers; Step S103: Adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image that corresponds to the second image and has the illumination characteristics of the first image.

2. The method for collecting training data for a neural network model for recovering low-light images according to claim 1, characterized in that, Step S102 includes: Calculate the color histograms of each color channel in the first image and the second image respectively; Based on the color histograms of each color channel in the first image and the second image, the probability density of pixel intensity in each color channel of the first image and the second image is calculated respectively. Based on the probability density of pixel intensity in each color channel of the first image and the second image, calculate the sum of the probabilities that the pixel intensity in each color channel of the first image and the second image is less than or equal to the preset pixel intensity.

3. The method for collecting training data for a neural network model for recovering low-light images according to claim 2, characterized in that, According to the formula: ； Calculate the color histogram of the target image; The target image includes: the first image and the second image; The pixel intensity in the color histogram of a single color channel of the target image The number of pixels; These are the width and height of the target image, respectively; This indicates that the target image is in the color channel. The image in coordinates Pixel value at; , They are red, green, and blue, respectively. , It is a positive integer; For indicator functions, when hour, Otherwise, it is 0.

4. The method for collecting training data for a neural network model for recovering low-light images according to claim 3, characterized in that, According to the formula: ； Calculate the probability density of pixel intensity in each single color channel of the target image; in, The pixel intensity in each single color channel of the target image The probability density; The pixel intensity in the color histogram of a single color channel of the target image The number of pixels; The total number of pixels in the target image; , It is a positive integer.

5. The method for collecting training data for a neural network model for recovering low-light images according to claim 4, characterized in that, According to the formula: ； Calculate the sum of the probabilities that the intensity of all pixels in each single color channel of the target image is less than or equal to a preset pixel intensity; in, For the color channels of the target image All pixel intensities are less than or equal to The sum of probabilities; The pixel intensity in each single color channel of the target image The probability density; All of these are pixel intensities in the target image. ; , All are positive integers; , They are red, green, and blue, respectively.

6. The method for collecting training data for a neural network model for recovering low-light images according to claim 1, characterized in that, In step S103, Based on mapping relationship The intensity of each pixel in the second image All were adjusted to the pixel intensity of the first image. The third image is obtained.

7. A training data collection system for a neural network model for recovering low-light images, characterized in that, include: The image acquisition unit is configured to randomly select a first image from a first image set of a low-illuminance image set, and randomly select a second image from a second image set of a normal-illuminance image set. The mapping construction unit is configured to construct a mapping relationship from the color histogram of the second image to the color histogram of the first image; wherein, according to the formula: ； Construct a mapping relationship between the color histogram of the second image and the color histogram of the first image. In the formula, For all pixels in each single color channel of the first image, the intensity is less than or equal to The sum of probabilities; The second image contains all pixels with an intensity less than or equal to the color channel corresponding to the first image. The sum of probabilities; , All are positive integers; The illuminance matching unit is configured to adjust the intensity of each pixel in the second image based on the mapping relationship to obtain a third image that corresponds to the second image and has the illuminance characteristics of the first image.

Images