Image enhancement method, device, electronic device and computer readable storage medium

Abstract

The present invention proposes an image enhancement method, device, electronic device and computer-readable storage medium, wherein the image enhancement method includes constructing an image enhancement training data set, the data set includes an original image and a corresponding standard image; constructing an image enhancement network model, inputting the original image into the image enhancement network model, and outputting an enhanced image corresponding to the original image, the image enhancement network model includes a coding model and a lookup table generation model, the coding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; using the standard image and the enhanced image to determine the first loss function, and using the color lookup table to determine the second loss function; adjusting the parameters of the coding model and the lookup table generation model according to the first loss function and the second loss function, thereby obtaining a trained image enhancement network model. The method according to the present invention improves the enhancement effect of the image enhancement network model.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to an image enhancement method, device, electronic device and computer-readable storage medium. Background Art

[0002] The imaging environment in coal mines is quite special. Weak light or low illumination conditions make the images taken underground have defects such as low brightness and low contrast, large noise, color distortion, etc., which seriously affect the ground monitoring system's grasp of the underground situation and have a direct impact on the subsequent recurrence analysis. With the advancement of intelligent mining, the application of vision-based intelligent analysis in coal mines is increasing. High-quality images are a necessary condition for visual analysis. Therefore, underground low-illumination image enhancement is an important step to improve image quality and prepare for subsequent image analysis.

[0003] Traditional underground low-light image enhancement often uses histogram equalization and Retinex model-based methods, etc. These methods can enhance the overall contrast effect of the image to a certain extent. However, such methods usually involve more empirical parameters, the enhancement effect is easily distorted, and the processing time of a single frame image is long. With the rapid development of deep learning technology, the enhancement effect of deep learning-based image enhancement methods generally exceeds that of traditional methods. However, in order to achieve better enhancement effects, deep learning-based enhancement methods often use complex network models, which makes the computing, storage and other resource overheads large, making it difficult to apply in real time in coal mines. In addition, the enhancement effect of the image enhancement network model obtained using the existing training method needs to be improved. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first object of the present invention is to propose an image enhancement method to improve the enhancement effect of the image enhancement network model.

[0006] The second objective of the present invention is to provide an image enhancement device.

[0007] A third objective of the present invention is to provide an electronic device.

[0008] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above-mentioned purpose, the first aspect of the present invention proposes an image enhancement method, including: constructing an image enhancement training data set, the image enhancement training data set including an original image and a corresponding standard image; constructing an image enhancement network model, inputting the original image into the image enhancement network model, and outputting an enhanced image corresponding to the original image, wherein the image enhancement network model includes an encoding model and a lookup table generation model, the encoding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; using the standard image and the enhanced image to determine a first loss function, and using the color lookup table to determine a second loss function; adjusting the parameters of the encoding model and the lookup table generation model according to the first loss function and the second loss function, thereby obtaining a trained image enhancement network model.

[0010] In the method of the embodiment of the present invention, the standard image is an image under ideal lighting. Considering that the difference between the standard image and the enhanced image can more intuitively reflect the enhancement effect of the model, in addition, the enhanced image is affected by the color lookup table. Therefore, the standard image and the enhanced image are used to determine the first loss function to supervise the difference between the standard image and the enhanced image. The second loss function is determined according to the color lookup table to supervise the impact of the color lookup table. The first loss function and the second loss function are comprehensively used to train the image enhancement network model, avoiding the use of a single loss function for training as in the prior art, thereby improving the enhancement effect of the image enhancement network model.

[0011] In an image enhancement method of an embodiment of the first aspect of the present invention, the first loss function includes a brightness difference loss function, which is determined based on the standard image and the enhanced image, and the second loss function includes a monotonicity loss function and a perturbation fluctuation loss function, which are determined based on a color lookup table.

[0012] In an image enhancement method of an embodiment of the first aspect of the present invention, the brightness difference loss function is determined based on the standard image and the enhanced image, including: for each first pixel in the standard image, determining the second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of three times the difference, summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel; determining the average brightness difference of the enhanced image based on the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function.

[0013] In an image enhancement method of an embodiment of the first aspect of the present invention, the monotonicity loss function and the perturbation fluctuation loss function are determined according to a color lookup table, including: for each first-position color point in the color lookup table, determining the second-position color point adjacent to the first-position color point, and calculating the square of the color difference between the first-position color point and the second-position color point, determining the color difference average value of the color lookup table according to the square of the color difference corresponding to each first-position color point, and using the color difference average value as the perturbation fluctuation loss function; calculating the color difference between the first-position color point and the second-position color point, truncating the part where the color difference is less than 0, obtaining the forward gradient value of the first-position color point, determining the forward gradient value average value of the color lookup table according to the forward gradient value corresponding to each first-position color point, and using the forward gradient value average value as the monotonicity loss function.

[0014] In an image enhancement method of an embodiment of the first aspect of the present invention, constructing an image enhancement training data set includes: acquiring original images under different illuminations in different scenes, and acquiring corresponding standard images in the same scene as the original images.

[0015] In an image enhancement method of an embodiment of the first aspect of the present invention, obtaining a corresponding standard image in the same scene as the original image includes: using feature point registration technology to perform image registration on image information in the same scene to obtain a standard image corresponding to the original image.

[0016] In an image enhancement method of an embodiment of the first aspect of the present invention, the original image is input into the image enhancement network model, and outputting an enhanced image corresponding to the original image includes: numerically quantizing the original image according to the representation scale of the color lookup table to obtain quantitative features, and according to the quantitative features of the original image, using a bilinear interpolation method to interpolate and search for corresponding output pixels in the color lookup table to output an enhanced image.

[0017] To achieve the above-mentioned purpose, the second aspect of the present invention proposes an image enhancement device, including an acquisition module for acquiring an image enhancement training data set, wherein the image enhancement training data set includes an original image and a corresponding standard image; an enhancement processing module for constructing an image enhancement network model, inputting the original image into the image enhancement network model, and outputting an enhanced image corresponding to the original image, wherein the image enhancement network model includes an encoding model and a lookup table generation model, the encoding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; a first loss function is determined using the standard image and the enhanced image, and a second loss function is determined using the color lookup table; parameters of the encoding model and the lookup table generation model are adjusted according to the first loss function and the second loss function, so as to obtain a trained image enhancement network model.

[0018] In the device of the embodiment of the present invention, the standard image is an image under ideal lighting. Considering that the difference between the standard image and the enhanced image can more intuitively reflect the enhancement effect of the model, in addition, the enhanced image is affected by the color lookup table. Therefore, the standard image and the enhanced image are used to determine the first loss function to supervise the difference between the standard image and the enhanced image. The second loss function is determined according to the color lookup table to supervise the impact of the color lookup table. The first loss function and the second loss function are comprehensively used to train the image enhancement network model, avoiding the use of a single loss function for training as in the prior art, thereby improving the enhancement effect of the image enhancement network model.

[0019] In an image enhancement device of an embodiment of the second aspect of the present invention, the first loss function includes a brightness difference loss function, which is determined based on the standard image and the enhanced image, and the second loss function includes a monotonicity loss function and a disturbance fluctuation loss function, which are determined based on a color lookup table.

[0020] In an image enhancement device of an embodiment of the second aspect of the present invention, the brightness difference loss function is determined based on the standard image and the enhanced image, including: for each first pixel in the standard image, determining the second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of three times the difference, summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel; determining the average brightness difference of the enhanced image based on the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function.

[0021] In an image enhancement device of an embodiment of the second aspect of the present invention, the monotonicity loss function and the disturbance fluctuation loss function are determined according to a color lookup table, including: for each first-position color point in the color lookup table, determining a second-position color point adjacent to the first-position color point, and calculating the square of the color difference between the first-position color point and the second-position color point, determining the color difference average value of the color lookup table according to the square of the color difference corresponding to each first-position color point, and using the color difference average value as the disturbance fluctuation loss function; calculating the color difference between the first-position color point and the second-position color point, truncating the part of the color difference less than 0, obtaining the forward gradient value of the first-position color point, determining the forward gradient value average value of the color lookup table according to the forward gradient value corresponding to each first-position color point, and using the forward gradient value average value as the monotonicity loss function.

[0022] In an image enhancement device of an embodiment of the second aspect of the present invention, in an acquisition module, constructing an image enhancement training data set includes: acquiring original images under different illuminations in different scenes, and acquiring corresponding standard images in the same scene as the original images.

[0023] In an image enhancement device of an embodiment of the second aspect of the present invention, the obtaining of a corresponding standard image in the same scene as the original image includes: using feature point registration technology to perform image registration on image information in the same scene to obtain a standard image corresponding to the original image.

[0024] In an image enhancement device of an embodiment of the second aspect of the present invention, in an enhancement processing module, the original image is input into the image enhancement network model, and outputting an enhanced image corresponding to the original image includes: numerically quantizing the original image according to the representation scale of the color lookup table to obtain quantitative features, and according to the quantitative features of the original image, using a bilinear interpolation method to interpolate and search for corresponding output pixels in the color lookup table to output an enhanced image.

[0025] To achieve the above-mentioned purpose, the third aspect of the present invention proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image enhancement method of the first aspect of the present invention.

[0026] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the image enhancement method of the first aspect of the present invention.

[0027] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A schematic diagram of a flow chart of an image enhancement method provided by an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of an image enhancement network model provided by an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a flow chart of a method for obtaining a brightness difference loss function provided by an embodiment of the present invention;

[0032] Figure 4 A schematic flow chart of a method for obtaining a disturbance fluctuation loss function and a monotonicity loss function provided in an embodiment of the present invention;

[0033] Figure 5 A flowchart of offline training of the image enhancement network model provided by an embodiment of the present invention;

[0034] Figure 6 An online test flow chart of the image enhancement network model provided by an embodiment of the present invention;

[0035] Figure 7 A schematic diagram of the online test effect provided by an embodiment of the present invention;

[0036] Figure 8 A schematic structural diagram of an image enhancement device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limitations on the present invention. Any process or method description in the flow chart or otherwise described herein may be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a custom logic function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, in which functions may be performed in a substantially simultaneous manner or in reverse order according to the functions involved, which should be understood by a person skilled in the art of the art to which the embodiments of the present invention belong.

[0038] The image enhancement method and apparatus according to the embodiments of the present invention are described below with reference to the accompanying drawings.

[0039] Figure 1 A schematic diagram of a flow chart of an image enhancement method provided by an embodiment of the present invention.

[0040] The embodiment of the present invention provides an image enhancement method to improve the enhancement effect of the image enhancement network model, such as Figure 1 As shown, the image enhancement method comprises the following steps:

[0041] Step S10, constructing an image enhancement training data set.

[0042] In step S10, the image enhancement training data set may include original images and corresponding standard images. The images in the image enhancement training data set may be acquired through an image acquisition device, which includes but is not limited to acquisition devices such as cameras and monitors. The camera may be a SLR camera.

[0043] In step S10, constructing an image enhancement training data set may include: acquiring original images under different illuminations in different scenes, acquiring corresponding standard images in the same scene as the original images, and using the acquired original images and corresponding standard images to form an image enhancement training data set. Wherein, acquiring the corresponding standard images in the same scene as the original images includes performing image registration on image information in the same scene through an image feature point registration technology, and acquiring an image data pair (i.e., an original image and a paired standard image), and the standard image is an image under normal illumination.

[0044] In step S10, the sizes of the images obtained under different illuminations of different scenes may be the same or different. When the sizes of the images are different, the image data pair needs to be cropped to obtain a valid image data pair. In this case, cropping can make the sizes of the images in the image data pair consistent, which is convenient for improving the accuracy of the subsequent training model.

[0045] In this embodiment, a coal mine is taken as an example to construct an image enhancement training data set for an underground coal mine. Specifically, multiple image information of multiple scenes (i.e., places) such as the working face, tunnel, and chamber in the underground coal mine is collected. Image information under different illumination conditions (including low illumination and normal illumination) is collected in each scene, and the normal illumination image is recorded as I N0 , the low illumination image is recorded as I L0 Due to the disturbance of the image acquisition device, it is easy to cause the images collected in the same scene to be misaligned. It is necessary to align the images in the same scene. The alignment process includes image I N0 and image I L0 The SIFT features are extracted respectively, and the RANSAC algorithm is used to match the feature points. Then, the homography matrix of the image pair is obtained by using the paired feature points. N0 and image I L0 Then the effective common part of the image is cut out to form a valid image data pair (I L ,I N ), I L To align the cropped low-light image, I N To align the cropped normal illumination image. In addition, in this embodiment, in order to better accelerate network training, M pairs of images are formed, the image size is uniformly 480×720, 80% of the image data are randomly selected as training data, and the remaining 20% ​​of the data are used as test data. In other embodiments, the number of image data pairs, the size of the image, and the ratio of training data to test data are not limited thereto.

[0046] Step S11, constructing an image enhancement network model.

[0047] In step S11, the image enhancement network model may include two parts: an encoding model and a lookup table generation model (i.e., a decoding model). The encoding model is used to encode the original image to obtain the base point information of the lookup table, and the lookup table generation model is used to generate a color lookup table using the base point information of the lookup table. The original image is input into the image enhancement network model, and an enhanced image corresponding to the original image can be output. Specifically: the image enhancement network model numerically quantizes the original image according to the representation scale of the color lookup table to obtain quantitative features, and according to the quantitative features of the original image, a bilinear interpolation method is used to interpolate and search for the corresponding output pixels in the color lookup table to output the enhanced image. The enhanced image is denoted as I E In this case, an end-to-end low-light image enhancement network is constructed through a neural network, and the neural network is constructed using encoding and decoding methods to generate a color lookup table. Through the lookup table color mapping, the original image is enhanced.

[0048] In this embodiment, Figure 2 A schematic diagram of the structure of the image enhancement network model provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown in FIG. 1 , the encoding model may include a first structural block consisting of an upsampling layer (Upsample), a convolution layer (Conv), a normalization layer (BatchNorm), an activation layer (LeakyReLU) and a maximum pooling layer (MaxPool), three structural blocks consisting of a convolution layer, a normalization layer and an activation layer, and a fifth structural block consisting of an average pooling layer (AvgPool), a convolution layer and an activation layer. The original image is encoded after a series of modules, which is the base point of the lookup table. The specific network structure parameters of the encoding model can be shown in Table 1.

[0049] Table 1 Specific network structure parameters of the encoding model

[0050]

[0051]

[0052] In this embodiment, as shown in Table 1, a transformation layer (ReShape) can also be set in the 5th structural block of the encoding model. The transformation layer is set after the activation layer and is used to transform the size of the feature map output by the activation layer, so that the output dimension of the encoding model meets the network data dimension required by the decoding model.

[0053] like Figure 2As shown, the decoding model may include multiple structural blocks. In this embodiment, the number of structural blocks of the decoding model is 4. Each structural block includes an upsampling layer, a convolution layer, a normalization layer, and an activation layer. The base point information is input into the decoding model, and a color lookup table is obtained after upsampling, convolution, regularization, and activation operations. The color lookup table is used to characterize the mapping relationship between the pixel information of the original image and the pixel information of the enhanced image. In this embodiment, the color lookup table may be a three-dimensional color lookup table (3D LUT). The pixel information of the original image may include the RGB color value of the image. The specific network structure parameters of the decoding model may be as shown in Table 2.

[0054] Table 2 Specific network structure parameters of the decoding model

[0055]

[0056]

[0057] In this embodiment, as shown in Table 2, a transformation layer (ReShape) may also be set in the fourth structural block of the decoding model. The transformation layer is set after the upsampling layer, and is used to convert the feature map output by the upsampling layer into a tensor that is convenient for machine operation. For example, in this embodiment, the transformation layer in the fourth structural block converts the feature map of -1×33×33×99 output by the upsampling layer into a five-dimensional tensor of -1×3×33×33×33. At this time, the color lookup table represents the mapping relationship between the original image and the enhanced image through the five-dimensional tensor, and the original image of size -1×3×H×W is mapped through the five-dimensional tensor to obtain an enhanced image of size -1×3×H×W.

[0058] Step S12, constructing the target loss function of the image enhancement network model.

[0059] In step S12, the target loss function includes a first loss function and a second loss function, and the first loss function can be determined by the standard image and the enhanced image. In this embodiment, the first loss function can include a brightness difference loss function, and the brightness difference loss function can be determined according to the standard image and the enhanced image. Figure 3 A schematic diagram of a flow chart of a method for obtaining a brightness difference loss function provided by an embodiment of the present invention is shown in FIG. Figure 3As shown, the determination method includes: for each first pixel in the standard image, determining the second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of the three times the difference (step S120), summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel (step S121); determining the average brightness difference of the enhanced image according to the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function (step S122), the brightness difference loss function is as shown in formula (1):

[0060] Loss1=||I E -I N || 2 +3×||I E -I N || 1 (1)

[0061] Where ||*|| 2 represents the 2-norm, ||*|| 1 represents the 1 norm, Loss1 represents the brightness difference loss function, I E represents the enhanced image, I N is a normal illumination image (i.e., a standard image).

[0062] In step S12, the second loss function can be determined using a color lookup table. In this embodiment, the second loss function can be determined by the characteristics of the color lookup table itself, and the second loss function includes a monotonicity loss function determined according to the tonality of the color lookup table and a disturbance fluctuation loss function determined according to the smoothness of the color lookup table. Figure 4 A flow chart of a method for obtaining a disturbance fluctuation loss function and a monotonicity loss function provided in an embodiment of the present invention is shown in FIG. Figure 4 As shown, the determination method includes: for each first position color point in the color lookup table, determining the second position color point adjacent to the first position color point (step S123); and calculating the square of the color difference between the first position color point and the second position color point, determining the color difference average value of the color lookup table according to the square of the color difference corresponding to each first position color point, and using the color difference average value as the disturbance fluctuation loss function (step S124); calculating the color difference between the first position color point and the second position color point, truncating the part of the color difference less than a preset value (such as 0), and obtaining the forward gradient value of the first position color point (step S125); determining the forward gradient value average value of the color lookup table according to the forward gradient value corresponding to each first position color point, and using the forward gradient value average value as the monotonicity loss function (step S126). The disturbance fluctuation loss function is:

[0063]

[0064] Where Loss2 represents the perturbation fluctuation loss function, c represents the color space of the color table, and the color space adopts RGB format, so c∈{r,g,b}, c(i,j,k) represents the value of the color table at the coordinate (i,j,k), that is, the value of the first position color point, c (i+1,j,k) 、c (i,j+1,k) and c (i,j,k+1) They all represent the value of the second adjacent color point after c(i,j,k).

[0065] The monotonicity loss function is:

[0066]

[0067] Wherein Loss3 represents a monotonic loss function, wherein g(x)=max(0,x). In other embodiments, the preset value may not be 0, and the preset value may be set based on actual needs.

[0068] In this embodiment, the target loss function Loss is as follows:

[0069] Loss=Loss1+C1×Loss2+C2×Loss3 (2)

[0070] Wherein, C1 and C2 are weight coefficients. In this embodiment, the weight coefficients may be set to C1=0.05 and C2=0.005.

[0071] Step S13, training the image enhancement network model to obtain a trained image enhancement network model.

[0072] In step S13, the image enhancement network model is trained according to the target loss function, that is, the parameters of the encoding model and the lookup table generation model are adjusted using the first loss function and the second loss function, so as to obtain a trained image enhancement network model. The training process includes two parts: an offline training phase and an online testing phase.

[0073] In the offline training stage, a certain proportion of image data pairs are selected from the image enhancement training data set obtained in step S10 as training sample data, and the image enhancement network model is trained offline using the training sample data. Figure 5 This is a flowchart of the offline training of the image enhancement network model provided by the embodiment of the present invention. Figure 5As shown in the figure, during the offline training stage, the original image is first augmented by random cropping, random flipping, brightness adjustment, and hue adjustment to expand the diversity of the training sample data. Then the augmented original image is used as the input of the encoding model, and the enhanced image, standard image, and color lookup table are used to determine the target loss function to conduct supervised training on the network model.

[0074] When training network parameters in the offline training phase, the learning rate can be set by using the cosine hot start annealing training method. The initial cycle of the cosine annealing method is set to 100 epochs, the cycle scale factor is set to 2, the initial learning rate is set to 0.0016, and the minimum learning rate is set to 0.0001. The total training cycle is set to 800 epochs, of which the last 100 epochs use a fixed minimum learning rate. Thus, the cosine hot start annealing training method is used to perform multiple iterations of training on the enhancement network to obtain a trained image enhancement network model.

[0075] In the online testing phase, the remaining image data pairs are selected from the image enhancement training data set obtained in step S10 as test sample data, and the image enhancement network model that has completed offline training is tested online using the test sample data. Taking the coal mine underground image enhancement training data set as an example, this embodiment uses NVIDIA RTX 2080 GPU to perform low-light image enhancement test in coal mines. Figure 6 The following is a flowchart of an online test of an image enhancement network model provided by an embodiment of the present invention. Figure 7 The figure is a schematic diagram of the online test effect provided by the embodiment of the present invention. The original image in the test sample data pair is directly input into the image enhancement network model that has completed offline training to obtain an enhanced image, and then the enhanced image is compared with the standard image to verify the enhancement effect. Through testing, it takes less than 2ms to enhance the 4K input image (resolution size is 4096×2160), which has high real-time performance. The enhancement effect of this embodiment on the low-light image in the coal mine is as follows: Figure 7 As shown, Figure 7 The images in the left column are low-light images (original images) taken underground, the images in the middle column are images enhanced by the method of this example, and the images in the right column are standard images. It can be seen that this embodiment can effectively enhance low-light original images such as low-light images in coal mines.

[0076] The image enhancement method of this embodiment first performs same-scene image data pair registration, image coding network construction, decoding network construction, enhancement network loss function construction, and enhancement network training method construction, and then the low-light image in the paired image is augmented by image transformation and input into the image coding network to generate three-dimensional base point data after image encoding, and the three-dimensional base point data is input into the decoding network to obtain a three-dimensional color lookup table corresponding to the low-light image, and the low-light image is mapped to an enhanced image through the three-dimensional color lookup table, and the first loss function value and the second loss function value of the network training are determined according to the enhanced image, the three-dimensional color lookup table and the reference image, and a supervised network is trained by using a hot-start cosine annealing method. Training, wherein since the standard image is an image under ideal illumination (i.e., an image under normal illumination), considering that the difference between the standard image and the enhanced image can more intuitively reflect the enhancement effect of the model, in addition, the enhanced image is affected by the color lookup table, so the first loss function is determined using the standard image and the enhanced image to supervise the difference between the standard image and the enhanced image, and the second loss function is determined according to the color lookup table to supervise the influence caused by the color lookup table, and the image enhancement network model is trained by comprehensively utilizing the first loss function and the second loss function, avoiding the use of a single loss function for training as in the prior art, thereby improving the enhancement effect of the image enhancement network model. In addition, in the present invention, the image enhancement network parameters obtained after training are small, only 970K parameters, and the processing time for 4K images is less than 2ms, which has high real-time performance, and the method steps are simple, the design is reasonable, and the calculation time is short, which greatly simplifies the process of low-light image enhancement, and achieves good results in low-light image enhancement in coal mines, effectively adapts to the needs of special lighting environments in coal mines, and provides basic conditions for subsequent visual tasks.

[0077] Based on the trained image enhancement network model of this embodiment, if an original image to be enhanced of any size is collected in real time and the original image to be enhanced is input into the trained image enhancement network model, a corresponding enhanced image can be obtained.

[0078] In order to implement the above embodiment, the present invention also provides an image enhancement device.

[0079] Figure 8 A schematic structural diagram of an image enhancement device provided by an embodiment of the present invention.

[0080] like Figure 8 As shown, the image enhancement device 1 includes an acquisition module 10 and an enhancement processing module 20 .

[0081] In this embodiment, the acquisition module 10 is used to acquire an image enhancement training data set. The image enhancement training data set includes an original image and a corresponding standard image. The acquisition module 10 can be an image acquisition device, which includes but is not limited to acquisition devices such as a camera and a monitor. The camera can be a SLR camera.

[0082] In the acquisition module 10, constructing the image enhancement training data set includes: acquiring original images under different illuminations in different scenes, and acquiring corresponding standard images in the same scene as the original images. Acquiring the corresponding standard images in the same scene as the original images includes: performing image registration on the image information in the same scene using feature point registration technology to acquire the standard image corresponding to the original image.

[0083] The enhancement processing module 20 is used to construct an image enhancement network model, input the original image into the image enhancement network model, and output an enhanced image corresponding to the original image, wherein the image enhancement network model includes a coding model and a lookup table generation model, the coding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; the first loss function is determined using the standard image and the enhanced image, and the second loss function is determined using the color lookup table; the parameters of the coding model and the lookup table generation model are adjusted according to the first loss function and the second loss function, so as to obtain a trained image enhancement network model.

[0084] In the enhancement processing module 20, the first loss function includes a brightness difference loss function, which is determined according to the standard image and the enhanced image, and the second loss function includes a monotonicity loss function and a perturbation fluctuation loss function, which are determined according to the color lookup table. The method for determining the brightness difference loss function includes: for each first pixel in the standard image, determining the second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of three times the difference, summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel; determining the average brightness difference of the enhanced image according to the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function.

[0085] The method for determining the monotonicity loss function and the perturbation fluctuation loss function includes: for each first-position color point in the color lookup table, determining the second-position color point adjacent to the first-position color point, and calculating the square of the color difference between the first-position color point and the second-position color point, determining the average color difference value of the color lookup table according to the square of the color difference value corresponding to each first-position color point, and using the average color difference value as the perturbation fluctuation loss function; calculating the color difference between the first-position color point and the second-position color point, truncating the part where the color difference is less than 0, obtaining the forward gradient value of the first-position color point, determining the average forward gradient value of the color lookup table according to the forward gradient value corresponding to each first-position color point, and using the average forward gradient value as the monotonicity loss function.

[0086] In the enhancement processing module 20, the original image is input into the image enhancement network model, and the enhanced image corresponding to the original image is output, including: the original image is numerically quantized according to the representation scale of the color lookup table to obtain quantitative features, and according to the quantitative features of the original image, a bilinear interpolation method is used to interpolate and search for corresponding output pixels in the color lookup table to output the enhanced image.

[0087] It should be noted that the above explanation of the embodiment of the image enhancement method is also applicable to the image enhancement device of this embodiment, and will not be repeated here.

[0088] In order to implement the above embodiments, the present invention also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the aforementioned image enhancement method of the present invention.

[0089] In order to implement the above embodiment, the present invention further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image enhancement method in the above-mentioned image enhancement method embodiment of the present invention.

[0090] For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with an instruction execution system, device or apparatus or in conjunction with such instruction execution systems, devices or apparatus. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, because the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory. The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc.

[0091] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above embodiments can be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module.

[0092] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

Claims

1. An image enhancement method, characterized in that: include: Constructing an image enhancement training data set, wherein the image enhancement training data set includes an original image and a corresponding standard image; Constructing an image enhancement network model, inputting the original image into the image enhancement network model, and outputting an enhanced image corresponding to the original image, wherein the image enhancement network model includes a coding model and a lookup table generation model, the coding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; Determine a first loss function using the standard image and the enhanced image, and determine a second loss function using the color lookup table; Adjusting parameters of the encoding model and the lookup table generation model according to the first loss function and the second loss function, thereby obtaining a trained image enhancement network model; The step of inputting the original image into the image enhancement network model and outputting an enhanced image corresponding to the original image comprises: numerically quantizing the original image according to the representation scale of the color lookup table to obtain a quantized feature, and using a bilinear interpolation method to interpolate and search for corresponding output pixels in the color lookup table according to the quantized feature of the original image to output an enhanced image; The first loss function includes a brightness difference loss function, and the brightness difference loss function is determined according to the standard image and the enhanced image, including: for each first pixel in the standard image, determining a second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of three times the difference, and summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel; determining the average brightness difference of the enhanced image according to the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function, The second loss function includes a monotonicity loss function and a perturbation fluctuation loss function, and the monotonicity loss function and the perturbation fluctuation loss function are determined according to a color lookup table and include: for each first-position color point in the color lookup table, determine the second-position color point adjacent to the first-position color point, and calculate the square of the color difference between the first-position color point and the second-position color point, determine the color difference average value of the color lookup table according to the square of the color difference corresponding to each first-position color point, and use the color difference average value as the perturbation fluctuation loss function; calculate the color difference between the first-position color point and the second-position color point, truncate the part where the color difference is less than 0, obtain the forward gradient value of the first-position color point, determine the forward gradient value average value of the color lookup table according to the forward gradient value corresponding to each first-position color point, and use the forward gradient value average value as the monotonicity loss function.

2. The image enhancement method according to claim 1, characterized in that: The constructing of the image enhancement training data set includes: acquiring original images under different illuminations in different scenes, and acquiring corresponding standard images in the same scene as the original images.

3. The image enhancement method according to claim 2, characterized in that: The obtaining of a corresponding standard image in the same scene as the original image comprises: performing image registration on image information in the same scene using a feature point registration technology to obtain a standard image corresponding to the original image.

4. An image enhancement device, characterized in that: include: An acquisition module is used to acquire an image enhancement training data set, wherein the image enhancement training data set includes an original image and a corresponding standard image; The enhancement processing module is used to construct an image enhancement network model, input the original image into the image enhancement network model, and output an enhanced image corresponding to the original image, wherein the image enhancement network model includes a coding model and a lookup table generation model, the coding model is used to encode the original image to obtain base point information, and the lookup table generation model is used to generate a color lookup table using the base point information; a first loss function is determined using the standard image and the enhanced image, and a second loss function is determined using the color lookup table; parameters of the coding model and the lookup table generation model are adjusted according to the first loss function and the second loss function, thereby obtaining A trained image enhancement network model, wherein the first loss function includes a brightness difference loss function, and the brightness difference loss function is determined according to the standard image and the enhanced image, including: for each first pixel in the standard image, determining the second pixel corresponding to the first pixel in the enhanced image, and calculating the square of the difference between the first pixel and the second pixel and the absolute value of three times the difference, summing the square and the absolute value to obtain the brightness difference corresponding to the first pixel; determining the average brightness difference of the enhanced image according to the brightness difference corresponding to each first pixel, and using the average brightness difference as the brightness difference loss function, The second loss function includes a monotonicity loss function and a perturbation fluctuation loss function, and the monotonicity loss function and the perturbation fluctuation loss function are determined according to a color lookup table and include: for each first-position color point in the color lookup table, determining a second-position color point adjacent to the first-position color point, and calculating the square of the color difference between the first-position color point and the second-position color point, determining the color difference average value of the color lookup table according to the square of the color difference corresponding to each first-position color point, and using the color difference average value as the perturbation fluctuation loss function; calculating the color difference between the first-position color point and the second-position color point, truncating the part where the color difference is less than 0, obtaining the forward gradient value of the first-position color point, determining the forward gradient value average value of the color lookup table according to the forward gradient value corresponding to each first-position color point, and using the forward gradient value average value as the monotonicity loss function; The enhancement processing module is also used to numerically quantize the original image according to the representation scale of the color lookup table to obtain quantitative features. According to the quantitative features of the original image, a bilinear interpolation method is used to interpolate and search for corresponding output pixels in the color lookup table to output an enhanced image.

5. The image enhancement device according to claim 4, characterized in that: In the acquisition module, the construction of the image enhancement training data set includes: acquiring original images under different illuminations in different scenes, and acquiring corresponding standard images in the same scene as the original images.

6. The image enhancement device according to claim 5, characterized in that: The obtaining of a corresponding standard image in the same scene as the original image comprises: performing image registration on image information in the same scene using a feature point registration technology to obtain a standard image corresponding to the original image.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image enhancement method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the image enhancement method according to any one of claims 1-3.

Images