Low-light image enhancement model construction method, device, and system
Convolutional neural network constructs a curve parameter estimation network, uses first-order and second-order parameters to perform low-light image enhancement, and iterative updates through loss function, solving the problem of image blurring and color offset in low-light image enhancement, and improving image quality .
Patent Information
- Application Number
- CN202410874474.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-02
AI Technical Summary
The existing low-light image enhancement methods will become more and more blurry when the number of iterations increases, and color shifts are prone to occur after processing of different training models, resulting in poor image quality.
Convolutional neural network is used to construct curve parameter estimation network, image enhancement is performed through first-order and second-order parameters, and iteratively updates based on loss functions such as image color proportional consistency, spatial consistency, exposure level, color constant and brightness monotonicity. Until the convergence condition is reached, a low-light image enhancement model is obtained.
It effectively avoids image blur, improves the clarity and color consistency of low-light images, and improves image quality.
Smart Images

Figure CN118411301B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image enhancement, and in particular to a method, device, system, computer equipment and computer-readable storage medium for constructing a low-light image enhancement model. Background Art
[0002] Lighting is an important factor affecting image quality. Images obtained in scenes with poor lighting conditions often suffer from loss of details, unclear resolution, and low signal-to-noise ratio. These problems are not conducive to human observation and computer vision recognition. Therefore, in order to improve low-light images, many low-light image enhancement methods have been proposed to improve the quality of the original image and restore image information.
[0003] However, the expected effect is often not achieved after enhancing low-light images. The enhanced low-light images still have problems such as color distortion, noise and blur. On the one hand, as the number of enhancements increases, the obtained image will become more and more blurred; on the other hand, when processing low-light images based on different training models, color cast problems will also occur. These factors lead to the problem of poor quality of enhanced images. Summary of the invention
[0004] The embodiments of the present application provide a method, device, and system for constructing a low-light image enhancement model to at least solve the problem of poor image quality in the related art caused by an increase in the number of iterations and the processing of low-light images by different training models.
[0005] In a first aspect, an embodiment of the present application provides a method for constructing a low-light image enhancement model, comprising:
[0006] The image to be processed is input into a curve parameter estimation network, and first-order parameters and second-order parameters are constructed according to the output results. The curve parameter estimation network is a convolutional neural network, which is used to output the RGB channel change parameters of each pixel point of the image;
[0007] Acquire a first enhanced image according to the first-order parameters and the image to be processed, and acquire a second enhanced image according to the second-order parameters and the first enhanced image;
[0008] A loss function is constructed according to the image to be processed and the second enhanced image, and the curve parameter estimation network is iteratively updated based on the loss function until a convergence condition is reached, thereby obtaining a low-light image enhancement model.
[0009] In one embodiment, the step of inputting the image to be processed into a curve parameter estimation network and constructing first-order parameters and second-order parameters according to the output results includes:
[0010] Inputting the image to be processed into the curve parameter estimation network to obtain an output result, wherein the output result includes RGB channel estimation parameters, R channel estimation parameters, G channel estimation parameters, and B channel estimation parameters;
[0011] Constructing first-order parameters according to the RGB channel estimation parameters, wherein the first-order parameters include R parameters, G parameters and B parameters of the RGB channel estimation parameters;
[0012] Second-order parameters are constructed according to the R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters.
[0013] In one embodiment, the step of acquiring the first enhanced image according to the first-order parameter and the image to be processed includes:
[0014] Extracting R pixels, G pixels and B pixels of each pixel in the image to be processed;
[0015] Acquire a first enhanced R pixel according to the R parameter and the R pixel;
[0016] Acquire a first enhanced G pixel according to the G parameter and the G pixel;
[0017] Acquire a first enhanced B pixel according to the B parameter and the B pixel;
[0018] A first enhanced image is acquired according to the first enhanced R pixel, the first enhanced G pixel and the first enhanced B pixel of each pixel point.
[0019] In one embodiment, the step of acquiring the second enhanced image according to the second-order parameter and the first enhanced image includes:
[0020] Obtaining an R pixel difference, a G pixel difference, and a B pixel difference according to the difference between the square of the pixel and the pixel of each pixel point of the first enhanced image;
[0021] Acquire a second enhanced R pixel according to the R pixel difference and the R channel estimation parameter;
[0022] Acquire a second enhanced G pixel according to the G pixel difference and the G channel estimation parameter;
[0023] Acquire a second enhanced B pixel according to the B pixel difference and the B channel estimation parameter;
[0024] A second enhanced image is acquired according to the second enhanced R pixel, the second enhanced G pixel and the second enhanced B pixel of each pixel point.
[0025] In one embodiment, the loss function includes:
[0026] An image color ratio consistency loss function, used to characterize the color ratio consistency between the second enhanced image and the image to be processed;
[0027] A spatial consistency loss function, used to make the neighborhood of the second enhanced image and the image to be processed consistent;
[0028] an exposure level loss function, used to characterize the exposure degree of the second enhanced image;
[0029] A color constant loss function, used to characterize the color deviation of the second enhanced image;
[0030] A brightness monotonicity loss function is used to characterize the monotonicity of adjacent pixels of the second enhanced image.
[0031] In one embodiment, the loss function of the image color ratio consistency is configured as:
[0032]
[0033] Among them, q and p are any channels in RGB channels, and p and q channels are different.
[0034] represents the average intensity value of the p channel of the second enhanced image,
[0035] represents the average intensity value of the q channel of the second enhanced image,
[0036] represents the average intensity value of the p channel of the image to be processed,
[0037] Represents the average intensity value of the q channel of the image to be processed.
[0038] In one embodiment, before inputting the image to be processed into the curve parameter estimation network, the method further includes:
[0039] A low-light image is acquired, and normalization processing is performed on the low-light image to obtain the image to be processed.
[0040] In a second aspect, an embodiment of the present application provides a low-light image enhancement method, which is applied to a low-light image enhancement model obtained by the construction method described in any one of the first aspects, and the method includes:
[0041] Get the image to be processed;
[0042] Inputting the image to be processed into the low-light image enhancement model, and obtaining RGB channel change parameters of each pixel based on a curve parameter estimation network in the low-light image enhancement model;
[0043] The image to be processed is enhanced according to the RGB channel change parameters to obtain an enhanced image.
[0044] In a third aspect, an embodiment of the present application provides a low-light image enhancement model construction device, including:
[0045] A construction module is used to input the image to be processed into a curve parameter estimation network, and construct first-order parameters and second-order parameters according to the output results. The curve parameter estimation network is a convolutional neural network, which is used to output the RGB channel change parameters of each pixel point of the image;
[0046] A first acquisition module, used for acquiring a first enhanced image according to the first-order parameters and the image to be processed, and acquiring a second enhanced image according to the second-order parameters and the first enhanced image;
[0047] The second acquisition module constructs a loss function according to the image to be processed and the second enhanced image, iteratively updates the curve parameter estimation network based on the loss function until a convergence condition is reached, and obtains a low-light image enhancement model.
[0048] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for constructing a low-light image enhancement model as described in the first aspect above is implemented.
[0049] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a low-light image enhancement model as described in the first aspect above.
[0050] The low-light image enhancement model construction method, device, and system provided in the embodiments of the present application have at least the following technical effects.
[0051] The image to be processed is input into the curve parameter estimation network, and the first-order parameters and second-order parameters are constructed according to the output results. The first enhanced image is obtained according to the first-order parameters and the image to be processed; the second enhanced image is obtained according to the second-order parameters and the first enhanced image. Then, a loss function is constructed according to the image to be processed and the second enhanced image, and the curve parameter estimation network is iteratively updated based on the loss function until the convergence condition is reached to obtain a low-light image enhancement model. In this model, the first-order parameters and the image to be processed to obtain the first enhanced image can enlarge the adjacent pixel values, so that image blur can be avoided in the subsequent continuous iteration process, and the low-light image can be enhanced, thereby improving the problem of poor low-light image quality.
[0052] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0054] Figure 1 is a flow chart of a method for constructing a low-light image enhancement model according to an exemplary embodiment;
[0055] Figure 2 is a schematic diagram of a curve parameter estimation network provided by an exemplary embodiment;
[0056] Figure 3 A flowchart of a low-light image enhancement method according to an exemplary embodiment;
[0057] Figure 4 is a block diagram of a low-light image enhancement model building device according to an exemplary embodiment;
[0058] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0060] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.
[0061] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0062] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0063] In a first aspect, the present application provides a method for constructing a low-light image enhancement model. Figure 1 It is a flowchart of the low-light image enhancement model construction method. Figure 1 As shown, the low-light image enhancement model construction method includes:
[0064] Step S101: input the image to be processed into a curve parameter estimation network, and construct first-order parameters and second-order parameters according to the output results. The curve parameter estimation network is a convolutional neural network, which is used to output the RGB channel change parameters of each pixel point of the image.
[0065] Figure 2 is a schematic diagram of a curve parameter estimation network provided by an exemplary embodiment, such as Figure 2As shown in the figure, the curve parameter estimation network consists of seven convolutional neural networks with symmetrical structures. In each convolutional neural network, each convolution layer consists of 32 depth-wise separable convolutions, and each depth-wise separable convolution consists of a depth-wise convolution and a point-wise convolution. The size of the convolution kernel of each depth-wise convolution is 3×3, with a step size of 1; the size of the convolution kernel of each point-wise convolution is 1×1, with a step size of 1. The ReLU activation function is used after each convolution layer, and the Tanh activation function is used after the last convolution layer.
[0066] The number of output channels of the network can be set according to the actual application requirements. Each channel outputs the RGB channel change parameters of each pixel of the image, where the change parameter is the value of the change between each pixel and the pixel of the image to be processed after the image to be processed is enhanced.
[0067] In one embodiment, the number of output channels of the curve parameter estimation network is set to 4. The first channel is used to output the estimated parameters of the RGB channels, the second channel is used to output the estimated parameters of the R channel, the third channel is used to output the estimated parameters of the G channel, and the fourth channel is used to output the estimated parameters of the B channel. Therefore, the image to be processed is input into the curve parameter estimation network to obtain the output result, wherein the output result includes the estimated parameters of the RGB channels, the estimated parameters of the R channel, the estimated parameters of the G channel, and the estimated parameters of the B channel.
[0068] The RGB channel estimation parameters are used to construct the first-order parameters. Specifically, the first-order parameters include the R parameter, G parameter, and B parameter of the RGB channel estimation parameters.
[0069] The R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters are used to construct the second-order parameters. Specifically, the second-order parameters are composed of the R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters.
[0070] The first-order parameter is used to expand the difference between adjacent pixel values in the image to avoid image blur. The second-order parameter is used to characterize the pixel value mapping range. After the subsequent continuous iteration process, the model can improve its ability to adjust the brightness of the image. In this way, the number of network output channels is reduced, the complexity of the curve parameter estimation network structure is reduced, and the training efficiency is improved.
[0071] In one embodiment, the number of output channels of the curve parameter estimation network is set to 6. The first channel is used to output the estimated parameters of the R channel, the second channel is used to output the estimated parameters of the G channel, the third channel is used to output the estimated parameters of the B channel, the fourth channel is used to output the estimated parameters of the R channel, the fifth channel is used to output the estimated parameters of the G channel, and the sixth channel is used to output the estimated parameters of the B channel. In this case, the estimated parameters output by the first channel, the second channel, and the third channel are used as first-order parameters, and the estimated parameters output by the fourth channel, the fifth channel, and the sixth channel are used as second-order parameters.
[0072] In addition, before inputting the image to be processed into the curve parameter estimation network in step S101, the method provided by the embodiment of the present invention further includes:
[0073] Obtain low-light images and normalize them to normalize the data of low-light images to between [0,1], thereby eliminating the dimensional differences between image data to improve the performance and stability of the model.
[0074] Based on the normalized low-light image, augmentation processing is performed on it. The augmentation processing methods include: image rotation and image flipping. This increases the number and diversity of images and improves the generalization ability of the model.
[0075] In addition, the augmented image is downsampled by a certain multiple to obtain the image to be processed. The downsampling multiple can be selected from 1 to 300 according to the requirements of graphics card memory and time consumption in actual applications. The larger the downsampling multiple, the smaller the model time consumption and graphics card memory occupation, and the lower the performance. Downsampling the image reduces the image dimension and image size, thereby reducing the computational workload of the convolutional neural network and preventing the model from overfitting.
[0076] Step S102: acquiring a first enhanced image according to the first-order parameters and the image to be processed, and acquiring a second enhanced image according to the second-order parameters and the first enhanced image.
[0077] The steps of obtaining the first enhanced image according to the first-order parameters and the image to be processed in step S102 are as follows:
[0078] Step S1021, extracting R pixels, G pixels and B pixels of each pixel in the image to be processed.
[0079] Step S1022: Obtain a first enhanced R pixel according to the R parameter and the R pixel, obtain a first enhanced G pixel according to the G parameter and the G pixel, and obtain a first enhanced B pixel according to the B parameter and the B pixel.
[0080] Specifically, step S1022 is performed according to the following first-order enhancement formula.
[0081] in, is the first enhanced image, is the image to be processed, It is the first-order parameter diagram.
[0082] The above first-order enhancement formula can enlarge the pixel value difference of the image to be processed, which is equivalent to image sharpening, thereby improving the clarity of the blurred image caused by continuous iteration. For example, for a certain pixel point in the image to be processed, its pixel value is 3, and its adjacent pixel value is 4, then the difference between adjacent pixels is 1, and the first-order parameter corresponding to the pixel point is iteratively learned by the curve parameter estimation network, and the first-order parameter of the pixel point is 1.2. Then the first-order enhancement formula can be obtained:
[0083]
[0084] Then the difference between adjacent pixels of the image to be processed changes from 1 to 2.2, thereby widening the difference between adjacent pixels of the image to be processed.
[0085] Step S1023: Acquire a first enhanced image according to the first enhanced R pixel, the first enhanced G pixel, and the first enhanced B pixel of each pixel point.
[0086] In addition, according to the example in step S101, the first-order parameters include R parameters, G parameters and B parameters of RGB channel estimation parameters. Based on this, when obtaining the first enhanced image according to the first-order parameters and the image to be processed in step S102, the first-order parameter map mapped by the first-order parameters is first obtained based on the first-order parameters, and three copies of the first-order parameter map are copied, each of which includes R parameters, G parameters and B parameters. When processing according to an image enhancement formula, the R parameter in the first first-order parameter map is calculated with the R pixel of the R channel of the image to be processed; the G parameter in the second first-order parameter map is calculated with the G pixel of the G channel of the image to be processed; and the B parameter in the third first-order parameter map is calculated with the B pixel of the B channel of the image to be processed.
[0087] Since the first-order enhancement formula is a linear function, three copies of the first-order parameter map are copied and each copy is processed with the RGB channels of the image to be processed respectively, so that one set of parameters can be applied to achieve linear changes in the image to be processed, reducing the complexity of the calculation.
[0088] The first enhanced image is used as the basis for the second-order enhancement formula processing, and the first enhanced image is processed according to the second-order enhancement formula. The steps of obtaining the second enhanced image according to the second-order parameters and the first enhanced image in step S102 are as follows:
[0089] Second-order parameters are constructed according to the R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters, and a second-order parameter map mapped by the second-order parameters is obtained.
[0090] Step S1024, obtaining an R pixel difference, a G pixel difference, and a B pixel difference according to the difference between the square of the pixel and the pixel of each pixel point of the first enhanced image;
[0091] Step S1025, obtaining a second enhanced R pixel according to the R pixel difference and the R channel estimation parameters; obtaining a second enhanced G pixel according to the G pixel difference and the G channel estimation parameters; obtaining a second enhanced B pixel according to the B pixel difference and the B channel estimation parameters.
[0092] Specifically, step S1025 is performed according to the following second-order enhancement formula.
[0093] in, is the second enhanced image, is the first enhanced image, It is the second-order parameter diagram.
[0094] The second-order parameters include R channel estimation parameters, G channel estimation parameters, and B channel estimation parameters. Therefore, the second-order parameter map mapped by the second-order parameters includes R channel estimation parameter maps, G channel estimation parameter maps, and B channel estimation parameter maps. Thus, the R channel estimation parameter map is operated with the pixels of the R channel of the first enhanced image to obtain the second enhanced R pixel, the G channel estimation parameter map is operated with the pixels of the G channel of the first enhanced image to obtain the second enhanced G pixel, and the B channel estimation parameter map is operated with the pixels of the B channel of the first enhanced image to obtain the second enhanced B pixel.
[0095] Step S1026: Acquire a second enhanced image according to the second enhanced R pixel, the second enhanced G pixel, and the second enhanced B pixel of each pixel point.
[0096] After multiple iterations of the second-order enhancement formula, any high-order function can be fitted, thereby expanding the mapping range of pixel values, enhancing the adjustment ability of the model, and improving the generalization of the model for image processing in different brightness ranges.
[0097] The first-order parameter graph mapped by the first-order parameter and the parameter graph mapped by the second-order parameter are a curve graph. It is a curve fitted by the first-order parameter and the second-order parameter. Therefore, the curve must meet the following conditions:
[0098] 1. The value of each pixel in the second enhanced image is in the range of [0, 1].
[0099] 2. The curve is a monotonic curve, which is used to preserve the monotonicity of the pixels in the domain.
[0100] 3. The curve is simple and differentiable to ensure that backpropagation can be performed when the subsequent model performs iterative operations.
[0101] The first-order parameter map and the image to be processed are calculated according to the first-order enhancement formula, and the second-order parameter map and the first enhanced image are calculated according to the second-order enhancement formula. The first-order enhancement formula is added before the second-order enhancement formula to expand the difference between adjacent pixel values of the image, thereby avoiding the problem of image blurring caused by the adjacent pixel values becoming close to each other after the second-order enhancement formula in the subsequent multiple iterations. The image to be processed is processed according to the first-order enhancement formula, so that the second enhanced image obtained by the second-order enhancement formula does not have the problem of image blurring.
[0102] Step S103: construct a loss function according to the image to be processed and the second enhanced image, and iteratively update the curve parameter estimation network based on the loss function until a convergence condition is reached to obtain a low-light image enhancement model.
[0103] A loss function is constructed based on the image to be processed and the second enhanced image. The loss function includes:
[0104] The image color ratio consistency loss function is used to characterize the color ratio consistency between the second enhanced image and the image to be processed. The image color ratio consistency loss function is:
[0105]
[0106] Among them, q and p are any channels in RGB channels, and p and q channels are different.
[0107] represents the average intensity value of the second enhanced image p channel,
[0108] represents the average intensity value of the q channel of the second enhanced image,
[0109] Represents the average intensity value of the p channel of the image to be processed,
[0110] Indicates the average intensity value of the q channel of the image to be processed.
[0111] If the images to be processed are obtained under unnatural light or in unnatural environments, these images to be processed do not completely conform to the law of color constants. Therefore, when only the subsequent color constant loss function is used, the model learning will be updated in the direction of wrong parameters. Therefore, an image color ratio consistency loss function is added to constrain the color ratio of the image to be processed and the second enhanced image so that their color ratios are consistent.
[0112] For example, in warehousing and circulation scenarios, cartons of the same product are stacked together, and these cartons have large areas of red. The camera obtains the image to be processed, and the overall color of the image is biased towards red. Through the image color ratio consistency loss function, the enhanced image is kept consistent with the image to be processed, preventing the model from continuously enhancing red during the learning process, resulting in updating incorrect parameters and obtaining a color-biased image.
[0113] The spatial consistency loss function is used to make the neighborhood of the second enhanced image and the image to be processed consistent. The image color ratio consistency loss function is:
[0114]
[0115] Where K is the total number of patches in the image, Ω is a 4x4 patch, Y is the second enhanced image, and I is the image to be processed.
[0116] The spatial consistency loss function makes the difference in the average intensity values of adjacent patches of the second enhanced image consistent with the difference in the average intensity values of adjacent patches of the image to be processed.
[0117] The loss function of the exposure level is used to characterize the exposure degree of the second enhanced image. The loss function of the exposure level is:
[0118]
[0119] Where M is a 16x16 non-overlapping local area and E is the brightness level.
[0120] The image to be processed is a low-light image that has been normalized so that the brightness value of the image to be processed is between [0, 1]. Therefore, the brightness level range is [0, 1]. Therefore, the brightness level E is set to 0.6 to prevent the second enhanced image from being overexposed or underexposed.
[0121] The exposure level loss function is used to ensure that the image is overexposed or underexposed by controlling the exposure level.
[0122] The color constant loss function is used to characterize the color deviation of the second enhanced image. The color constant loss function is:
[0123]
[0124] in, represents the average intensity value of the second enhanced image p channel, It represents the average intensity value of the q channel of the second enhanced image, where p and q are any one of the R channel, G channel and B channel in the RGB channel, and the q channel and the p channel are different.
[0125] The loss function of the color constant is used to correct the overall deviation of the second enhanced image.
[0126] Brightness monotonicity loss function is used to characterize the monotonicity of adjacent pixels of the second enhanced image. Brightness monotonicity loss function is:
[0127]
[0128] Where N represents the number of iterations, and and represent the gradients of the first-order parameter map and the second-order parameter map along the x and y directions, Represents the parameter diagram of different channels at the corresponding number of iterations.
[0129] The brightness monotonicity loss function ensures the monotonicity of adjacent pixels by making the gradients in the x and y directions of the second-order parameter map as small as possible.
[0130] Based on the above loss function, the curve parameter estimation network is iteratively updated until the convergence condition is reached, and a low-light image enhancement model is obtained. Based on the model, the low-light image can be enhanced to obtain an enhanced image.
[0131] In summary, the embodiment of the present application provides a method for constructing a low-light image enhancement model. The low-light image is preprocessed to obtain the image to be processed, and the image to be processed is input into the curve parameter estimation network. The output result is used to construct the first-order parameters and the second-order parameters, and the first-order parameter map mapped by the first-order parameters and the second-order parameter map mapped by the second-order parameters are obtained. According to the first-order parameter map and the first-order enhancement formula, the pixel values between adjacent pixels of the image to be processed are enlarged, thereby ensuring the clarity of the enhanced image. According to the second-order parameter map and the second-order enhancement formula, the enhanced image is obtained, and the image to be processed is made consistent with the enhanced image color through the image color ratio consistency loss function, and the spatial consistency loss function, the exposure level loss function, the color constant loss function, and the brightness monotonicity loss function, and the parameters of the curve parameter network are continuously iterated and updated until the convergence condition is met to obtain a low-light image enhancement model. , thereby improving the clarity of the enhanced image and maintaining the color consistency of the enhanced image, thereby improving the image quality after image enhancement.
[0132] In a second aspect, an embodiment of the present application provides a low-light image enhancement method. Figure 3 FIG. 1 is a flow chart of a low-light image enhancement method according to an exemplary embodiment. Figure 3 As shown, the method is applied to the low-light image enhancement model obtained by the low-light image enhancement model construction method in the first aspect.
[0133] Step S201: Obtain an image to be processed.
[0134] Step S202: input the image to be processed into a low-light image enhancement model, and obtain the RGB channel change parameters of each pixel based on the curve parameter estimation network in the low-light image enhancement model.
[0135] Step S203: Enhance the image to be processed according to the RGB channel variation parameters to obtain an enhanced image.
[0136] After preprocessing, the low-light image is obtained to obtain the image to be processed, and the image to be processed is input into the low-light enhancement model. The change parameters of the RGB channels of each pixel are obtained in the curve parameter estimation network. Based on the RGB channel change parameters, the image to be processed is processed according to the first-order enhancement formula and the second-order enhancement formula in the model to obtain an enhanced image.
[0137] After the low-light image is enhanced by the low-light image enhancement model, the enhanced image with high clarity and no color cast can be obtained, and the quality of the enhanced low-light image can be guaranteed.
[0138] In a third aspect, an embodiment of the present application provides a low-light image enhancement model construction device. Figure 4 FIG. 1 is a block diagram of a low-light image enhancement model building device according to an exemplary embodiment. Figure 4 As shown, the low-light image enhancement model building device includes:
[0139] The construction module is used to input the image to be processed into the curve parameter estimation network, and construct the first-order parameters and second-order parameters according to the output results. The curve parameter estimation network is a convolutional neural network, which is used to output the RGB channel change parameters of each pixel point of the image.
[0140] The first acquisition module is used to acquire a first enhanced image according to the first-order parameters and the image to be processed, and to acquire a second enhanced image according to the second-order parameters and the first enhanced image.
[0141] The second acquisition module constructs a loss function according to the image to be processed and the second enhanced image, and iteratively updates the curve parameter estimation network based on the loss function until the convergence condition is reached, thereby obtaining a low-light image enhancement model.
[0142] In summary, the low-light image enhancement model device provided by the present application, in the construction module, inputs the image to be processed into the curve parameter estimation network, and constructs the first-order parameters and the second-order parameters according to the output results. In the first acquisition module, the first-order parameters and the image to be processed are used to obtain the first enhanced image according to the first-order image enhancement formula; the second-order parameters and the first enhanced image are used to obtain the second enhanced image according to the second-order image enhancement formula. In the second acquisition module, a loss function is constructed according to the image to be processed and the second enhanced image, and the curve parameter estimation network is iteratively updated based on the loss function until the convergence condition is reached to obtain the low-light image enhancement model. Based on this model, the low-light image can be enhanced, thereby improving the problem of poor quality of the enhanced low-light image.
[0143] It should be noted that the low-light image enhancement model device provided in this embodiment is used to implement the above-mentioned implementation mode, and the descriptions that have been made will not be repeated. As used above, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for predetermined functions. Although the device described in the above embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0144] In a fourth aspect, an embodiment of the present application provides an electronic device, Figure 5 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device may include a processor 81 and a memory 82 storing computer program instructions.
[0145] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0146] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.
[0147] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .
[0148] The processor 81 implements any one of the low-light image enhancement model methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .
[0149] In one embodiment, the low-light image enhancement model device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.
[0150] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.
[0151] The bus 80 includes hardware, software, or both, and couples the components of the low-light image enhancement model device to each other. The bus 80 includes, but is not limited to, at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0152] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the low-light image enhancement model method provided in the first aspect is implemented.
[0153] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.
[0154] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the low-light image enhancement model method provided in the first aspect.
[0155] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.
[0156] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0157] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for constructing a low-light image enhancement model, characterized in that: include: Inputting the image to be processed into the curve parameter estimation network to obtain an output result, wherein the output result includes RGB channel estimation parameters, R channel estimation parameters, G channel estimation parameters, and B channel estimation parameters, the curve parameter estimation network is a convolutional neural network, which is used to output RGB channel variation parameters of each pixel point of the image, and the curve parameter estimation network includes four output channels, including a first channel for outputting estimation parameters of RGB channels, a second channel for outputting estimation parameters of the R channel, a third channel for outputting estimation parameters of the G channel, and a fourth channel for outputting estimation parameters of the B channel; Constructing first-order parameters according to the RGB channel estimation parameters, wherein the first-order parameters include R parameters, G parameters and B parameters of the RGB channel estimation parameters; Constructing second-order parameters according to the R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters; Acquire a first enhanced image according to the first-order parameters and the image to be processed, and acquire a second enhanced image according to the second-order parameters and the first enhanced image; Constructing a loss function according to the image to be processed and the second enhanced image, iteratively updating the curve parameter estimation network based on the loss function until a convergence condition is reached, and obtaining a low-light image enhancement model, wherein the loss function includes a loss function of image color ratio consistency; The loss function of the image color ratio consistency is configured as: Among them, q and p are any channels in RGB channels, and p and q channels are different. represents the average intensity value of the p channel of the second enhanced image, represents the average intensity value of the q channel of the second enhanced image, represents the average intensity value of the p channel of the image to be processed, Represents the average intensity value of the q channel of the image to be processed.
2. The method for constructing a low-light image enhancement model according to claim 1, characterized in that: The step of acquiring a first enhanced image according to the first-order parameter and the image to be processed includes: Extracting R pixels, G pixels and B pixels of each pixel in the image to be processed; Acquire a first enhanced R pixel according to the R parameter and the R pixel; Acquire a first enhanced G pixel according to the G parameter and the G pixel; Acquire a first enhanced B pixel according to the B parameter and the B pixel; A first enhanced image is acquired according to the first enhanced R pixel, the first enhanced G pixel, and the first enhanced B pixel of each pixel point.
3. The method for constructing a low-light image enhancement model according to claim 2, characterized in that: The step of acquiring a second enhanced image according to the second-order parameter and the first enhanced image comprises: Obtaining an R pixel difference, a G pixel difference, and a B pixel difference according to the difference between the square of the pixel and the pixel of each pixel point of the first enhanced image; Acquire a second enhanced R pixel according to the R pixel difference and the R channel estimation parameter; According to the G pixel difference and the G channel estimation parameter, a second enhanced G pixel is obtained. According to the B pixel difference and the B channel estimation parameter, a second enhanced B pixel is obtained. A second enhanced image is acquired according to the second enhanced R pixel, the second enhanced G pixel and the second enhanced B pixel of each pixel point.
4. According to the method for constructing a low-light image enhancement model according to claim 1, the loss function comprises: An image color ratio consistency loss function, used to characterize the color ratio consistency between the second enhanced image and the image to be processed; A spatial consistency loss function, used to make the neighborhood of the second enhanced image and the image to be processed consistent; an exposure level loss function, used to characterize the exposure degree of the second enhanced image; A color constant loss function, used to characterize the color deviation of the second enhanced image; A brightness monotonicity loss function is used to characterize the monotonicity of adjacent pixels of the second enhanced image.
5. The method for constructing a low-light image enhancement model according to claim 1, characterized in that: Before the image to be processed is input into the curve parameter estimation network, the method further includes: A low-light image is acquired, and normalization processing is performed on the low-light image to obtain the image to be processed.
6. A low-light image enhancement method, characterized in that: The method is applied to the low-light image enhancement model obtained by the construction method according to any one of claims 1 to 5, and the method comprises: Get the image to be processed; Inputting the image to be processed into the low-light image enhancement model, and obtaining the RGB channel change parameter of each pixel based on the curve parameter estimation network in the low-light image enhancement model; The image to be processed is enhanced according to the RGB channel change parameters to obtain an enhanced image.
7. A low-light image enhancement model construction device, characterized in that: include: A first acquisition module is used to input the image to be processed into a curve parameter estimation network to obtain an output result, wherein the output result includes RGB channel estimation parameters, R channel estimation parameters, G channel estimation parameters, and B channel estimation parameters, and the curve parameter estimation network is a convolutional neural network, which is used to output RGB channel variation parameters of each pixel point of the image, and the curve parameter estimation network includes four output channels, including a first channel for outputting estimation parameters of RGB channels, a second channel for outputting estimation parameters of the R channel, a third channel for outputting estimation parameters of the G channel, and a fourth channel for outputting estimation parameters of the B channel; A first construction module is used to construct first-order parameters according to the RGB channel estimation parameters, wherein the first-order parameters include R parameters, G parameters and B parameters of the RGB channel estimation parameters; A second construction module is used to construct second-order parameters according to the R channel estimation parameters, the G channel estimation parameters, and the B channel estimation parameters; A second acquisition module, used for acquiring a first enhanced image according to the first-order parameters and the image to be processed, and acquiring a second enhanced image according to the second-order parameters and the first enhanced image; The third acquisition module constructs a loss function according to the image to be processed and the second enhanced image, iteratively updates the curve parameter estimation network based on the loss function until a convergence condition is reached, and obtains a low-light image enhancement model, wherein the loss function includes a loss function of image color ratio consistency; The loss function of the image color ratio consistency is configured as: Among them, q and p are any channels in RGB channels, and p and q channels are different. represents the average intensity value of the p channel of the second enhanced image, represents the average intensity value of the q channel of the second enhanced image, represents the average intensity value of the p channel of the image to be processed, Represents the average intensity value of the q channel of the image to be processed.
8. An electronic device, characterized in that: including memory, processor, and A computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for constructing a low-light image enhancement model as described in any one of claims 1 to 6 is implemented.
