Image enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides an image enhancement method and device, electronic equipment and a storage medium. The method comprises the steps of determining a target image according to color information and brightness information of a to-be-enhanced image; inputting the target image into an image enhancement model to determine a residual image between the to-be-enhanced image and the corresponding highlight image based on the brightness information and the color information; the residual image is used for representing the enhancement degree of the to-be-enhanced image; and superposing the to-be-enhanced image and the residual image to obtain a highlight image corresponding to the to-be-enhanced image. The image can be enhanced based on the color information and the brightness information of the to-be-enhanced image, and the image enhancement effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image enhancement method, device, electronic equipment and storage medium. Background Art

[0002] Image enhancement refers to the process of improving the visibility, contrast, and overall quality of captured images through various techniques and algorithms. Image enhancement methods are mainly based on traditional image processing techniques such as histogram equalization and contrast stretching.

[0003] However, traditional image processing techniques have poor image enhancement effects. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide an image enhancement method, device, electronic device and storage medium, which can enhance the image based on the color information and brightness information of the image to be enhanced, thereby improving the image enhancement effect.

[0005] In a first aspect, an embodiment of the present application provides an image enhancement method, the image enhancement method comprising:

[0006] Determine the target image according to the color information and brightness information of the image to be enhanced;

[0007] The target image is input into the image enhancement model to determine the residual image between the image to be enhanced and the corresponding highlight image based on the brightness information and the color information; the residual image is used to characterize the enhancement degree of the image to be enhanced;

[0008] The image to be enhanced is superimposed with the residual image to obtain a highlight image corresponding to the image to be enhanced.

[0009] In a possible implementation, determining the target image according to the color information and brightness information of the image to be enhanced includes:

[0010] Grayscale the image to be enhanced to obtain a grayscale image;

[0011] Invert the grayscale image to obtain a target grayscale image containing brightness information;

[0012] The image to be enhanced and the target grayscale image are merged according to the dimension to obtain the target image; the image to be enhanced contains color information.

[0013] In a possible implementation, the image enhancement model includes an encoder and a decoder; inputting a target image into the image enhancement model to determine a residual image between the image to be enhanced and a corresponding highlight image based on brightness information and color information, including:

[0014] The target image is input into the encoder to obtain an image feature matrix based on brightness information and color information; the image feature matrix integrates the global information and local image of the target image; the eigenvalues ​​in the image feature matrix are used to characterize the enhancement degree of the image to be enhanced;

[0015] The image feature matrix is ​​input into the decoder to obtain the residual image.

[0016] In a possible implementation, the encoder includes a multi-head self-attention neural network unit and a residual convolution network unit; the target image is input into the encoder to obtain an image feature matrix based on brightness information and color information, including:

[0017] Inputting the target image into the multi-head self-attention neural network unit to determine the feature value of each pixel in the first image feature matrix based on the brightness information and color information of each pixel in the target image;

[0018] Inputting the target image into the residual convolution network unit to determine the feature value of each pixel in the local area corresponding to each local image in the second image feature matrix based on the brightness information and color information of all pixels in each local image in the target image;

[0019] The first image feature matrix and the second image feature matrix are subjected to feature fusion to obtain an image feature matrix.

[0020] In one possible implementation, the image enhancement model is trained by the following steps:

[0021] Determine the target image sample according to the color information and brightness information of the image sample;

[0022] Input the target image sample into the image enhancement model to obtain the residual image between the image sample and the corresponding highlight image;

[0023] Superimposing the residual image between the image sample and the corresponding highlight image and the image sample to obtain the highlight image corresponding to the image sample;

[0024] Calculate the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample;

[0025] The model parameters of the image enhancement model are optimized based on the loss value until the image enhancement model meets the preset training completion conditions.

[0026] In a possible implementation, calculating the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample includes:

[0027] Substitute the real highlight image and the highlight image corresponding to the image sample into the following formula to obtain the loss value of the image enhancement model;

[0028]

[0029] Among them, Loss is the loss value of the image enhancement model, X GT is the highlight image corresponding to the image sample, G(X) is the true highlight image, X is the image sample, and ∈ is a positive number.

[0030] In a second aspect, an embodiment of the present application further provides an image enhancement device, the device comprising:

[0031] A determination module, used to determine a target image according to color information and brightness information of the image to be enhanced;

[0032] An input module, used to input a target image into an image enhancement model, so as to determine a residual image between the image to be enhanced and a corresponding highlight image based on brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced;

[0033] The superposition module is used to superimpose the image to be enhanced with the residual image to obtain a highlight image corresponding to the image to be enhanced.

[0034] In a possible implementation, the determination module is specifically used to grayscale the image to be enhanced to obtain a grayscale image; invert the grayscale image to obtain a target grayscale image containing brightness information; merge the image to be enhanced and the target grayscale image according to the dimension to obtain the target image; and the image to be enhanced contains color information.

[0035] In a possible implementation, the determination module is specifically used to grayscale the image to be enhanced to obtain a grayscale image; invert the grayscale image to obtain a target grayscale image containing brightness information; merge the image to be enhanced and the target grayscale image according to the dimension to obtain the target image; and the image to be enhanced contains color information.

[0036] In one possible implementation, the image enhancement model includes an encoder and a decoder; an input module, specifically used to input a target image into the encoder to obtain an image feature matrix based on brightness information and color information; the image feature matrix integrates the global information and local image of the target image; the eigenvalues ​​in the image feature matrix are used to characterize the degree of enhancement of the image to be enhanced; and the image feature matrix is ​​input into the decoder to obtain a residual image.

[0037] In one possible implementation, the encoder includes a multi-head self-attention neural network unit and a residual convolution network unit; an input module is specifically used to input a target image into the multi-head self-attention neural network unit to determine the eigenvalue of each pixel in a first image feature matrix based on the brightness information and color information of each pixel in the target image; the target image is input into the residual convolution network unit to determine the eigenvalue of each pixel in a local area corresponding to each local image in a second image feature matrix based on the brightness information and color information of all pixels in each local image in the target image; and the first image feature matrix and the second image feature matrix are feature-fused to obtain an image feature matrix.

[0038] In a possible implementation, the input module is further used for:

[0039] Determine the target image sample according to the color information and brightness information of the image sample;

[0040] Input the target image sample into the image enhancement model to obtain the residual image between the image sample and the corresponding highlight image;

[0041] Superimposing the residual image between the image sample and the corresponding highlight image and the image sample to obtain the highlight image corresponding to the image sample;

[0042] Calculate the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample;

[0043] The model parameters of the image enhancement model are optimized based on the loss value until the image enhancement model meets the preset training completion conditions.

[0044] In a possible implementation, the input module is specifically used to substitute the real highlight image and the highlight image corresponding to the image sample into the following formula to obtain the loss value of the image enhancement model;

[0045]

[0046] Among them, Loss is the loss value of the image enhancement model, X GT is the highlight image corresponding to the image sample, G(X) is the true highlight image, X is the image sample, and ∈ is a positive number.

[0047] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of any image enhancement method of the first aspect.

[0048] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any image enhancement method in the first aspect are executed.

[0049] The embodiment of the present application provides an image enhancement method, device, electronic device and storage medium, the method comprising: determining a target image according to the color information and brightness information of the image to be enhanced; inputting the target image into an image enhancement model to determine a residual image between the image to be enhanced and the corresponding highlight image based on the brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced; the image to be enhanced is superimposed with the residual image to obtain a highlight image corresponding to the image to be enhanced. The present application can enhance an image based on the color information and brightness information of the image to be enhanced, thereby improving the image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of an image enhancement method provided by an embodiment of the present application is shown;

[0052] Figure 2 A schematic diagram of the structure of an image enhancement model provided in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of the process of training an image enhancement model provided in an embodiment of the present application is shown;

[0054] Figure 4 A schematic structural diagram of an image enhancement device provided in an embodiment of the present application is shown;

[0055] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0057] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0058] In order to enable those skilled in the art to use the contents of this application, the following implementation is provided in conjunction with a specific application scenario, the "field of image processing technology". For those skilled in the art, the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the "field of image processing technology", it should be understood that this is only an exemplary embodiment.

[0059] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0060] An image enhancement method provided in an embodiment of the present application is described in detail below.

[0061] Reference Figure 1 As shown, it is a flowchart of an image enhancement method provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0062] S101: Determine a target image according to color information and brightness information of an image to be enhanced.

[0063] In the implementation manner of the present application, the image to be enhanced refers to a three-channel RGB image that needs to improve visibility, contrast and overall quality, such as a dark light image (also called a low-light image, which refers to an image taken in an environment with insufficient light). RGB images can provide rich color information, specifically the color information of the three channels of red (R), green (G) and blue (B) in the image to be enhanced. Brightness information refers to the degree of lightness and darkness in the image to be enhanced.

[0064] In addition, the smaller the brightness of each pixel in the image to be enhanced, the greater the brightness of each pixel in the target image, thereby introducing the frequency domain information of the image to be enhanced into the image enhancement model, emphasizing the low-brightness areas in the image to be enhanced and suppressing the high-brightness areas, which helps the model to focus more flexibly on the different frequency components of the image, improves the perception ability of the image enhancement model, and makes the obtained residual image more accurate.

[0065] Specifically, the target image is determined by the following steps:

[0066] Step 1: grayscale the image to be enhanced to obtain a grayscale image.

[0067] In the implementation manner of the present application, the grayscale image is a single-channel representation of the image to be enhanced; the grayscale image Gray is obtained by weighted averaging the three color channels (red, green and blue) in the image to be enhanced, and the specific formula is:

[0068] Gray=0.2989×R+0.5870×G+0.1140×B.

[0069] Here, the grayscale image retains the brightness information of the image to be enhanced but loses the color information. It is usually used to capture the structure, texture, and brightness distribution of the image to be enhanced.

[0070] Step 2: Invert the grayscale image to obtain a target grayscale image containing brightness information.

[0071] In the implementation manner of the present application, negating a grayscale image means converting the value of each pixel in the grayscale image to the complement of its maximum possible value. In an 8-bit grayscale image, the value of each pixel ranges from 0 (black) to 255 (white), and the negation operation converts the value p of each pixel into 255-p.

[0072] Here, by inverting the grayscale image, the image enhancement model can focus more on areas with low brightness and suppress areas with high brightness, which helps the model to focus more flexibly on different frequency components of the image, improves the perception ability of the image enhancement model, and makes the resulting residual image more accurate.

[0073] Step 3: Merge the image to be enhanced and the target grayscale image according to the dimension to obtain the target image; the image to be enhanced contains color information.

[0074] In an embodiment of the present application, the three channel information (color information) of the image to be enhanced and the single channel information (brightness information) of the target grayscale image are merged according to the dimensions to form a target image containing four channels.

[0075] For example, the dimension of the image to be enhanced is (H, W, 3)(H, W, 3), where H and W are the height and width of the image to be enhanced respectively. The dimension of the grayscale image is (H, W, 1)(H, W, 1), that is, the grayscale image is a single-channel image. After merging the two, a new four-channel image (i.e., the target image) is obtained with the dimension of (H, W, 4)(H, W, 4).

[0076] S102: Input the target image into the image enhancement model to determine a residual image between the image to be enhanced and the corresponding highlight image based on brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced.

[0077] In the implementation mode of the present application, the image enhancement model is used to generate a residual image. Compared with the existing deep learning model that directly generates a highlight image, the image enhancement model of the present application reduces complexity, improves generalization, facilitates model convergence, shortens model training time, and improves image enhancement effect. The image enhancement model is obtained by training image samples and corresponding highlight images, and includes an encoder and a decoder. Figure 2 , which is a schematic diagram of the structure of the image enhancement model provided in an embodiment of the present application.

[0078] Specifically, the target image is input into the image enhancement model through the following steps to determine the residual image between the image to be enhanced and the corresponding highlight image:

[0079] Step 1: Input the target image into the encoder to obtain an image feature matrix based on brightness information and color information; the image feature matrix integrates the global information and local image of the target image; the eigenvalues ​​in the image feature matrix are used to characterize the enhancement degree of the image to be enhanced.

[0080] In an embodiment of the present application, the encoder includes a multi-head self-attention neural network unit and a residual convolution network unit.

[0081] Specifically, the target image is input into the encoder through the following steps to obtain the image feature matrix:

[0082] a. Input the target image into a multi-head self-attention neural network unit to determine the feature value of each pixel in the first image feature matrix based on the brightness information and color information of each pixel in the target image.

[0083] In an embodiment of the present application, the multi-head self-attention neural network unit includes multiple self-attention heads; each self-attention head corresponds to its own weight matrix to allow the image enhancement model to learn multiple different feature representations at the same time.

[0084] Specifically, the target image is projected into each attention head in the multi-head self-attention neural network unit through linear mapping; for each self-attention head, the eigenvalue (four-dimensional eigenvalue, including color information and brightness information) of each pixel in the target image is converted into a query vector, a key vector and a value vector through the embedding layer of the self-attention head; the value vector of each pixel is weighted based on the query vector and the key vector of each pixel to obtain the weighted feature of each pixel; all the weighted features of each pixel are concatenated to obtain the eigenvalue of each pixel in the first image feature matrix.

[0085] Among them, Query is used to represent the position and information (color information and brightness information) of the pixel, which determines the relationship between the pixel and other pixels. Key represents the degree of association between the pixel and other pixels, which determines the importance of the pixel. Value represents the specific information of the pixel, which is the output after calculating the attention weight.

[0086] Specifically, the value vector of each pixel is weighted based on the query vector and key vector of each pixel to obtain the weighted features of each pixel, including:

[0087]

[0088] Among them, Q is the query vector of each pixel, K is the key vector of each pixel, V is the value vector of each pixel, Attention (Q, K, V) is the weighted feature of each pixel, and the softmax function is used to ensure is normalized, is the scaling factor.

[0089] Furthermore, the multi-head self-attention neural network unit also includes: Layer normalization and residual connection: After each self-attention head calculation is completed, a residual connection (skip connection) is usually added, and layer normalization (LayerNormalization) is performed to improve the stability of training. Feed-Forward Neural Network (FFN): After the multi-head self-attention layer, there is usually a feed-forward neural network (usually a two-layer fully connected network) for further processing to enhance the nonlinear ability of the model. Back propagation and parameter update: After multiple self-attention calculations and feed-forward network processing, the model updates the weights through the back propagation algorithm to minimize the loss function, thereby optimizing the model.

[0090] Here, the application of multi-head self-attention in image enhancement is mainly reflected in the global correlation modeling of image features and the weighting of the importance of different spatial positions. This attention mechanism helps to capture long-range dependencies in the image, and at the same time can integrate information from different regions, which helps to improve the effect of low-light image enhancement. The following is the role of multi-head self-attention in low-light image enhancement: 1. Global correlation modeling: Images usually contain a lot of low-frequency information and global structure. Multi-head self-attention allows the model to learn global correlation in different heads, so as to better capture the long-range dependencies in the image, which helps to enhance the overall brightness and contrast. 2. Spatial position attention: Information in different regions of the image may have different effects on enhancing images of different brightness. Multi-head self-attention can focus on different regions in spatial position, so that the model can more specifically adjust the brightness and details of the corresponding regions, thereby improving the visual effect of the image. 3. Fine-grained feature integration: Multi-head self-attention allows the model to focus on fine-grained features of different spatial positions at the same time, which helps to retain more image details. By integrating the attention outputs of different heads, the model can more comprehensively understand the content of the image and enhance it. 4. Fighting low-light noise: Images are usually accompanied by more noise. Multi-head self-attention helps model the noise in the image and can suppress or reduce the impact of noise during the enhancement process, improving image quality. 5. Multi-scale information integration: The multi-head self-attention mechanism allows the model to focus on features in the image at different scales, which helps integrate multi-scale information. This is very important for image enhancement because in low-light conditions, it may be necessary to deal with brightness changes and details at different scales. 6. Nonlinear modeling: The multi-head self-attention mechanism introduces nonlinear transformations, which helps to more flexibly model the complex relationship between image features. This is crucial for generating realistic images in low-light conditions. 7. Conditional generation: The multi-head self-attention mechanism can easily combine conditional information, such as lighting conditions or shooting environment, so that the generation network is more flexible and adaptable to generate images that better meet specific conditions. In summary, as a powerful attention mechanism, multi-head self-attention can help the model better understand the image content, capture global correlation and detail information in image enhancement, thereby effectively improving the visual effect of the image and making the image more visual.

[0091] b. Input the target image into the residual convolution network unit to determine the feature value of each pixel in the local area corresponding to each local image in the second image feature matrix based on the brightness information and color information of all pixels in each local image in the target image.

[0092] In the embodiment of the present application, nine residual convolutional network blocks are connected in the residual convolutional network unit. The addition of residual blocks alleviates the disadvantages of using only conventional convolutional networks, namely, many parameters and high network complexity, which makes training difficult, and the problem of gradient vanishing is prone to occur in the back propagation of gradient descent, making it difficult to continue training. The local jump line of the residual block allows the gradient to be directly transmitted back to the previous structure without blocking.

[0093] c. Fusing the first image feature matrix and the second image feature matrix to obtain an image feature matrix.

[0094] In an embodiment of the present application, the image feature matrix with the lowest dimension in the first image feature matrix and the second image feature matrix is ​​upsampled so that the dimensions of the first image feature matrix and the second image feature matrix are the same; the upsampled first image feature matrix and the second image feature matrix are feature fused by splicing, weighted summation, etc. to obtain an image feature matrix.

[0095] Optionally, the image feature matrix is ​​reduced in dimensionality after fusion to reduce computational complexity or mitigate the risk of overfitting.

[0096] Here, the fusion of multi-head self-attention and convolutional neural network in the image enhancement task can significantly improve the model's processing effect on images under low light conditions. The following is the role of their fusion in image enhancement: 1. Global correlation and local feature integration: Multi-head self-attention helps capture the global correlation of images, while convolutional neural network is good at extracting local features. Through fusion, global and local information can be integrated more comprehensively, so that the model can better understand the structure of the image and be more sensitive to details and overall information in low-light environments. 2. Multi-scale information fusion: Convolutional neural network processes images through multi-scale convolution kernels, and multi-head self-attention can focus on features at different spatial locations. Combining the two helps integrate information at different scales and improve the perception of multi-scale features in low-light images, so as to better cope with challenges in low-light environments. 3. Long-distance dependency modeling: Multi-head self-attention performs well in capturing long-distance dependencies, while convolutional neural network can also effectively process local relationships of images. In image enhancement tasks, this fusion helps to better model long-distance dependencies in images and improve the model's ability to understand the overall image. 4. Low-light noise suppression: The combination of multi-head self-attention and convolutional neural network helps to suppress noise in images under low-light conditions. By jointly processing the global structure and local texture of the image, the model can more effectively suppress the interference of noise on the enhancement results.

[0097] Step 2: Input the image feature matrix into the decoder to obtain the residual image.

[0098] In the implementation manner of the present application, the decoder includes a first deconvolution layer, a second deconvolution layer, a first DBR module, and a second DBR module, wherein both the first DBR module and the second DBR module include a deconvolution unit, a normalization unit, and an activation function unit.

[0099] Specifically, the image feature matrix is ​​input into the first deconvolution layer to obtain the first deconvolution feature; the first deconvolution feature is input into the second deconvolution layer to obtain the second deconvolution feature; the second deconvolution feature is input into the first DBR module to obtain the DBR feature; the first DBR feature is input into the second DBR module to obtain the residual image.

[0100] Furthermore, the second deconvolution feature is input into the first DBR module to obtain the DBR feature, including: inputting the second deconvolution feature into the deconvolution unit in the first DBR module to obtain the third deconvolution feature; inputting the third deconvolution feature into the normalization unit in the first DBR module to obtain the first normalized feature; and inputting the first normalized feature into the activation function unit in the first DBR module to obtain the DBR feature.

[0101] Furthermore, the first DBR feature is input into the second DBR module to obtain a residual image, including: inputting the first DBR feature into the deconvolution unit in the second DBR module to obtain a fourth deconvolution feature; inputting the fourth deconvolution feature into the normalization unit in the second DBR module to obtain a second normalized feature; and inputting the second normalized feature into the activation function unit in the second DBR module to obtain a residual image.

[0102] Here, the decoder is mainly responsible for decoding the potential image feature matrix into a residual image. The specific functions include the following aspects: 1. Feature restoration: The main task of the decoder is to gradually restore the potential high-level features to the details and structure of the original image. The convolution layer gradually maps the abstract feature representation back to the original resolution of the image through deconvolution (also known as transposed convolution or upsampling) operations. 2. Spatial position restoration: In image generation, the decoder gradually restores the spatial position information in the original image through the upsampling operation of the convolution layer. This helps to ensure that the generated image is consistent with the real image at the pixel level. 3. Feature integration: The convolution layer of the decoder can integrate global and local feature information to restore the details and structure in the image. Through multi-level convolution operations, the decoder can gradually merge features at different levels of abstraction to make the generated image richer and more realistic. 4. Generate image structure: The decoder can generate the details, texture and structure of the image through convolution operations. Each convolution layer can be regarded as a stage of a generator, which is responsible for generating part of the information in the image, such as edges, textures, etc. 5. Learning image representations: During the training of the decoder, the weights of the convolutional layers are adjusted so that the network can learn meaningful representations in the input latent space. These representations can be mapped back to the image space during the generation process, thereby generating high-quality images.

[0103] S103: Superimpose the image to be enhanced and the residual image to obtain a highlight image corresponding to the image to be enhanced.

[0104] The present application provides an image enhancement method, which includes: determining a target image according to the color information and brightness information of an image to be enhanced; inputting the target image into an image enhancement model to determine a residual image between the image to be enhanced and a corresponding highlight image based on the brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced; and the image to be enhanced and the residual image are superimposed to obtain a highlight image corresponding to the image to be enhanced. The present application can enhance an image based on the color information and brightness information of the image to be enhanced, thereby improving the image enhancement effect.

[0105] Reference Figure 3 As shown, it is a flow chart of the training image enhancement model provided in the embodiment of the present application. The exemplary steps of the embodiment of the present application are described below:

[0106] S301 : Determine a target image sample according to color information and brightness information of the image sample.

[0107] Specifically, the image samples are grayed to obtain grayscale image samples; the grayscale image samples are inverted to obtain target grayscale image samples containing brightness information; the image samples and the target grayscale image samples are merged according to the dimension to obtain the target image samples.

[0108] S302: Input the image sample into the image enhancement model to obtain a residual image between the image sample and the corresponding highlight image.

[0109] S303: Superimpose the residual image between the image sample and the corresponding highlight image and the image sample to obtain the highlight image corresponding to the image sample.

[0110] S304: Calculate the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample.

[0111] Specifically, the real highlight image and the highlight image corresponding to the image sample are substituted into the following formula to obtain the loss value of the image enhancement model;

[0112]

[0113] Among them, Loss is the loss value of the image enhancement model, X GT is the highlight image corresponding to the image sample, G(X) is the true highlight image, X is the image sample, and ∈ is a positive number.

[0114] S305. Optimize the model parameters of the image enhancement model based on the loss value until the image enhancement model meets the preset training completion condition.

[0115] In an implementation manner of the present application, the preset training completion condition may be that the loss value of the image enhancement model is less than a preset loss value or the number of iterations of the image enhancement model reaches a preset number.

[0116] Based on the same inventive concept, an image enhancement device corresponding to the image enhancement method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned image enhancement method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0117] Reference Figure 4 FIG. 1 is a schematic diagram of an image enhancement device provided in an embodiment of the present application, wherein the image enhancement device comprises:

[0118] A determination module 401 is used to determine a target image according to color information and brightness information of the image to be enhanced;

[0119] An input module 402 is used to input a target image into an image enhancement model to determine a residual image between the image to be enhanced and a corresponding highlight image based on brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced;

[0120] The superposition module 403 is used to superimpose the image to be enhanced and the residual image to obtain a highlight image corresponding to the image to be enhanced.

[0121] In a possible implementation, the determination module 401 is specifically used to grayscale the image to be enhanced to obtain a grayscale image; invert the grayscale image to obtain a target grayscale image containing brightness information; merge the image to be enhanced and the target grayscale image according to the dimension to obtain the target image; and the image to be enhanced contains color information.

[0122] In one possible implementation, the image enhancement model includes an encoder and a decoder; an input module 402 is specifically used to input the target image into the encoder to obtain an image feature matrix based on brightness information and color information; the image feature matrix integrates the global information and local image of the target image; the eigenvalues ​​in the image feature matrix are used to characterize the degree of enhancement of the image to be enhanced; and the image feature matrix is ​​input into the decoder to obtain a residual image.

[0123] In one possible implementation, the encoder includes a multi-head self-attention neural network unit and a residual convolution network unit; the input module 402 is specifically used to input the target image into the multi-head self-attention neural network unit to determine the eigenvalue of each pixel in the first image feature matrix based on the brightness information and color information of each pixel in the target image; the target image is input into the residual convolution network unit to determine the eigenvalue of each pixel in the local area corresponding to each local image in the second image feature matrix based on the brightness information and color information of all pixels in each local image in the target image; the first image feature matrix and the second image feature matrix are feature-fused to obtain an image feature matrix.

[0124] In a possible implementation, the input module 402 is further configured to:

[0125] Input the image sample into the image enhancement model to obtain a residual image between the image sample and the corresponding highlight image;

[0126] Superimposing the residual image between the image sample and the corresponding highlight image and the image sample to obtain the highlight image corresponding to the image sample;

[0127] Calculate the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample;

[0128] The model parameters of the image enhancement model are optimized based on the loss value until the image enhancement model meets the preset training completion conditions.

[0129] In a possible implementation, the input module 402 is specifically used to substitute the real highlight image and the highlight image corresponding to the image sample into the following formula to obtain the loss value of the image enhancement model;

[0130]

[0131] Among them, Loss is the loss value of the image enhancement model, X GT is the highlight image corresponding to the image sample, G(X) is the true highlight image, X is the image sample, and ∈ is a positive number.

[0132] The embodiment of the present application provides an image enhancement device, which includes: a determination module 401, which is used to determine a target image according to the color information and brightness information of the image to be enhanced; an input module 402, which is used to input the target image into the image enhancement model to determine the residual image between the image to be enhanced and the corresponding highlight image based on the brightness information and color information; the residual image is used to characterize the enhancement degree of the image to be enhanced; and a superposition module 403, which is used to superimpose the image to be enhanced with the residual image to obtain a highlight image corresponding to the image to be enhanced. The present application can enhance an image based on the color information and brightness information of the image to be enhanced, thereby improving the image enhancement effect.

[0133] like Figure 5 As shown, an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus, the memory 502 stores machine-readable instructions executable by the processor 501, when the electronic device is running, the processor 501 and the memory 502 communicate through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the above-mentioned image enhancement method.

[0134] Specifically, the memory 502 and the processor 501 can be general-purpose memories and processors, which are not specifically limited here. When the processor 501 runs the computer program stored in the memory 502, the image enhancement method can be executed.

[0135] Corresponding to the above-mentioned image enhancement method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image enhancement method are executed.

[0136] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0137] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the information processing method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0140] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image enhancement method, characterized in that: The method comprises: Determine the target image according to the color information and brightness information of the image to be enhanced; Inputting the target image into an image enhancement model to determine a residual image between the image to be enhanced and a corresponding highlight image based on the brightness information and the color information; the residual image is used to characterize the enhancement degree of the image to be enhanced; The image to be enhanced is superimposed on the residual image to obtain a highlight image corresponding to the image to be enhanced.

2. The image enhancement method according to claim 1, characterized in that: The step of determining the target image according to the color information and brightness information of the image to be enhanced comprises: Gray-scaling the image to be enhanced to obtain a gray-scale image; Inverting the grayscale image to obtain a target grayscale image containing the brightness information; The image to be enhanced and the target grayscale image are merged according to the dimension to obtain the target image; the image to be enhanced contains color information.

3. The image enhancement method according to claim 1 or 2, characterized in that: The image enhancement model includes an encoder and a decoder; the inputting the target image into the image enhancement model to determine a residual image between the image to be enhanced and a corresponding highlight image based on the brightness information and the color information includes: The target image is input into the encoder to obtain an image feature matrix based on the brightness information and the color information; the image feature matrix integrates the global information and the local image of the target image; the eigenvalues ​​in the image feature matrix are used to characterize the enhancement degree of the image to be enhanced; The image feature matrix is ​​input into the decoder to obtain the residual image.

4. The image enhancement method according to claim 3, characterized in that: The encoder includes a multi-head self-attention neural network unit and a residual convolution network unit; the target image is input into the encoder to obtain an image feature matrix based on the brightness information and the color information, including: Inputting the target image into the multi-head self-attention neural network unit to determine the feature value of each pixel in the first image feature matrix based on the brightness information of each pixel in the target image and the color information; Inputting the target image into the residual convolution network unit to determine the feature value of each pixel in the local area corresponding to each local image in the second image feature matrix based on the brightness information and the color information of all pixels in each local image in the target image; The first image feature matrix and the second image feature matrix are subjected to feature fusion to obtain the image feature matrix.

5. The image enhancement method according to claim 1, characterized in that: The image enhancement model is trained by the following steps: Determine the target image sample according to the color information and brightness information of the image sample; Inputting the target image sample into the image enhancement model to obtain a residual image between the image sample and the corresponding highlight image; Superimposing a residual image between the image sample and a corresponding highlight image and the image sample to obtain a highlight image corresponding to the image sample; Calculating a loss value of the image enhancement model based on a real highlight image and a highlight image corresponding to the image sample; The model parameters of the image enhancement model are optimized based on the loss value until the image enhancement model meets a preset training completion condition.

6. The image enhancement method according to claim 5, characterized in that: Calculating the loss value of the image enhancement model based on the real highlight image and the highlight image corresponding to the image sample includes: Substituting the real highlight image and the highlight image corresponding to the image sample into the following formula to obtain the loss value of the image enhancement model; Among them, Loss is the loss value of the image enhancement model, X GT is the highlight image corresponding to the image sample, G(X) is the true highlight image, X is the image sample, and ∈ is a positive number.

7. An image enhancement device, characterized in that: The device comprises: A determination module, used to determine a target image according to color information and brightness information of the image to be enhanced; An input module, used for inputting the target image into an image enhancement model, so as to determine a residual image between the image to be enhanced and a corresponding highlight image based on the brightness information and the color information; the residual image is used for representing the enhancement degree of the image to be enhanced; The superposition module is used to superimpose the image to be enhanced with the residual image to obtain a highlight image corresponding to the image to be enhanced.

8. The image enhancement device according to claim 7, characterized in that: The determination module is specifically used for: Gray-scaling the image to be enhanced to obtain a gray-scale image; Inverting the grayscale image to obtain a target grayscale image containing the brightness information; Merging the image to be enhanced and the target grayscale image according to the dimension to obtain the target image; The image to be enhanced contains color information.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the image enhancement method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image enhancement method according to any one of claims 1 to 6 are executed.

Images