Image enhancement method, electronic device and computer readable storage medium

Abstract

The application relates to the field of image processing, in particular to an image enhancement method, an electronic device and a computer readable storage medium. The image enhancement method comprises: performing downsampling on an original image to obtain a plurality of target images with lower resolution than the original image; and inputting the plurality of target images into a contrast coefficient prediction model respectively, so that a contrast enhancement coefficient of the original image at a plurality of lower resolutions can be obtained by the model; and the contrast enhancement coefficient after upsampling can be used for gray value adjustment of the original image to obtain an enhanced image. In this way, the original image is decomposed into a plurality of low-resolution images through downsampling, so that the contrast coefficient prediction model can perform contrast enhancement on the original image with high resolution by using the enhancement coefficient at a low resolution while maintaining low-resolution computing power, thereby reducing the requirement of the contrast enhancement algorithm on computing power and improving the usability of the contrast enhancement model.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image enhancement method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] During image acquisition, due to factors such as the acquisition equipment, shooting environment (e.g., hazy weather), and specific modules in the ISP (Image Signal Processor, also known as the imaging engine) pathway (e.g., the global brightening module), the acquired image may exhibit insufficient contrast. To balance the overall image contrast, contrast enhancement algorithms can be used to increase the grayscale difference between dark and bright areas, making the image clearer and more vivid.

[0003] Contrast enhancement algorithms mainly include statistical algorithms (such as linear stretching and histogram equalization) and neural network-based algorithms. However, statistical algorithms are prone to problems such as uneven brightness or sudden brightness changes, resulting in poor visual effects; neural network-based algorithms lack adjustability and have extremely high requirements for device computing power, making them less usable. Summary of the Invention

[0004] This application provides an image enhancement method, an electronic device, and a computer-readable storage medium to solve the problems of poor visual effects and limited applicability of image contrast enhancement.

[0005] In a first aspect, embodiments of this application provide an image enhancement method, including:

[0006] The original image is downsampled to obtain multiple target images with different resolutions;

[0007] The target image is input into the contrast coefficient prediction model, and the contrast enhancement coefficient output by the contrast coefficient prediction model is obtained; the contrast coefficient prediction model is a model based on a deep neural network or a filter model based on bilateral filtering.

[0008] Based on the resolution ratio of the original image to the target image, the contrast enhancement coefficient is upsampled to obtain the target enhancement coefficient;

[0009] The enhanced image is obtained by adjusting the grayscale values ​​of pixels in the original image according to the target enhancement coefficient.

[0010] In one possible implementation, before performing downsampling on the original image, the following is also included:

[0011] Obtain the image to be enhanced and the previous frame image;

[0012] The Bayer array filtering operator is used to filter the image to be enhanced and the previous frame image respectively, so as to obtain the filtering result of the current frame and the filtering result of the previous frame.

[0013] The residual image is obtained by subtracting the filtering result of the current frame from the filtering result of the previous frame.

[0014] The absolute value of the gray value of the moving region in the residual image is multiplied by a preset noise coefficient to obtain the confidence mask of the moving region. The moving region is the area in the residual image where the gray value is greater than the preset gray value threshold. The noise coefficient is inversely proportional to the noise variance of the image to be enhanced.

[0015] Multiply the motion region in the image to be enhanced by a confidence mask to obtain the motion region denoising result;

[0016] Multiply the static region in the current frame's filtering result by the static region in the previous frame's filtering result to obtain the static region noise reduction result. The static region is the area in the residual image whose gray value is less than or equal to the gray value threshold.

[0017] The noise reduction results for moving regions and stationary regions are superimposed to obtain the original image.

[0018] In one possible implementation, before upsampling the contrast enhancement coefficients based on the resolution ratio of the original image to the target image, the following steps are also included:

[0019] The target image and contrast enhancement coefficients are input into the contrast modulation model; the contrast modulation model is a neural network model used to modulate the input coefficients according to the input image.

[0020] Obtain the modulation result output by the contrast modulation model; the modulation result is used to perform upsampling based on the resolution ratio of the original image to the target image to obtain the target enhancement coefficient.

[0021] In one possible implementation, adjusting the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain the enhanced image includes:

[0022] Multiply the enhancement coefficients of multiple targets to obtain the comprehensive enhancement coefficient;

[0023] The enhanced image is obtained by mapping the gray values ​​of pixels in the original image to the target range based on the comprehensive enhancement coefficient.

[0024] In one possible implementation, downsampling is performed on the original image to obtain multiple target images of different resolutions, including:

[0025] The pooling layer with a preset pooling kernel performs average pooling or maximum pooling on the gray values ​​of pixels in the original image to obtain the pooled pixels.

[0026] Combine all pooled pixels to obtain the target image at the target resolution; the target resolution is inversely proportional to the preset pooling kernel.

[0027] In one possible implementation, upsampling is performed on the contrast enhancement coefficients to obtain the target enhancement coefficients, including:

[0028] Calculate the ratio of the original image resolution to the target image resolution to obtain the upsampling factor;

[0029] The target image is reconstructed based on the upsampling ratio to obtain the full-resolution image of the target image; the full-resolution image contains multiple empty pixels, which are pixels without contrast enhancement coefficients;

[0030] Select multiple neighboring pixels with contrast enhancement coefficients as reference pixels;

[0031] Linear interpolation is performed based on the contrast enhancement coefficient of the reference pixel to obtain the contrast enhancement coefficient of the empty pixel;

[0032] The target enhancement coefficient is obtained by combining the contrast enhancement coefficients of each pixel in the full-resolution image.

[0033] In one possible implementation, the deep neural network is one of the following: convolutional neural network, fully connected neural network, recurrent neural network, or long short-term memory network.

[0034] Secondly, this application provides an electronic device, comprising:

[0035] The downsampling module is used to downsample the original image to obtain multiple target images with different resolutions;

[0036] The coefficient determination module is used to input the target image into the contrast coefficient prediction model and obtain the contrast enhancement coefficient output by the contrast coefficient prediction model; the contrast coefficient prediction model is a model based on a deep neural network or a filter model based on bilateral filtering.

[0037] The upsampling module is used to upsample the contrast enhancement coefficient based on the ratio of the resolution of the original image to that of the target image, so as to obtain the upsampled target enhancement coefficient.

[0038] The enhancement module is used to adjust the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain the enhanced image.

[0039] In one possible implementation, the electronic device also includes: an image acquisition module and a sensor interface;

[0040] The image acquisition module is used to communicate with the image sensor via the sensor interface to obtain the raw image input from the image sensor.

[0041] Thirdly, this application provides a computer device including a memory and a processor. The memory stores a computer program, and the processor invokes and executes the computer program from the memory to implement the steps of the image enhancement method described in the first aspect.

[0042] Fourthly, this application provides a computer-readable storage medium including at least one computer instruction for performing the steps of the image enhancement method described in the first aspect.

[0043] The technical solution provided in this application allows for downsampling of the original image to obtain multiple target images with lower resolutions. These target images are then input into a contrast coefficient prediction model to obtain contrast enhancement coefficients for the original image at multiple lower resolutions. These contrast enhancement coefficients can be upsampled and used to adjust the grayscale values ​​of the original image, resulting in an enhanced image. In this way, by decomposing the original image into multiple low-resolution images through downsampling, the contrast coefficient prediction model can enhance the contrast of the high-resolution original image using the enhancement coefficients at low resolutions while maintaining low-resolution computational power. This reduces the computational requirements of the contrast enhancement algorithm and improves its applicability. Attached Figure Description

[0044] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an image enhancement method provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of a noise reduction preprocessing procedure for an input image according to an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a modulation contrast enhancement coefficient in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram illustrating a process for adjusting the grayscale values ​​of pixels in an original image based on a target enhancement coefficient, as described in an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of a process for performing downsampling on the original image in an embodiment of this application;

[0050] Figure 6This is a schematic diagram of a process for performing upsampling on the contrast enhancement coefficient in an embodiment of this application;

[0051] Figure 7 This is a partial pixel diagram of a full-resolution image in an embodiment of this application;

[0052] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0053] Figure 9 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0054] To facilitate the explanation of the technical solution of this application, some concepts involved in this application will be explained first below.

[0055] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0056] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0057] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0058] Image contrast enhancement aims to increase the grayscale difference between dark and bright areas in an image, making the image clearer and more vivid. Image contrast enhancement can mainly be achieved in two ways: one is through statistical enhancement algorithms, and the other is through end-to-end contrast enhancement of the input image using a deep learning-based neural network model.

[0059] Statistical enhancement algorithms improve the contrast difference between dark and bright areas of an image by altering its grayscale. However, this approach relies excessively on the grayscale distribution of the input image itself, resulting in unstable contrast enhancement. For example, histogram equalization adjusts grayscale levels using the image's pixel histogram. This method assumes a uniform grayscale distribution, but in many high-dynamic-range scenes, the grayscale distribution is non-uniform, leading to insufficient contrast even after histogram equalization. Furthermore, these methods require calculating the histogram for the entire image, necessitating significant computation time and resources.

[0060] Deep learning-based neural network models, by learning from a large number of training images, outperform statistical enhancement algorithms such as histogram equalization in terms of contrast enhancement visual effects. However, they require significantly more computational resources. In particular, as the resolution of input images increases, the computational demands on the model also rise. Models processing high-resolution images have extremely large parameters, making them unsuitable for deployment on edge devices. Therefore, deep neural network-based contrast enhancement methods have excessively high computational requirements and limited applicability.

[0061] To address the aforementioned technical problems, embodiments of this application provide an image enhancement method, an electronic device, and a computer-readable storage medium, which can reduce the dependence of contrast enhancement algorithms on computing power and improve their applicability.

[0062] Figure 1 This is a flowchart illustrating an image enhancement method provided in an embodiment of this application. The image enhancement method includes:

[0063] Step S210: Downsampling is performed on the original image to obtain multiple target images with different resolutions.

[0064] The original image can be the image that needs contrast enhancement. Downsampling can include pooling operations such as max pooling and average pooling.

[0065] In some possible implementations, the pixels in the original image can be divided into several sub-regions based on the downsampling factor. All pixels in each sub-region are merged into a single pixel, and the average grayscale value of all pixels in the sub-region is used as the grayscale value of the merged pixel. This way, all the merged pixels can form a target image with a lower resolution than the original image. For example, with a downsampling factor of 4, the original image can be divided into sub-regions in the form of a 2×2 pixel matrix. Four pixels in the matrix can be merged into one pixel, and the average grayscale value of these four pixels can be used as the grayscale value of the merged pixel. Thus, by setting multiple different downsampling factors, multiple target images with lower resolutions than the original image can be obtained. It is understandable that the higher the downsampling factor, the lower the resolution of the resulting target image.

[0066] In some other possible implementations, if the number of pixels at the edge of the image is insufficient to form a sub-region, a certain number of pixels can be filled at the edge based on the gray values ​​of neighboring pixels to form a sub-region.

[0067] In some other possible implementations, downsampling of the original image can also be performed using a deep learning-based downsampling algorithm.

[0068] Step S220: Input the target image into the contrast coefficient prediction model and obtain the contrast enhancement coefficient output by the contrast coefficient prediction model.

[0069] The contrast coefficient prediction model can be a deep neural network-based model or a bilateral filter-based model. The deep neural network can be one of the following: Convolutional Neural Networks (CNN), Feedforward Neural Networks (FCN), Recurrent Neural Networks (RNN), or Long Short-Term Memory (LSTM). The contrast coefficient prediction model can output a contrast enhancement coefficient based on the resolution of the input image.

[0070] In some possible implementations, multiple target images of different resolutions can be input into the contrast coefficient prediction model. The contrast enhancement coefficient corresponding to each target image can be obtained through the contrast coefficient prediction model. This contrast enhancement coefficient can be used to adjust the gray values ​​of pixels in the target image to enhance the overall contrast of the image.

[0071] Step S230: Based on the resolution ratio of the original image to the target image, the contrast enhancement coefficient is upsampled to obtain the target enhancement coefficient.

[0072] Upsampling can include upsampling algorithms based on unilinear interpolation, bilinear interpolation, tribilinear interpolation, transposed convolution, etc.

[0073] In some possible implementations, the upsampling factor can be determined based on the resolution ratio of the original image to the target image, and the contrast enhancement coefficient of the target image can be upsampled to obtain the upsampled target enhancement coefficient. In some other possible implementations, the downsampling factor of the original image in step S210 can also be directly used as the upsampling factor of the corresponding target image to perform the upsampling operation.

[0074] In some possible implementations, after determining the upsampling factor, the target image can be reconstructed based on the upsampling factor, upsampling the target image into a reconstructed image with the same resolution as the original image. During the upsampling process, the contrast enhancement coefficient of the target image can also be transformed into the target enhancement coefficient of the reconstructed image through the same upsampling operation.

[0075] Step S240: Adjust the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain the enhanced image.

[0076] In some possible implementations, since each target image corresponds to a target enhancement coefficient, multiple target enhancement coefficients can be combined through mathematical operations to obtain a combined comprehensive enhancement coefficient. This comprehensive enhancement coefficient is then used to adjust the grayscale values ​​of pixels in the original image, thereby enhancing the grayscale difference between dark and bright areas, resulting in an enhanced image. In some possible implementations, the mathematical operations can include addition, subtraction, multiplication, division, exponentiation, logarithms, and other mathematical operations. In other possible implementations, the grayscale values ​​of pixels in the original image can be multiplied by the comprehensive enhancement coefficient, and the product can be used as the grayscale values ​​of pixels in the enhanced image.

[0077] In the above embodiments, the original image can be downsampled to obtain multiple target images with lower resolutions. These target images are then input into a contrast coefficient prediction model to obtain contrast enhancement coefficients for the original image at these lower resolutions. These contrast enhancement coefficients can then be upsampled and used to adjust the grayscale values ​​of the original image, resulting in an enhanced image. In this way, by decomposing the original image into multiple low-resolution images through downsampling, the contrast coefficient prediction model can enhance the contrast of the high-resolution original image using the enhancement coefficients at lower resolutions while maintaining low-resolution computational power. This reduces the computational requirements of the contrast enhancement algorithm and improves its applicability.

[0078] Figure 2 This is a schematic diagram of a noise reduction preprocessing procedure for an input image according to an embodiment of this application, such as... Figure 2 As shown, the process includes:

[0079] Step S201: Obtain the image to be enhanced and the previous frame image of the image to be enhanced.

[0080] In some possible implementations, for images captured in succession or images obtained from video decomposition, noise reduction preprocessing can be performed on the input image before image contrast enhancement to reduce image noise.

[0081] For example, when the image to be enhanced is a series of multiple frames captured by an image acquisition device under the same shooting environment, the frame to be enhanced can be used as the image to be enhanced. This frame is then combined with the preceding frame, and the static and dynamic regions in both frames are used for comprehensive filtering and noise reduction. The static region can be an area in both frames where the grayscale value remains unchanged or changes within a certain range, while the dynamic region can be an area outside the static region in both frames.

[0082] In some other possible implementations, the denoising result of the previous frame of the image to be enhanced can be directly obtained and used to perform the following denoising preprocessing steps in combination with the image to be enhanced.

[0083] Step S202: The Bayer array filter operator is used to perform filtering on the image to be enhanced and the previous frame image respectively to obtain the current frame filtering result and the previous frame filtering result.

[0084] In some possible implementations, the filtering operator can use a 3×3 or 5×5 Bayer luminance channel filter for simple pre-denoising. The image to be enhanced, after pre-denoising, yields the current frame's filtering result, and the previous frame's image, after pre-denoising, yields the previous frame's filtering result.

[0085] Step S203: Subtract the current frame filtering result from the previous frame filtering result to obtain the residual image.

[0086] In some possible implementations, the gray values ​​of each pixel in the current frame filtering result can be subtracted from the gray values ​​of each pixel in the previous frame filtering result, and the absolute value of the obtained gray value difference can be taken to obtain the residual image of the current frame filtering result and the previous frame filtering result.

[0087] Step S204: Multiply the absolute value of the gray value of the motion region in the residual image by a preset noise coefficient to obtain the confidence mask of the motion region.

[0088] The motion region is the area in the residual image whose gray value is greater than the preset gray value threshold; the noise coefficient is inversely proportional to the noise variance of the image to be enhanced.

[0089] In some possible implementations, the moving and stationary regions in the current frame's filtering result and the previous frame's filtering result can be obtained using a preset grayscale threshold and a residual image. It is understood that the pixel coordinates corresponding to the moving regions in the current frame's filtering result, the previous frame's filtering result, and the residual image are the same.

[0090] Since the motion region needs to consider not only the difference between the current frame and the previous frame, but also the impact of noise levels, some possible implementations can obtain a confidence mask for the motion region by multiplying the grayscale value of the motion region by a noise coefficient. This noise coefficient can be inversely proportional to the noise variance of the current frame's filtering result; the higher the noise, the smaller the noise coefficient. Ultimately, the confidence of the motion region will be limited to between 0 and 1.

[0091] In some other possible implementations, the grayscale threshold can be 0.

[0092] Step S205: Multiply the motion region in the image to be enhanced by a confidence mask to obtain the motion region denoising result.

[0093] Step S206: Multiply the stationary region in the current frame filtering result by the stationary region in the previous frame filtering result to obtain the stationary region noise reduction result.

[0094] The static region is the area in the residual image where the gray value is less than or equal to the gray value threshold.

[0095] In some possible implementations, the static region can be obtained by directly subtracting the moving region from the filtering result.

[0096] Step S207: The noise reduction results of the moving region and the noise reduction results of the stationary region are superimposed to obtain the original image.

[0097] In the above embodiments, for consecutive multi-frame images, before performing image enhancement on the current frame image, the previous frame image or the image after noise reduction of the previous frame image can be obtained. The moving region and the stationary region can be determined by the gray value changes of the two frames. Thus, the noise reduction effect on the current frame image can be improved by combining the confidence of the moving region with the advantages of temporal filtering, thereby improving the quality of the image to be enhanced.

[0098] Figure 3 This is a schematic flowchart of a modulation contrast enhancement coefficient in an embodiment of this application, as shown below. Figure 3 As shown, the process includes:

[0099] Step S221: Input the target image and contrast enhancement coefficients into the contrast modulation model.

[0100] The contrast modulation model is a neural network model used to modulate the input coefficients according to the input image.

[0101] In some possible implementations, an image processing model based on deep neural networks such as CNN or RRN can be pre-built. Several images and their corresponding contrast enhancement coefficients are used as training samples. The image processing model learns the features in the input images and modulates the input coefficients based on these learned features. The modulated coefficients are then output as the modulation result. If the modulation result does not meet the preset requirements, the model parameters can be modified until the modulation result meets the requirements. The contrast modulation model is then obtained based on the current model parameters. Modulating the input coefficients can include performing mathematical operations on them, such as addition, subtraction, multiplication, division, exponentiation, and logarithms.

[0102] In some other possible implementations, when training the image processing model, manual annotation or other methods can be used to enable the model to modulate the contrast enhancement coefficient of the input image based on features such as the flatness of the input image or the standard deviation of noise, so that the modulated contrast enhancement coefficient can improve the contrast enhancement effect of the input image.

[0103] Step S222: Obtain the modulation result output by the contrast modulation model.

[0104] The modulation result is used to perform upsampling based on the ratio of the resolution of the original image to that of the target image to obtain the target enhancement coefficient.

[0105] In some possible implementations, the modulation result output by the contrast modulation model can be obtained. This modulation result is the result of the contrast enhancement coefficient of the target image modulated by the contrast modulation model, and can be used to replace the contrast enhancement coefficient in step S230 to perform upsampling.

[0106] As an example, if the contrast enhancement coefficient of the target image is to enhance the brightness of a specific area in the target image by one brightness level, the contrast modulation model can modulate the enhancement coefficient by multiplying it by a constant. For example, when the constant is 3, the modulation result is to enhance the brightness of a specific area in the target image by three brightness levels.

[0107] In some other possible implementations, the contrast modulation model can be configured with multiple modulation levels, each corresponding to a different modulation intensity. The contrast modulation model can select the user's preferred modulation level in response to user input. Here, the modulation intensity can be used to represent the degree to which the contrast modulation model changes the input coefficients.

[0108] In the above embodiments, before upsampling the contrast enhancement coefficient, the contrast enhancement coefficient can be modulated using a contrast modulation model to adjust the degree of image contrast enhancement. Since the contrast coefficient prediction model is fixed after practical application, the resulting contrast enhancement coefficient for the target image is also fixed and lacks adjustability. According to the method in this embodiment, the input contrast enhancement coefficient can be modulated using a contrast modulation model, and the modulated coefficient can be used to perform upsampling to obtain the target enhancement coefficient, thus improving the adjustability of the contrast enhancement process.

[0109] Figure 4 This is a schematic diagram illustrating a process for adjusting the grayscale values ​​of pixels in the original image based on a target enhancement coefficient, as described in an embodiment of this application. Figure 4 As shown, the process includes:

[0110] Step S2402: Multiply the multiple target enhancement coefficients to obtain the comprehensive enhancement coefficient.

[0111] In some possible implementations, after obtaining multiple target enhancement coefficients corresponding to target images of different resolutions through upsampling, a multiplication operation can be performed on the multiple target enhancement coefficients, and the resulting product can be used as the comprehensive enhancement coefficient.

[0112] Step S2404: Map the gray values ​​of pixels in the original image to the target range according to the comprehensive enhancement coefficient to obtain the enhanced image.

[0113] In some possible implementations, the gray values ​​of each pixel in the original image can be multiplied by the corresponding comprehensive enhancement coefficient, thereby mapping the gray values ​​of the original image to the target range, and the mapped image can be used as the enhanced image.

[0114] Figure 5 This is a schematic diagram of a process for performing downsampling on the original image in an embodiment of this application, as shown below. Figure 5 As shown, the process includes:

[0115] Step S2102: Perform average pooling or maximum pooling on the grayscale values ​​of pixels in the original image through a pooling layer with a preset pooling kernel to obtain the pooled pixels.

[0116] Pooling can be used to reduce the dimensionality of data in convolutional neural networks (CNNs) or recurrent neural networks (RNNs). The pooling kernel represents the size of the pooling window; the larger the pooling kernel, the smaller the resolution of the image after pooling.

[0117] Step S2104: Combine all pooled pixels to obtain the target image at the target resolution.

[0118] The target resolution is inversely proportional to the preset pooling kernel.

[0119] In some possible implementations, the pooling kernel can be set to 2 and the stride to 2 to perform average pooling on the original image. This divides the pixels in the original image into several 2×2 pooling regions, each containing 4 pixels. After pooling, these regions are merged into 1 pixel. The gray value of the merged pixel is the average of the gray values ​​of the 4 pixels before pooling. The resolution of the target image obtained through this pooling operation is one-quarter of that of the original image.

[0120] Understandably, by changing the size of the pooling kernel, it's possible to obtain target images with resolutions lower than the original image. In some other possible implementations, the target image can also be obtained using max pooling.

[0121] In the above embodiments, pooling operations can be used to downsample the original image to obtain multiple target images with lower resolutions than the original image, thereby reducing the complexity of the image input to the contrast coefficient prediction model, thus reducing the dependence of contrast enhancement on computing power and improving applicability.

[0122] Figure 6 This is a schematic diagram of a process for upsampling the contrast enhancement coefficient in an embodiment of this application, as shown below. Figure 6 As shown, the process includes:

[0123] Step S2301: Calculate the ratio of the original image resolution to the target image resolution to obtain the upsampling factor.

[0124] In some possible implementations, the ratio of the original image resolution to the target image resolution can be used as the upsampling factor of the target image.

[0125] Step S2302: Reconstruct the target image according to the upsampling ratio to obtain the full-resolution image of the target image.

[0126] A full-resolution image contains multiple empty pixels, which are pixels that do not have a contrast enhancement coefficient. A full-resolution image is an image where the target image and the original image are at the same resolution.

[0127] Step S2303: Select multiple neighboring pixels with contrast enhancement coefficients as reference pixels.

[0128] Step S2304: Perform linear interpolation based on the contrast enhancement coefficient of the reference pixel to obtain the contrast enhancement coefficient of the empty pixel.

[0129] Step S2305: Combine the contrast enhancement coefficients of each pixel in the full-resolution image to obtain the target enhancement coefficient.

[0130] As an example, Figure 7 This is a partial pixel diagram of a full-resolution image. After reconstructing the full-resolution image based on the upsampling ratio, the full-resolution image contains multiple empty pixels and multiple pixels with contrast enhancement coefficients, such as... Figure 7 As shown, each square represents a pixel, and the number in the square represents the contrast enhancement coefficient of that pixel. The four neighboring pixels with contrast enhancement coefficients of an empty pixel can be selected as reference pixels, and the contrast enhancement coefficient of the empty pixel can be obtained through bilinear interpolation. This method can be used to obtain the contrast enhancement coefficients of all pixels in the full-resolution image, thus yielding the target enhancement coefficient corresponding to the entire image.

[0131] Corresponding to the aforementioned embodiments of image enhancement methods, this application also provides an embodiment of an electronic device 300. For example... Figure 8 As shown, the electronic device includes:

[0132] The downsampling module 310 is used to perform downsampling on the original image to obtain multiple target images with different resolutions;

[0133] The coefficient determination module 320 is used to input the target image into the contrast coefficient prediction model and obtain the contrast enhancement coefficient output by the contrast coefficient prediction model; the contrast coefficient prediction model is a model based on a deep neural network or a filter model based on bilateral filtering.

[0134] The upsampling module 330 is used to upsample the contrast enhancement coefficient based on the ratio of the resolution of the original image to the target image, so as to obtain the upsampled target enhancement coefficient.

[0135] The enhancement module 340 is used to adjust the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain the enhanced image.

[0136] In some possible implementations, the electronic device 300 also includes an image acquisition module and a sensor interface.

[0137] The image acquisition module is used to communicate with the image sensor through the sensor interface to obtain the raw image input from the image sensor.

[0138] In some possible implementations, the sensor interface can communicate with an image sensor outside the electronic device 300 to obtain an input image. The image sensor can be an external image acquisition device such as a camera or camcorder.

[0139] Specific limitations regarding the electronic device 300 can be found in the limitations of the image enhancement method described above, and will not be repeated here. Each module in the aforementioned electronic device 300 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0140] In one exemplary embodiment, the present application can utilize a computer device to implement the above-described image enhancement method.

[0141] In one possible implementation, the computer device is structured as follows: Figure 9 As shown, the computer device 100 includes at least one processor 110, a memory 120, a communication bus 130, and at least one communication interface 140.

[0142] The processor 110 can be a general-purpose central processing unit (CPU), network processor (NP), microprocessor, or one or more integrated circuits for implementing the scheme of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0143] Processor 110 may include one or more CPUs. Computer device 100 may include multiple processors 110. Each of these processors 110 may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). It should be noted that processor 110 here may refer to one or more devices, circuits, and / or processing cores used for processing data (such as computer program instructions).

[0144] The memory 120 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions; it may also be a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. Optionally, the memory 120 may exist independently and be connected to the processor 110 via a communication bus 130; the memory 120 may also be integrated with the processor 110.

[0145] Communication bus 130 is used to transfer information between components (such as between the processor and memory). Communication bus 120 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The diagram uses only one communication bus, but this does not imply that there is only one bus or one type of bus. Communication interface 140 is used for the computer device 100 to communicate with other devices or communication networks.

[0146] The communication interface 140 includes a wired communication interface or a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a Wireless Local Area Network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0147] In some embodiments, the memory 120 may be used to store a computer program that executes the scheme of this application, and the processor 110 may execute the computer program stored in the memory 120. For example, the computer device 100 may use the processor 110 to call and execute the computer program stored in the memory 120 to implement the steps of the image enhancement method provided in the embodiments of this application. It should be understood that the image enhancement method provided in this application can be applied to an image enhancement device, which may be implemented by software, hardware, or a combination of software and hardware as part or all of the processor 110 and integrated into the computer device 100.

[0148] Furthermore, it should be understood that the embodiments of this application can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. This computer program product includes a computer program. When the computer program is loaded and run on a computer device, all or part of the processes or functions shown in the embodiments of this application are generated.

[0149] The computer program may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program may be transferred from one website, terminal, server or data center to another website, terminal, server or data center via wired or wireless means.

[0150] The computer-readable storage medium can be any available medium that can be accessed by a computer device, or a data storage device such as a server or data center that integrates one or more available media.

[0151] It should be understood that the above are merely specific implementation methods of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. An image enhancement method, characterized in that, include: The original image is downsampled to obtain multiple target images with different resolutions; The target image is input into the contrast coefficient prediction model, and the contrast enhancement coefficient output by the contrast coefficient prediction model is obtained; The contrast coefficient prediction model is a deep neural network-based model or a bilateral filter-based model. Based on the ratio of the resolution of the original image to the resolution of the target image, the contrast enhancement coefficient is upsampled to obtain the target enhancement coefficient; The grayscale values ​​of the pixels in the original image are adjusted according to the target enhancement coefficient to obtain the enhanced image.

2. The image enhancement method according to claim 1, characterized in that, Before performing downsampling on the original image, the following is also included: Obtain the image to be enhanced and the previous frame image of the image to be enhanced; The image to be enhanced and the previous frame image are filtered by the Bayer array filter operator to obtain the current frame filtering result and the previous frame filtering result. The residual image is obtained by subtracting the current frame filtering result from the previous frame filtering result; The absolute value of the gray value of the moving region in the residual image is multiplied by a preset noise coefficient to obtain a confidence mask for the moving region. The moving region is the region in the residual image whose gray value is greater than a preset gray value threshold. The noise coefficient is inversely proportional to the noise variance of the image to be enhanced. Multiply the motion region in the image to be enhanced by the confidence mask to obtain the motion region denoising result; Multiply the static region in the current frame filtering result by the static region in the previous frame filtering result to obtain the static region noise reduction result. The static region is the region in the residual image whose gray value is less than or equal to the gray value threshold. The noise reduction results of the moving region and the noise reduction results of the stationary region are superimposed to obtain the original image.

3. The image enhancement method according to claim 1, characterized in that, Before upsampling the contrast enhancement coefficient based on the resolution ratio of the original image to the target image, the method further includes: The target image and the contrast enhancement coefficient are input into a contrast modulation model; the contrast modulation model is a neural network model used to modulate the input coefficients according to the input image. Obtain the modulation result output by the contrast modulation model; the modulation result is used to perform upsampling based on the resolution ratio of the original image to the target image to obtain the target enhancement coefficient.

4. The image enhancement method according to claim 1, characterized in that, The step of adjusting the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain the enhanced image includes: Multiply the multiple target enhancement coefficients together to obtain the comprehensive enhancement coefficient; The gray values ​​of pixels in the original image are mapped to the target range based on the comprehensive enhancement coefficient to obtain the enhanced image.

5. The image enhancement method according to claim 1, characterized in that, The downsampling of the original image to obtain multiple target images of different resolutions includes: The grayscale values ​​of pixels in the original image are subjected to average pooling or maximum pooling by a pooling layer with a preset pooling kernel to obtain the pooled pixels. Combine all pooled pixels to obtain a target image with the target resolution; the target resolution is inversely proportional to the preset pooling kernel.

6. The image enhancement method according to claim 1, characterized in that, The upsampling of the contrast enhancement coefficient to obtain the target enhancement coefficient includes: Calculate the ratio of the original image resolution to the target image resolution to obtain the upsampling factor; The target image is reconstructed according to the upsampling ratio to obtain a full-resolution image of the target image; the full-resolution image contains multiple empty pixels, which are pixels without contrast enhancement coefficients; Select multiple neighboring pixels with contrast enhancement coefficients as reference pixels; Linear interpolation is performed based on the contrast enhancement coefficient of the reference pixel to obtain the contrast enhancement coefficient of the empty pixel; The target enhancement coefficient is obtained by combining the contrast enhancement coefficients of each pixel in the full-resolution image.

7. The image enhancement method according to claim 1, characterized in that, The deep neural network is one of the following: convolutional neural network, fully connected neural network, recurrent neural network, and long short-term memory network.

8. An electronic device, characterized in that, include: The downsampling module is used to downsample the original image to obtain multiple target images with different resolutions; The coefficient determination module is used to input the target image into the contrast coefficient prediction model and obtain the contrast enhancement coefficient output by the contrast coefficient prediction model; the contrast coefficient prediction model is a model based on a deep neural network or a filter model based on bilateral filtering. An upsampling module is used to upsample the contrast enhancement coefficient based on the ratio of the resolution of the original image to that of the target image, so as to obtain the upsampled target enhancement coefficient. An enhancement module is used to adjust the grayscale values ​​of pixels in the original image according to the target enhancement coefficient to obtain an enhanced image.

9. The electronic device according to claim 8, characterized in that, Also includes: Image acquisition module, sensor interface; The image acquisition module is used to communicate with the image sensor through the sensor interface to obtain the original image input by the image sensor.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes at least one computer instruction for causing a computer to perform the steps of the image enhancement method as described in any one of claims 1-7.

Images