Image processing methods, electronic devices, and computer-readable storage media

Abstract

This application provides an image processing method, an electronic device, and a computer-readable storage medium. Relating to the field of computer vision technology, the method includes: extracting global and local features of an original image; determining the black level value of each unit pixel in the original image based on the global and local features, wherein the black level value is generated by device noise from the sensor that acquired the original image; performing image correction based on the black level value of each unit pixel to obtain intermediate image features; performing denoising processing based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features; and performing image enhancement processing on the reconstructed image features to obtain a target image. In the technical solution of this application embodiment, this application solves the technical problem in related technologies where the image quality after processing under low-light conditions is unsatisfactory.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the field of low-light imaging, most existing technologies rely on single lighting conditions or noise models for image enhancement: users optimize image quality by adjusting parameters or selecting preset modes. However, these solutions are only optimized for specific lighting or noise types and are difficult to adapt to complex and changing environments. Furthermore, because existing enhancement algorithms lack the ability to perceive detailed features, especially under low-light conditions, text information and edge details are easily lost, leading to a decline in image quality. This limits the precision with which users can control the image enhancement effect and reduces the applicability of image processing algorithms in low-light conditions. Summary of the Invention

[0003] This application provides an image processing method, an electronic device, and a computer-readable storage medium to alleviate or solve the technical problem of unsatisfactory image quality after processing under low-light conditions in related technologies.

[0004] In a first aspect, embodiments of this application provide an image processing method, including:

[0005] Extract global and local features from the original image;

[0006] Based on global and local features, the black level value of each unit pixel in the original image is determined. The black level value is generated by the device noise of the sensor that acquired the original image.

[0007] Image correction is performed based on the black level value of each unit pixel to obtain intermediate image features;

[0008] Denoising is performed based on intermediate image features, and the denoised intermediate image features are then repaired to obtain reconstructed image features.

[0009] Image enhancement processing is performed on the reconstructed image features to obtain the target image.

[0010] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0011] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0012] Based on the image processing method of the first aspect described above, this application has at least the following beneficial effects or advantages:

[0013] This application provides a novel low-light enhancement framework, aiming to introduce additional control information such as noise suppression and detail restoration into the enhancement process based on multimodal fusion of illumination conditions, noise types, and detail features, thereby achieving high-quality image enhancement under multimodal control. The end-to-end design of this application not only significantly improves the overall quality of image processing but also dynamically adapts to different hardware characteristics and illumination conditions, solving the performance bottleneck problem caused by modular separation in traditional methods. Compared with solutions provided in related technologies, it can achieve better image processing capabilities under low-light conditions.

[0014] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0015] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0016] Figure 1 A flowchart illustrating an image processing method according to an embodiment of this application is shown;

[0017] Figure 2 A schematic diagram of the U-Net for the image processing method according to an embodiment of this application is shown;

[0018] Figure 3 A schematic framework diagram of the image processing method according to an embodiment of this application is shown;

[0019] Figure 4 A schematic diagram of an image processing apparatus according to an embodiment of this application is shown;

[0020] Figure 5 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0021] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0022] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0023] The following terms will be used in the following text:

[0024] Image signal processing algorithms are the key technology for converting raw Bayer array images captured by CMOS sensors into high-quality RGB images.

[0025] CMOS (Complementary Metal-Oxide-Semiconductor) sensors use complementary MOSFET (Metal-Oxide-Semiconductor Field-Effect Transistor) technology to construct logic gates and other electronic circuits. They consist of multiple photosensitive elements and are capable of converting light signals into electrical signals.

[0026] A Bayer array is a common type of color filter array (CFA).

[0027] Black level correction (BLC) is defined as the level value output by a CMOS sensor when the pixel value is 0. Due to limitations in sensor hardware, even in completely dark environments, a CMOS sensor will still output a non-zero baseline signal, known as the black level. Without correction, this baseline signal will cause an overall brightness shift in the image, affecting the accuracy of subsequent processing steps.

[0028] White balance (WB) is an image processing technique designed to correct color deviations in an image, ensuring that it appears as true white under different light sources. By adjusting the intensity ratio of the red, green, and blue channels in an image, it compensates for the color temperature of the light source, making colors look more natural and accurate, thus restoring true colors and improving the visual effect of the image.

[0029] Gamma correction (GC) is a technique used to adjust the brightness and contrast of an image. It changes the brightness level of an image by applying a non-linear transformation, making the image display effect more consistent with the human eye's perception of light.

[0030] In related technologies, existing image signal processing methods are mainly divided into two categories: data-driven and model-driven.

[0031] Data-driven methods achieve better image restoration results by effectively utilizing limited data. The core of these methods lies in data augmentation, which improves model performance by generating diverse training data. However, the main limitation of data-driven methods is their insufficient generalization ability. The effect of data augmentation is often closely related to hardware devices, leading to significant differences in performance across different hardware platforms. Furthermore, these methods are highly dependent on data quality and scale, making it difficult to achieve ideal results in scenarios where data is scarce.

[0032] Model-driven methods focus on improving image processing performance by optimizing model structure and algorithms. However, model-based methods typically have lower PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) values ​​than data-driven methods, and their overall performance is still slightly inferior to that of data-driven methods.

[0033] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0034] Figure 1 A flowchart illustrating an embodiment of the image processing method of this application is shown, such as... Figure 1 As shown, the method may include steps S101 to S105.

[0035] Step S101: Extract global and local features from the original image;

[0036] Step S102: Based on global and local features, determine the black level value of each unit pixel in the original image. The black level value is generated by the device noise of the sensor that acquired the original image.

[0037] Step S103: Perform image correction based on the black level value of each unit pixel to obtain intermediate image features;

[0038] Step S104: Denoise the intermediate image features and repair the denoised intermediate image features to obtain the reconstructed image features;

[0039] Step S105: Perform image enhancement processing on the reconstructed image features to obtain the target image.

[0040] For example, the aforementioned execution entity can be of various types, such as an image processing unit integrated on the edge or a cloud server. Edge integration can be applied to mobile terminals where the image acquisition device and image processing unit are jointly integrated, such as in a mobile phone. It can also be integrated separately into the image acquisition device or into a connected host computer. The aforementioned image acquisition device can be various forms of photoelectric conversion devices, such as CMOS devices, CCD (Charge-Coupled Device) sensors, EMCCD (Electron Multiplying Charge-Coupled Device) sensors, BSI-CMOS (Back-Side Illuminated CMOS) sensors, etc. Cloud server integration can involve transmitting the original image from the image acquisition device and then performing image processing in the cloud.

[0041] For example, the aforementioned black level represents a form of noise generated by a photoelectric acquisition device in the absence of light. In some embodiments, the black level can be replaced by a dark current form, i.e., the current generated inside the image sensor due to thermal excitation.

[0042] For example, the above processing flow is applied in extremely low light environments and can maintain good image quality at lower light intensities compared to existing technologies, such as illuminance below 0.1 lux.

[0043] In the embodiments provided in this application, global and local features are extracted from the original image. Global features include overall information about the image, while local features focus on specific regions of the image, such as edges and textures. Based on the extracted global and local features, a black level value is determined for each pixel unit in the original image. The black level value reflects the device noise generated by the image sensor during image acquisition. The image is corrected using the determined black level value of each pixel to eliminate the influence of device noise, thereby obtaining intermediate image features that improve the dynamic range and signal-to-noise ratio of the image under low-light conditions. The intermediate image features are then denoised to further reduce noise in the image. Since distortion may be introduced during image processing, especially under low-light conditions, the denoised intermediate image features need to be repaired to restore the details and structure of the image, resulting in reconstructed image features. The reconstructed image features are then subjected to image enhancement processing to improve the visual effect of the image, resulting in the final target image.

[0044] By predicting the black level value of each unit pixel, sensor noise is eliminated, avoiding the problem of image correction inaccuracy caused by traditional global correction. Combined with effective noise separation and image restoration, edges and details are preserved, significantly improving the clarity and naturalness of the image to achieve better image processing results.

[0045] For example, the extraction process in step S101 can be performed in several ways. For global feature extraction, global average pooling or global max pooling can be used to compress the feature map into a vector dimension, capturing the overall information of the image. Global average pooling takes the average value of the spatial dimension for each channel of the feature map. Global max pooling takes the maximum spatial value for each channel. For local feature extraction, convolution or local attention can be used. A convolutional layer can contain multiple convolutional kernels, each responsible for detecting different types of features, such as edges, textures, and corners. Convolution is performed by sliding the convolutional kernel across the input image. At each location, the convolutional kernel is multiplied element-wise with the local region of the image and summed to generate an output value (such as a pixel in the feature map). Local attention divides the image into non-overlapping or overlapping local windows (such as 8x8 pixels), calculates self-attention within each window, focuses on the local region, and avoids excessive smoothing of low-light noise by global attention.

[0046] According to the embodiments provided in this application, step S102: determining the black level value of each unit pixel in the original image based on global features and local features may include the following specific steps:

[0047] Global and local features are fused using a fully connected layer to obtain fused features;

[0048] The fused features are input into the black level prediction layer, which determines the black level value of each unit pixel based on the sensor features.

[0049] In some embodiments provided in this application, for low-light image processing, global features characterize the overall illumination and color distribution of the image, while local features capture pixel-level noise and edge details. The global and local features are concatenated and input into a fully connected layer to generate fused features. Global features provide macroscopic patterns of sensor noise, such as the black level distribution in the original image, while local features adapt to pixel-level differences. By fusing local features, pixel-by-pixel black level prediction is achieved, improving the signal-to-noise ratio in low-light areas. It should be noted that, under low-brightness imaging conditions where black level noise dominates, incorporating local black level correction can help recover more shadow details.

[0050] For example, by learning weights and biases, the aforementioned global and local features are integrated into a unified feature vector, i.e., fused features. Optionally, a weighted summation of the features can be performed, where the weights are learned by the network during training to maximize the expressive power of the features.

[0051] For example, the aforementioned black level prediction layer can be obtained by combining one or more linear layers and activation layers. The main function of the linear layer is to map the input data to a new space, achieving a linear combination of features. In neural networks, it is used to connect different layers, transmit and transform feature information, and is also called a fully connected layer. The aforementioned activation layer can be a ReLU layer (Rectified Linear Unit layer), a Sigmoid layer, or a Tanh layer (hyperbolic tangent layer). The ReLU layer increases the expressive power of the neural network by introducing nonlinearity. The Sigmoid layer maps the input value to between 0 and 1, interpreting the output as a probability value. The Tanh layer maps the input value to between -1 and 1. Similar to the Sigmoid layer, the Tanh layer increases the expressive power of the neural network through nonlinear transformation. The Tanh function has a wider output range and is centrally symmetric.

[0052] For ease of understanding, an illustrative example is provided. The weights and biases in the fully connected layer are initialized. Weights are typically initialized to random values ​​to break symmetry and prevent all neurons from learning the same features, while biases are initialized to 0 or small random values. During the forward propagation phase, global and local features are passed as input to the fully connected layer. Each neuron computes a weighted sum of its inputs plus the bias, and outputs the result through an activation function (such as ReLU). The loss between the network output and the true label is calculated. This loss can be set in various forms, such as mean squared error or cross-entropy loss. The gradient of the loss function with respect to the weights and biases is calculated using the backpropagation algorithm, calculating the contribution of each weight and bias to the final loss. The weights and biases are updated based on the calculated gradient using an optimization algorithm (such as gradient descent, Adam, etc.).

[0053] In this way, neural networks can automatically learn how to maximize the expressive power of features by adjusting weights and biases, thereby effectively integrating global and local features to form a unified feature vector. This learning process enables the network to capture complex patterns and relationships in the input data, improving the model's predictive performance.

[0054] For example, in low-light image processing, sensor characteristics directly affect the calculation of the black level value. These sensor characteristics can be various, such as photosensitivity, sensor rated operating temperature, and dark current (black level) distribution. Higher photosensitivity results in more significant noise and a stronger black level. The relationship between sensor rated operating temperature and device noise can be expressed as follows: increased temperature exacerbates thermal noise (e.g., the dark current of CMOS sensors exhibits an exponential relationship with temperature). These characteristics can also be represented by sensor model, with different models having different base noise levels.

[0055] Compared to global black level correction, the pixel-by-pixel correction method has the advantage that each pixel in an image sensor may have different dark current levels due to manufacturing differences, temperature changes, or circuit noise. Non-uniformity in the sensor manufacturing process leads to inconsistent light responses from different pixels, resulting in inconsistent black levels.

[0056] For example, the black level prediction layer can adaptively predict black levels based on sensor characteristics, ensuring the model can capture and learn sensor characteristics and their impact on black levels, such as collecting data from different sensor types, different lighting conditions, and different environments. The dataset should be sufficiently diverse to cover possible application scenarios. The true black level value corresponding to each image sample should be labeled using laboratory measurements or known reference images. Sensor-specific features, such as sensor model, manufacturing batch, temperature readings, etc., can be integrated into the model as part of the input. In addition to image data, sensor metadata can be used as an additional input modality to help the model understand the source and conditions of the data. Furthermore, if available data is limited, a transfer learning strategy can be used to fine-tune the black level prediction layer, starting from a pre-trained black level prediction layer on a large amount of data, for data from a specific sensor model. The model's performance can be evaluated under different sensors and conditions to ensure the black level prediction layer has good generalization ability and adaptability. Preferably, in practical applications, the black level prediction layer can also employ an online learning strategy to adjust its predictions based on real-time feedback to adapt to dynamic changes in sensor characteristics.

[0057] According to the embodiments provided in this application, in step S103 above, image correction is performed based on the black level value of each unit pixel to obtain intermediate image features, which may include the following specific steps:

[0058] Subtract the corresponding black level value from the original pixel value of each unit pixel to obtain the first correction value for each unit pixel;

[0059] Intermediate image features are generated based on the first correction value for each unit pixel.

[0060] In the embodiments provided in this application, for each unit pixel in the image, its original pixel value is subtracted from the corresponding black level value to obtain a first correction value, in order to eliminate the noise generated by the image sensor under no-light conditions, i.e., the black level signal. Based on the first correction value of each unit pixel, intermediate image features are generated, which are the corrected image data and form the basis for further image processing (such as denoising, enhancement, etc.).

[0061] According to the embodiments provided in this application, the method further includes the following specific steps:

[0062] The fused features are input into the white balance prediction layer to determine the white balance value for each unit pixel;

[0063] Color temperature correction is performed based on the white balance value of each unit pixel to obtain a second correction value for each unit pixel;

[0064] The intermediate image features are obtained based on the first and second correction values ​​for each unit pixel.

[0065] In the embodiments provided in this application, the predicted white balance value of each pixel is used to perform color temperature correction on the original image, for example, adjusting the intensity of the red, green, and blue channels in the image to compensate for the color temperature deviation of the light source. The first correction value (from black level correction) and the second correction value (from white balance correction) of each pixel are combined to obtain the final intermediate image features. Through the above processing, the model can dynamically predict the white balance value according to the image content, adapting to different lighting conditions and sensor characteristics. The generated intermediate image features provide high-quality input data for subsequent image processing steps, thereby improving the performance of the entire image processing workflow.

[0066] For example, convolutional layers are used to extract global and local features of the image. After fusing these two types of features, fully connected layers are used to predict the black level / white balance value of each unit pixel in the image, transforming the traditional global black level / white balance value into a local black level / white balance value. The entire process is end-to-end learning, requiring no manual intervention to incorporate prior knowledge, saving the tedious manual parameter tuning and greatly freeing up resources.

[0067] According to the embodiments provided in this application, step S104: performing denoising based on intermediate image features and repairing the denoised intermediate image features to obtain reconstructed image features may include the following specific steps:

[0068] Wavelet decomposition is performed based on the intermediate image features to obtain multiple components of different frequencies. These components include a first component representing noise and edge information in the original image, and a second component representing image information of the original image.

[0069] Add noise to the first component to obtain the features after forward diffusion;

[0070] Prediction is obtained by making predictions based on the features after forward diffusion;

[0071] The predicted noise is denoised to obtain the first denoised component.

[0072] Wavelet reconstruction is performed based on the denoised first and second components to obtain the reconstructed image features.

[0073] In the embodiments provided in this application, wavelet decomposition of intermediate image features can be performed using a wavelet transform layer, decomposing them into multiple components of different frequencies. These components include high-frequency components (i.e., the first component) and low-frequency components (i.e., the second component). The first component contains noise and edge information from the original image, while the second component contains the main image information of the original image. Noise is added to the first component (noise and edge information) to simulate the noise diffusion process, obtaining forward-diffused features, which helps in subsequent noise prediction and denoising processing. Denoising is performed based on the predicted noise to obtain the denoised first component. The sharp edges and details of the image are restored by subtracting the predicted noise. Wavelet reconstruction is performed based on the denoised first and second components, recombinating the components of different frequencies to obtain reconstructed image features used to restore the integrity and details of the image. Through the above processing, noise (high frequency) is physically separated from the main content (low frequency), avoiding detail blurring caused by global denoising. It exhibits excellent adaptability to low light conditions. In extremely low light (0.1 lux), traditional denoising methods are prone to producing artifacts, while diffusion processing combined with generative inpainting can achieve more stable image quality.

[0074] For example, the multiple components (which may also be called multi-frequency sub-bands) obtained by the intermediate image feature decomposition can be represented as LL, LH, HL, HH, where the high-frequency components (LH, HL, HH) contain edges and noise (first component), and the low-frequency component (LL) represents the main content of the image (i.e., second component).

[0075] For example, the wavelet basis for wavelet decomposition can be of various types, such as Haar or Daubechies wavelets.

[0076] For example, a diffusion-based generative network is used for denoising. The diffusion model includes a U-Net architecture, adding skip connections between the encoder and decoder to preserve multi-scale features. Based on diffusion theory, a neural network is used for progressive denoising and generative inpainting, addressing issues such as image color distortion, blurred background edges, and loss of fine-grained features during the denoising process. Using a self-collected database, the restored image has a PSNR greater than 40dB and an SSIM greater than 0.85, conforming to human visual aesthetics. The data distribution is learned by simulating two processes: forward noise diffusion and backward denoising generation. The denoising task includes a forward process and a backward process. The forward process involves progressively adding noise to the image, transforming it into pure noise. The backward process involves training the network to progressively predict and remove noise from the image, restoring the original image.

[0077] Preferably, denoising is performed on the high-frequency components. Controllable noise is added to the high-frequency components (LH, HL, HH) to simulate noise distribution. Gaussian noise can be added stepwise to generate a diffusion sequence, thus obtaining the features after forward diffusion. A U-Net structure is used for noise prediction, minimizing the difference between predicted and actual noise. Predicted noise is removed stepwise according to time steps to obtain the denoised high-frequency components. The denoised components and the retained low-frequency components are then subjected to inverse wavelet transform to obtain the reconstructed image. Since the low-frequency components retain the original illumination and color, and the high-frequency components repair edges and textures, the reconstructed image features can achieve improved image signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0078] For example, other network architectures can also be used to perform denoising, such as ResNet (Residual Network), EfficientNet (Efficient Network), or Vision Transformer (ViT), to adapt to different computational resource requirements and performance demands. ResNet is a deep convolutional neural network that addresses the degradation problem in deep network training by introducing residual connections. It can effectively capture complex features in images and progressively remove noise. The deep structure of ResNet can be used to extract multi-level features of an image, adding a denoising layer after each residual block to progressively reduce the impact of noise. ResNet's skip connections can be used to combine the original features with the denoised features to preserve more image details. EfficientNet is a lightweight network architecture designed to maintain high performance while reducing computational resource requirements. Its performance can be optimized through compound scaling methods (simultaneously scaling depth, width, and resolution). Using EfficientNet as the backbone of a denoising network to extract key image features, specific denoising modules can be added to the top layers of EfficientNet to suit the needs of the denoising task. Vision Transformer is a network based on the Transformer architecture. It segments an image into multiple small blocks and uses a self-attention mechanism to capture long-range dependencies in the image. It leverages the global feature representation capability of the Transformer to recover clear image content from noisy images.

[0079] According to the embodiments provided in this application, predicting noise based on the features after forward diffusion can include the following specific steps:

[0080] The features after forward diffusion are downsampled to obtain multiple downsampled features, each corresponding to a different image scale.

[0081] Multiple downsampled features are upsampled one by one to obtain multiple upsampled features;

[0082] Multiple upsampled features are fused with their corresponding downsampled features to obtain the prediction noise.

[0083] In the embodiments provided in this application, the features after forward diffusion (i.e., features with added noise) are downsampled. Downsampling refers to reducing the spatial dimension of the features, which can be achieved through average pooling, max pooling, or other downsampling techniques. The purpose of downsampling is to extract features of the image at different scales, which helps to identify and separate noise and useful signals. The obtained multiple downsampled features are then upsampled one by one. Upsampling refers to increasing the spatial dimension of the features, which can be achieved through transposed convolution or bilinear interpolation. The purpose is to reconstruct the details of the image, preparing for subsequent noise prediction and removal. The upsampled features are then fused with the original downsampled features. This fusion can be achieved through element-wise addition or concatenation. The purpose of fusion is to combine feature information from different scales to more accurately predict noise. Through the above processing, combined with wavelet transform and multi-scale feature extraction, as well as downsampling and upsampling, the features of the image at different scales can be captured, helping the model to more comprehensively understand the image content.

[0084] The U-Net network is an encoder-decoder structure with powerful feature extraction and multi-scale information fusion capabilities. Figure 2 A schematic diagram of the U-Net for the image processing method according to an embodiment of this application is shown, as follows: Figure 2 As shown, the U-Net network processes images using four downsampling and upsampling operations. The encoder extracts local and global features of the image, while the decoder gradually recovers the image's detailed information. The encoder employs multi-layer convolution and downsampling operations to progressively extract high-level features; the decoder, through upsampling and skip connections, fuses the features from the encoder with the decoder's output, thus preserving the image's detailed information. The flexibility of the U-Net network allows it to adjust the size and depth of the convolutional kernels according to specific imaging requirements, adapting to different noise patterns and image content.

[0085] According to some embodiments provided in this application, the original image is in Bayer channel format, and the image enhancement processing includes demosaicing and image filtering. Image enhancement processing is applied to the reconstructed image features to obtain the target image, which may include the following specific steps:

[0086] A dynamic attention approach is used to perform demosaic processing on each unit pixel to obtain demosaic features, which are then converted from Bayer channel format to RGB channel format.

[0087] Image filtering is performed based on demosaic features to obtain the target image.

[0088] In the embodiments provided in this application, a dynamic attention mechanism is used to perform demosaic processing on each pixel. This dynamic attention mechanism can adapt to different image content and structures, helping to preserve and enhance image details such as edges and textures. It can adapt to various shooting conditions and scenes, making the image clearer and more vivid. The attention point is dynamically adjusted according to different regions of the image content, thereby accurately reconstructing the color information of each pixel, avoiding manual prior knowledge, and exhibiting universal adaptability across different hardware devices. Converting the Bayer format image features to RGB format allows for interpolation of the color information of each pixel to fill missing color channels. After completing the demosaic processing, image filtering is performed based on the obtained RGB format demosaic features to further improve the visual quality of the image.

[0089] For example, the Bayer channel format described above can include a single-channel Bayer format and a four-channel Bayer format. The single-channel Bayer format is monochrome sampling: each pixel records the intensity of only one color, and the missing two colors need to be recovered through demosaicing interpolation. The four-channel Bayer format is an extended version, breaking through the physical limitations of the RGB three-channel by adding new types of pixels (such as W and IR). However, it should be noted that each pixel only covers a single color filter. The newly added channel can be selected as W (full color), indicating no color filter, capturing all visible light, and improving the signal-to-noise ratio. IR (infrared) indicates an extended spectral response range. For the demosaicing processing of the four-channel Bayer matrix, it can be adjusted to calibrate the interpolation weights of neighboring RGB pixels using the W value, which can optimize color reproduction at edges.

[0090] According to the embodiments provided in this application, step S105: performing image filtering based on demosaic features to obtain the target image may include the following specific steps:

[0091] Based on the predetermined image style information, each channel of the RGB channel format with de-mosaic features is individually refined to obtain color correction features;

[0092] The color correction features are nonlinearly transformed by a predetermined gamma value to obtain the target image. The predetermined gamma value is used to adjust the brightness distribution of the image.

[0093] In the embodiments provided in this application, in response to user-defined image style information (such as "retro" or "cool tone"), the RGB channels are individually refined, and a color correction matrix is ​​used to adjust the color mapping relationship, such as enhancing red saturation. Based on the color-corrected image, a non-linear transformation is performed on each pixel value, and the brightness distribution is adjusted using a predetermined gamma value. By combining color stylization and adaptive gamma correction, a target image that both meets the visual needs of the human eye and retains key details can be generated.

[0094] For example, a color correction matrix (CCM) can be used to perform filter processing, combined with prompt (i.e. image style information) and through multi-layer convolution and non-linear activation functions, to refine the RGB three-channel image so that the image output can meet the user's expected image style, i.e., the filter function.

[0095] For example, the above-mentioned filtering process can also be performed using deep learning-based methods, such as using convolutional neural networks (CNNs) or Transformer networks to learn non-linear color mapping relationships.

[0096] For example, the volume tone can be fine-tuned by setting the bias vector according to the image style information, such as boosting the blue / cyan channel and weakening the red channel, so that the processed image has a cool tone effect.

[0097] For example, gamma correction is based on the fact that the human eye's perception of brightness is non-linear, meaning that the human eye is more sensitive to changes in dark areas than changes in bright areas. The gamma value is a parameter of a non-linear transformation used to describe this non-linear relationship.

[0098] The predetermined gamma value is used to adjust the brightness distribution of an image in an exponential function manner. If the gamma value is set <1, the image will brighten because the pixel values ​​in dark areas increase more rapidly. Similarly, if the gamma value is set >1, the image will darken because the pixel values ​​in bright areas decrease more rapidly. By adjusting the gamma value, the overall brightness of the image can be effectively controlled. An appropriate gamma value can enhance the image contrast, making image details more prominent, and making it more suitable for display devices or meeting specific visual effect requirements.

[0099] Based on the above example embodiments, this application also provides an optional implementation method. Figure 3 A schematic framework diagram of the image processing method according to an embodiment of this application is shown, such as... Figure 3 The diagram includes: image correction (black level combined with white balance correction), denoising network, depigmentation, and color and brightness adjustment.

[0100] In an optional embodiment of this application, black level correction is a crucial step in image processing, used to eliminate the inherent noise generated by the CMOS sensor under dark conditions. Black level is defined as the level value output by the CMOS sensor when the pixel value is 0. Due to limitations in sensor hardware characteristics, even in completely dark environments, the CMOS sensor will still output a non-zero baseline signal, called the black level. Without correction, this baseline signal will cause an overall brightness shift in the image, affecting the accuracy of subsequent processing steps. By introducing learnable black level correction parameters, it is possible to dynamically adapt to the hardware characteristics of different sensors, significantly improving the accuracy and robustness of the correction.

[0101] In an optional embodiment of this application, white balance is used to correct color deviations caused by differences in the color temperature of light sources. Different light sources (such as sunlight, fluorescent lamps, incandescent lamps, etc.) have different color temperatures, which can lead to unnatural color shifts in images. For example, images taken under incandescent lamps may appear yellowish, while images taken under fluorescent lamps may appear bluish. This invention, by dynamically adjusting the white balance parameters, can adapt to color temperature changes under different lighting conditions, ensuring the authenticity and consistency of image colors.

[0102] The network structure of the image correction part can be obtained by using a predetermined combination of fully connected layers and activation layers.

[0103] In the denoising network of the optional implementation of this application, the denoising network is a key module in the image signal processing pipeline, used to separate noise signals from the original Bayer array image while preserving the image's detailed information. The denoising network consists of a U-Net and a wavelet transform (including wavelet decomposition and wavelet reconstruction), combining the advantages of the spatial and frequency domains, significantly improving the denoising effect.

[0104] The U-Net network extracts local and global features from an image through an encoder and gradually recovers the image's detailed information through a decoder. The encoder uses multi-layer convolution and downsampling operations to progressively extract high-level features from the image; the decoder, on the other hand, fuses the features from the encoder with the decoder's output through upsampling and skip connections, thereby preserving the image's detailed information. The flexibility of the U-Net network allows it to adjust the size and depth of the convolution kernels according to specific imaging requirements, thus adapting to different noise patterns and image content.

[0105] To further improve denoising performance, wavelet transform was introduced as an auxiliary tool. Wavelet transform is a time-domain to frequency-domain transformation algorithm that decomposes an image into sub-bands of different frequencies, thereby separating noise signals from image details. Through wavelet transform, the high-frequency components (typically containing noise and edge information) and low-frequency components (containing the main content of the image) are effectively separated. By utilizing wavelet transform to extract high-frequency components and combining it with the multi-scale feature extraction capabilities of the U-Net network, dual optimization in both the frequency and spatial domains is achieved, significantly improving the accuracy and robustness of denoising.

[0106] In the demosaic section of the optional implementation of this application, the demosaic module is a key step in the image signal processing pipeline, which can convert the single-channel Bayer array image output by the CMOS sensor into an RGB three-channel image format. Since each pixel in the Bayer array image contains only one color information (red, green, or blue), traditional demosaic methods typically estimate the missing color information through interpolation algorithms. However, this interpolation method is prone to information loss, especially in areas with edges and complex textures, easily resulting in artifacts and color distortion.

[0107] To address this issue, this alternative implementation proposes a demosaic method based on convolutional neural networks. This method utilizes convolutional operations to deepen channels and extract features, thereby preserving detailed information in the original image. Specifically, the demosaic network progressively recovers missing color information from the Bayer array image through multiple layers of convolution and nonlinear activation functions. Compared to traditional interpolation methods, convolutional neural networks can better capture the local structure and global contextual information of an image, thus significantly improving the accuracy and visual effect of color reconstruction.

[0108] In an optional embodiment of this application, the color correction matrix is ​​used to adjust the color representation of the image according to ambient lighting conditions and convert it to the standard sRGB color gamut. Gamma correction, on the other hand, improves the visual effect of the image by balancing the brightness variations of the image through nonlinear mapping. These correction modules are optimized through end-to-end training and can automatically adapt to different imaging scenarios, thereby improving the overall image quality.

[0109] Through the collaborative work of the aforementioned multiple modules, this optional implementation not only effectively suppresses noise and restores color information but also preserves image detail features, significantly improving the overall image quality in low-light environments. Combining the U-Net network and wavelet transform denoising algorithms, the noise suppression effect is significantly improved through dual optimization in both the frequency and spatial domains. The multi-scale feature extraction capability of the U-Net network combined with the high-frequency component separation capability of wavelet transform effectively preserves image details while removing complex noise in low-light environments.

[0110] An end-to-end image processing algorithm is proposed, which, unlike traditional single denoising networks, integrates and optimizes multiple modules such as Black Level Correction (BLC), White Balance (WB), Denoising Network (DN), Demosaic (DM), Color Correction Matrix (CCM), and Gamma Correction (GC). Through end-to-end training and learnable parameters, it can dynamically adapt to different hardware characteristics and lighting conditions, improving the overall quality and efficiency of image processing in low-light environments. It can effectively capture fine-grained features of images and enhance color quality.

[0111] Corresponding to the application scenarios and methods provided in the embodiments of this application, Figure 4A schematic diagram of an image processing apparatus according to an embodiment of this application is shown, such as... Figure 4 As shown in the illustration, this application also provides an image processing apparatus, including:

[0112] The feature extraction module 401 is used to extract global and local features of the original image;

[0113] The black level processing module 402 is used to determine the black level value of each unit pixel in the original image based on global and local features. The black level value is generated by the device noise of the sensor that acquired the original image.

[0114] The correction module 403 is used to perform image correction based on the black level value of each unit pixel to obtain intermediate image features;

[0115] The denoising module 404 is used to perform denoising based on intermediate image features and repair the denoised intermediate image features to obtain reconstructed image features.

[0116] The image enhancement module 405 is used to perform image enhancement processing on the features of the reconstructed image to obtain the target image.

[0117] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0118] Figure 5 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can run on the processor 502. When the processor 502 executes the computer program, it implements the method described in the above embodiments. The number of memories 501 and processors 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and exchanging data.

[0119] In practical implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to complete communication. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0120] Optionally, in a specific implementation, if the memory 501, processor 502 and communication interface 503 are integrated on a single chip, the memory 501, processor 502 and communication interface 503 can communicate with each other through an internal interface.

[0121] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0122] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0123] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0124] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0125] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0126] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0127] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0128] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0129] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0130] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0131] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0132] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0134] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: extracting global features and local features of an original image; determining a black level value of each unit pixel in the original image based on the global features and the local features, the black level value being generated by device noise of a sensor that acquires the original image; performing image correction according to the black level value of each unit pixel to obtain intermediate image features; performing denoising processing based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features; performing image enhancement processing on the reconstructed image features to obtain a target image; wherein determining the black level value of each unit pixel in the original image based on the global features and the local features comprises: fusing the global features and the local features through a fully connected layer to obtain fused features; inputting the fused features into a black level prediction layer to determine the black level value of each unit pixel based on the characteristics of the sensor; The method further comprises: inputting the fused features into a white balance prediction layer to determine a white balance value of each unit pixel; performing color temperature correction based on the white balance value of each unit pixel to obtain a second correction value of each unit pixel; and obtaining the intermediate image features based on the first correction value of each unit pixel and the second correction value.

2. The method of claim 1, wherein, The image correction based on the black level value of each unit pixel to obtain intermediate image features comprises: subtracting the corresponding black level value from the original pixel value of each unit pixel to obtain a first correction value of each unit pixel; generating the intermediate image features based on the first correction value of each unit pixel.

3. The method of claim 1, wherein, The denoising processing based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features, comprises: performing wavelet decomposition based on the intermediate image features to obtain a plurality of components of different frequencies, the plurality of components including a first component representing noise and edge information in the original image, and a second component representing image information of the original image; adding noise to the first component to obtain a forward diffusion feature; performing prediction based on the forward diffusion feature to obtain predicted noise; performing denoising processing based on the predicted noise to obtain a denoised first component; performing wavelet reconstruction based on the denoised first component and the second component to obtain the reconstructed image features.

4. The method of claim 3, wherein, The prediction based on the forward diffusion feature to obtain predicted noise comprises: performing down-sampling processing on the forward diffusion feature to obtain a plurality of down-sampled features, the plurality of down-sampled features corresponding to different image scales respectively; performing up-sampling processing on the plurality of down-sampled features one by one to obtain a plurality of up-sampled features; fusing the plurality of up-sampled features with the corresponding down-sampled features to obtain the predicted noise.

5. The method according to any one of claims 1 to 4, characterized in that, The original image is in a Bayer channel format, the image enhancement processing includes demosaicing processing and image filter processing, and the image enhancement processing on the reconstructed image features to obtain a target image comprises: The dynamic attention mode is used to perform demosaicing on each unit pixel to obtain a demosaicing feature converted from the Bayer channel format to the RGB channel format; Image filter processing is performed based on the demosaicing feature to obtain a target image.

6. The method of claim 5, wherein, The image filter processing based on the demosaicing feature to obtain the target image comprises: Each channel of the RGB channel format of the demosaicing feature is respectively retouched based on predetermined image style information to obtain a color correction feature; The color correction feature is subjected to nonlinear transformation through a predetermined gamma value to obtain the target image, and the predetermined gamma value is used to adjust image brightness distribution. 7.An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1 to 6 when executing the computer program. 8.A computer readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.

Images