Image processing method, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention provides an image processing method, electronic equipment and a computer readable storage medium. The method relates to the technical field of computer vision, and comprises the following steps: extracting global features and local features of an original image; based on the global features and the local features, determining a black level value of each unit pixel in the original image, the black level value being generated by equipment noise of a sensor collecting the original image; performing image correction according to the black level value of each unit pixel to obtain an intermediate image feature; performing denoising processing based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features; and performing image enhancement processing on the reconstructed image features to obtain a target image. According to the technical scheme of the embodiment of the invention, the method solves a technical problem that the quality of an image processed under a low-light condition is not ideal in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of low-light imaging, existing technologies mostly rely on a single lighting condition or noise model for image enhancement: users optimize image quality by adjusting parameters or selecting preset modes. However, these solutions only optimize for specific lighting or noise types, making them difficult to adapt to complex and changing environments. Furthermore, existing enhancement algorithms lack the ability to perceive detailed features, especially in low-light conditions. Text information and edge details are easily lost, resulting in degraded image quality. This limits the user's control over the image enhancement effect and reduces the applicability of image processing algorithms in low-light conditions. Summary of the Invention

[0003] The embodiments of the present application provide an image processing method, an electronic device, and a computer-readable storage medium to alleviate or solve the technical problem in the related art of unsatisfactory image quality after processing under low-light conditions.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising: Extract global features and local features of the original image; Determine the black level value of each unit pixel in the original image based on the global features and the local features, where the black level value is generated by the device noise of the sensor that collects the original image; Perform image correction based on the black level value of each unit pixel to obtain intermediate image features; Denoising is performed based on the intermediate image features, and the denoised intermediate image features are repaired to obtain reconstructed image features; Perform image enhancement processing on the reconstructed image features to obtain the target image.

[0005] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any method of the embodiment of the present application when executing the computer program.

[0006] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0007] Based on the image processing method of the first aspect above, this application has at least the following beneficial effects or advantages: The embodiments of this application provide a novel low-light enhancement framework that aims to introduce additional control information such as noise suppression and detail recovery into the enhancement process based on the multimodal fusion of lighting conditions, noise types, and detail features, thereby achieving high-quality image enhancement under multimodal control. This end-to-end design not only significantly improves the overall quality of image processing, but also dynamically adapts to different hardware characteristics and lighting conditions, resolving the performance bottleneck caused by the discrete modules in traditional methods. Compared with the solutions provided in related technologies, it can provide better image processing capabilities in low-light conditions.

[0008] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0010] Figure 1 A flowchart of an image processing method according to an embodiment of the present application is shown; Figure 2 A U-Net schematic diagram of the image processing method according to an embodiment of the present application is shown; Figure 3 A schematic framework diagram of an image processing method according to an embodiment of the present application is shown; Figure 4 A schematic diagram of an image processing device according to an embodiment of the present application is shown; Figure 5 A block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0011] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0012] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0013] The following terms will be used in the following text: Image signal processing algorithm is the key technology to convert the original Bayer array image captured by CMOS sensor into high-quality RGB image.

[0014] CMOS (Complementary Metal-Oxide-Semiconductor) sensors use complementary MOSFET (Metal Oxide Semiconductor Field-Effect Transistor) technology to build logic gates and other electronic circuits. They are composed of multiple photosensitive elements and can convert light signals into electrical signals.

[0015] The Bayer array is a common color filter array (CFA).

[0016] Black level correction (BLC) is defined as the level at which a CMOS sensor outputs a pixel value of zero. Due to sensor hardware limitations, even in complete darkness, CMOS sensors still output a non-zero baseline signal, known as the black level. Without correction, this baseline signal can shift the overall brightness of the image, affecting the accuracy of subsequent processing steps.

[0017] White balance (WB) is an image processing technique designed to correct color deviations in images, ensuring a true white appearance under different light sources. By adjusting the intensity ratio of the red, green, and blue channels in an image and compensating for the color temperature of the light source, colors appear more natural and accurate, restoring true colors and improving image visual quality.

[0018] Gamma correction (GC) is a technique used to adjust image brightness and contrast. It changes the brightness level of an image by applying a nonlinear transformation so that the image display effect is more consistent with the human eye's perception of light.

[0019] In the related art, existing image signal processing methods are mainly divided into two categories, including: data-driven and model-driven.

[0020] Data-driven methods achieve better image restoration results by effectively utilizing limited data. The core of these methods lies in data augmentation, which improves model performance by generating diverse training data. However, the main limitation of data-driven methods is their lack of generalization ability. The effectiveness of data augmentation is often closely related to the hardware device, resulting in significant performance differences across different hardware platforms. Furthermore, these methods are highly dependent on data quality and scale, making them difficult to achieve ideal results in scenarios where data is scarce.

[0021] Model-driven approaches focus on improving image processing performance by optimizing model structures and algorithms. However, model-based approaches typically perform 1 to 2 points lower than data-driven approaches in terms of Peak Signal-to-Noise Ratio (PSNR) and SSIM (Structural Similarity Index), and their overall performance is still slightly inferior to that of data-driven approaches.

[0022] The following describes in detail the technical solution of this application and how it solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of this application in detail with reference to the accompanying drawings.

[0023] Figure 1 FIG. 1 shows a flow chart of an image processing method according to an embodiment of the present application. Figure 1 As shown, the method may include steps S101 to S105.

[0024] Step S101: extracting global features and local features of the original image; Step S102: determining a black level value of each unit pixel in the original image based on the global features and the local features, where the black level value is generated by device noise of a sensor that acquires the original image; Step S103: performing image correction according to the black level value of each unit pixel to obtain intermediate image features; Step S104: performing denoising based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features; Step S105: performing image enhancement processing on the reconstructed image features to obtain a target image.

[0025] Exemplarily, the aforementioned execution entities may be of various types, such as an image processing unit integrated into an edge-side or cloud-based server. Edge-side integration may involve integrating an image acquisition device and an image processing unit together in a mobile terminal, such as in a mobile phone. Alternatively, the image acquisition device may be separately integrated into the image acquisition device or into a connected host computer. The aforementioned image acquisition device may comprise various photoelectric conversion devices, such as CMOS devices, CCD (Charge-Coupled Device) sensors, EMCCD (Electron Multiplying Charge-Coupled Device) sensors, BSI-CMOS (Back-Side Illuminated CMOS) sensors, and the like. Cloud-based server integration may involve transmitting the original image from the image acquisition device and then performing image processing in the cloud.

[0026] For example, the black level represents a form of noise generated by a photoelectric acquisition device in the absence of light. In some embodiments, the black level can be replaced by a dark current, that is, a current generated by thermal excitation inside the image sensor.

[0027] Illustratively, the above processing flow is applied in an extreme low-light environment and can maintain good image quality at a lower light intensity than the prior art, for example, an illumination below 0.1 lux.

[0028] In the embodiments provided in the present application, global features and local features are extracted from the original image. Global features include the overall information of the image, while local features focus on specific areas of the image, such as edges and textures. Based on the extracted global and local features, a black level value is determined for each pixel unit in the original image. The black level value reflects the device noise generated by the image sensor when acquiring the image. The image is corrected using the determined black level value of each pixel to eliminate the influence of the device noise, thereby obtaining intermediate image features, thereby improving the dynamic range and signal-to-noise ratio of the image under low-light conditions. The intermediate image features are denoised to further reduce the noise in the image. Since distortion may be introduced during the image processing process, especially under low-light conditions, the denoised intermediate image features need to be repaired to restore the details and structure of the image to obtain reconstructed image features. The reconstructed image features are subjected to image enhancement processing to improve the visual effect of the image and obtain the final target image.

[0029] By predicting the black level value of each unit pixel, sensor noise is eliminated, avoiding the problem of image correction inaccuracy caused by traditional global correction. Combined with effective noise separation and image repair, edges and details are retained, the clarity and naturalness of the image are significantly improved to achieve better image processing effects.

[0030] Exemplarily, the extraction process in step S101 can be performed in a variety of ways. For global feature extraction, global average pooling or global maximum pooling can be used to compress the feature map into vector dimensions, capturing the overall image information. Global average pooling takes the average of the spatial dimensions of each channel in the feature map. Global maximum pooling takes the maximum spatial value of each channel. For local feature extraction, convolution or local attention can be used. A convolution layer can contain multiple convolution kernels, each responsible for detecting different types of features, such as edges, textures, and corners. Convolution is performed by sliding the convolution kernel over the input image. At each location, the convolution kernel is element-wise multiplied by the local region of the image and the sum is calculated to generate an output value (e.g., a pixel in the feature map). Local attention, on the other hand, divides the image into non-overlapping or overlapping local windows (e.g., 8x8 pixels). Self-attention is calculated within each window, focusing on the local region and avoiding over-smoothing of low-light noise caused by global attention.

[0031] According to the embodiment provided by the present application, in step S102: determining the black level value of each unit pixel in the original image based on the global features and the local features, the following specific steps may be included: The global features and local features are fused through the fully connected layer to obtain the fused features; The fused features are input into the black level prediction layer to determine the black level value of each unit pixel based on the sensor features.

[0032] In some embodiments provided in this application, for the low-light image processing process, global features characterize the overall illumination and color distribution of the image, and local features capture pixel-level noise and edge details. The global features and local features are spliced ​​and input into the fully connected layer to generate fused features. Global features provide macroscopic regularities of sensor noise, such as the distribution of black levels in the original image, and local features adapt to pixel-level differences. By fusing local features, pixel-by-pixel black level prediction is achieved, thereby improving the signal-to-noise ratio in low-light areas. It should be noted that for low-brightness imaging conditions, when black level noise is dominant, collective local black level correction can help restore more shadow details.

[0033] For example, by learning weights and biases, the global and local features are integrated into a unified feature vector, i.e., a fused feature. Optionally, a weighted summation of features is performed, where the weights are learned by the network during training to maximize the expressive power of the features.

[0034] Exemplarily, the black level prediction layer can be formed by combining one or more linear layers and activation layers. The main function of a linear layer is to map input data to a new space, achieving linear combination of features. In a neural network, it is used to connect different layers, transfer and transform feature information, and is also known as a fully connected layer. The activation layer can be a ReLU layer (Rectified Linear Unit layer), a Sigmoid layer, or a Tanh layer (Hyperbolic Tangent layer). The ReLU layer increases the expressive power of the neural network by introducing nonlinearity. The Sigmoid layer maps input values ​​between 0 and 1 and interprets the output as a probability value. The Tanh layer maps input values ​​between -1 and 1. Similar to the Sigmoid layer, the Tanh layer increases the expressive power of the neural network through nonlinear transformation. The output range of the Tanh function is wider and centrally symmetric.

[0035] To facilitate understanding, let's use an example to initialize the weights and biases in the fully connected layer. Weights are typically initialized to random values ​​to break symmetry and prevent all neurons from learning the same features. Biases are initialized to 0 or a small random value. During the forward propagation phase, global and local features are passed as input to the fully connected layer. Each neuron calculates the weighted sum of its inputs and adds the bias, generating an output through an activation function (such as ReLU). The loss between the network output and the true label is calculated. This loss can be implemented in various forms, such as mean squared error or cross-entropy loss. Backpropagation is used to calculate the gradient of the loss function with respect to the weights and biases, calculating the contribution of each weight and bias to the final loss. An optimization algorithm (such as gradient descent or Adam) is used to update the weights and biases based on the calculated gradients.

[0036] In this way, the neural network automatically learns how to maximize the expressive power of features by adjusting weights and biases, effectively integrating global and local features to form a unified feature vector. This learning process enables the network to capture complex patterns and relationships in the input data, improving the model's predictive performance.

[0037] For example, in low-light image processing, sensor characteristics directly influence black level calculation. These sensor characteristics can include various factors, such as sensitivity, rated sensor operating temperature, and dark current (black level) distribution. High sensitivity results in more pronounced noise and a stronger black level. The relationship between the sensor's rated operating temperature and device noise can be expressed as follows: increasing temperature exacerbates thermal noise (for example, CMOS dark current has an exponential relationship with temperature). These characteristics can also be expressed as sensor model, with different models exhibiting varying levels of floor noise.

[0038] Compared with the global black level correction method, the advantage of the pixel-by-pixel correction method is that each pixel in the image sensor may have a different dark current level due to manufacturing differences, temperature changes or circuit noise. The non-uniformity in the sensor manufacturing process causes different pixels to respond inconsistently to light, resulting in inconsistent black levels.

[0039] For example, the black level prediction layer can adaptively predict black levels based on sensor characteristics, ensuring that the model can capture and learn sensor characteristics and their impact on black levels. For example, data from different sensor types, different lighting conditions, and different environments can be collected. Ensure that the dataset contains sufficient diversity to cover all possible application scenarios. Annotate the true black level value corresponding to each image sample through laboratory measurements or using known reference images. Sensor-specific characteristics such as sensor model, manufacturing batch, and temperature readings can be integrated into the model as part of the input. In addition to image data, sensor metadata can be used as an additional input modality to help the model understand the source and conditions of the data. Furthermore, if available data is limited, a transfer learning strategy can be used, starting with a black level prediction layer pre-trained on a large amount of data and fine-tuning it for data from a specific sensor model. The model's performance can be evaluated under different sensors and conditions to ensure that the black level prediction layer has good generalization and adaptability. Preferably, in practical applications, the black level prediction layer can also adopt an online learning strategy to adjust its predictions based on real-time feedback to adapt to dynamic changes in sensor characteristics.

[0040] According to the embodiment provided by the present application, in the above step S103, image correction is performed according to the black level value of each unit pixel to obtain the intermediate image feature, which may include the following specific steps: Subtracting the corresponding black level value from the original pixel value of each unit pixel to obtain a first correction value for each unit pixel; Based on the first correction value of each unit pixel, an intermediate image feature is generated.

[0041] In the embodiments provided herein, for each unit pixel in an image, the corresponding black level value is subtracted from its original pixel value to obtain a first correction value. This is used to eliminate noise generated by the image sensor in dark conditions, namely, the black level signal. Based on this first correction value for each unit pixel, an intermediate image feature is generated. This intermediate image feature represents the corrected image data and forms the basis for further image processing (such as denoising and enhancement).

[0042] According to the embodiment provided in this application, the method further includes the following specific steps: Input the fused features into the white balance prediction layer to determine the white balance value of each unit pixel; Performing color temperature correction based on the white balance value of each unit pixel to obtain a second correction value for each unit pixel; An intermediate image feature is obtained based on the first correction value and the second correction value of each unit pixel.

[0043] In the embodiments provided herein, the predicted white balance value for each pixel is used to perform color temperature correction on the original image. For example, the intensity of the red, green, and blue channels in the image is adjusted to compensate for color temperature deviations of the light source. The first correction value (derived from black level correction) and the second correction value (derived from white balance correction) for each pixel are combined to generate the final intermediate image features. Through this processing, the model can dynamically predict white balance values ​​based on image content, adapting to varying lighting conditions and sensor characteristics. The generated intermediate image features provide high-quality input data for subsequent image processing steps, thereby improving the performance of the entire image processing pipeline.

[0044] For example, a convolutional layer extracts global and local image features. After fusing these two features, a fully connected layer predicts the black level and white balance values ​​for each pixel in the image, converting traditional global black level and white balance values ​​into local black level and white balance values. This entire process is end-to-end learning, eliminating the need for manual intervention and prior knowledge, significantly freeing up resources.

[0045] According to the embodiment provided by the present application, in step S104: performing denoising based on the intermediate image features and repairing the denoised intermediate image features to obtain reconstructed image features, the following specific steps may be included: performing wavelet decomposition based on the intermediate image features to obtain multiple components of different frequencies, the multiple components including a first component representing noise and edge information in the original image, and a second component representing image information of the original image; Add noise to the first component to obtain the features after forward diffusion; Make predictions based on the features after forward diffusion to obtain prediction noise; Performing denoising processing based on the predicted noise to obtain a denoised first component; Wavelet reconstruction is performed based on the denoised first and second components to obtain reconstructed image features.

[0046] In the embodiments provided herein, wavelet decomposition of intermediate image features can be performed using a wavelet transform layer, decomposing them into multiple components of different frequencies. These components include a high-frequency component (i.e., the first component) and a low-frequency component (i.e., the second component). The first component contains noise and edge information from the original image, while the second component contains the primary image information. Noise is added to the first component (noise and edge information) to simulate the noise diffusion process, resulting in forward diffusion features that facilitate subsequent noise prediction and denoising. Denoising is performed based on the predicted noise to obtain the denoised first component. By subtracting the predicted noise, the clear edges and details of the image are restored. Wavelet reconstruction is performed based on the denoised first and second components, recombining the components of different frequencies to obtain reconstructed image features, which are used to restore the integrity and details of the image. This process physically separates the noise (high frequency) from the primary image content (low frequency), avoiding the blurring of details caused by global denoising. It has good adaptability to low light conditions. In extremely low light (0.1 lux), traditional denoising is prone to artifacts, while diffusion processing combined with generative restoration can achieve more stable image quality.

[0047] Exemplarily, the multiple components (which may also be called multi-frequency subbands) obtained by decomposing the intermediate image features can be expressed as LL, LH, HL, and HH, where the high-frequency components (LH, HL, and HH) contain edges and noise (the first component), and the low-frequency component (LL) represents the main content of the image (i.e., the second component).

[0048] For example, the wavelet basis used in the wavelet decomposition process may be various, such as Haar or Daubechies wavelet.

[0049] For example, a Diffusion-based generative network is used for denoising. The Diffusion model includes a U-Net architecture, which adds jump connections between the encoder and decoder to retain multi-scale features. Relying on the diffusion theory, a neural network is used to gradually reduce noise and perform generative restoration. During the denoising process, problems such as image color distortion, blurred background edges, and loss of fine-grained features are solved. Relying on a self-collected database, the PSNR of the restored image is greater than 40dB and the SSIM is greater than 0.85, which is in line with the visual aesthetics of the human eye. The data distribution is learned by simulating the two processes of forward noise diffusion and reverse denoising generation. The denoising task includes a forward process and a reverse process. The forward process involves gradually adding noise to the image to convert it into pure noise. The reverse process involves training the network to gradually predict and remove noise from the noise to restore the image.

[0050] Optimally, high-frequency components are denoised by adding controllable noise to them (LH, HL, HH) to simulate noise distribution. Gaussian noise can be gradually added to generate a diffusion sequence, thus obtaining features after forward diffusion. A U-Net architecture is used for noise prediction, minimizing the difference between predicted noise and actual noise. The predicted noise is gradually removed over time to obtain denoised high-frequency components. The denoised components and the retained low-frequency components are then subjected to an inverse wavelet transform to obtain a reconstructed image. Because the low-frequency components preserve the original lighting and color, while the high-frequency components restore edges and texture, the reconstructed image features can improve the image signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0051] For example, other network structures can be used to perform denoising, such as ResNet (Residual Network), EfficientNet (Efficient Network), or Vision Transformer (ViT), to adapt to different computing resource and performance requirements. ResNet is a deep convolutional neural network that addresses the degradation problem in deep network training by introducing residual connections. It can effectively capture complex features in images and gradually remove noise. The deep structure of ResNet can be used to extract multi-level image features. Denoising layers are added after each residual block to gradually reduce the impact of noise. ResNet's skip connections combine original features with denoised features to preserve more image details. EfficientNet is a lightweight network structure designed to maintain high performance while reducing computing resource requirements. It can optimize network performance through compound scaling (simultaneously scaling depth, width, and resolution). Using EfficientNet as the backbone of the denoising network to extract key image features, specialized denoising modules can be added to the top layer of EfficientNet to meet the requirements of the denoising task. Vision Transformer is a network based on the Transformer architecture. It divides the image into multiple small blocks and uses the self-attention mechanism to capture long-distance dependencies in the image. It leverages the global feature representation capability of the Transformer to restore clear image content from noisy images.

[0052] According to the embodiment provided in the present application, performing prediction based on the features after forward diffusion to obtain predicted noise may include the following specific steps: The features after forward diffusion are downsampled to obtain multiple downsampled features, each of which corresponds to a different image scale. Upsampling multiple downsampled features one by one to obtain multiple upsampled features; Multiple upsampled features are fused with the corresponding downsampled features to obtain prediction noise.

[0053] In the embodiment provided in the present application, the features after forward diffusion (i.e., features with added noise) are downsampled. Downsampling refers to reducing the spatial dimension of the features, which can be achieved through average pooling, maximum pooling or other downsampling techniques. The purpose of the downsampling process is to extract the features of the image at different scales, which helps to identify and separate noise and useful signals. The multiple downsampled features obtained are upsampled one by one. Upsampling refers to increasing the spatial dimension of the features, which can be achieved through transposed convolution and bilinear interpolation. The purpose is to reconstruct the details of the image and prepare for subsequent noise prediction and removal. The upsampled features are fused with the original downsampled features. The above fusion process can be achieved by element-by-element addition or splicing. The purpose of fusion is to combine feature information at different scales to more accurately predict noise. Through the above processing, combined with wavelet transform and multi-scale feature extraction, as well as downsampling and upsampling, the features of the image at different scales can be captured, which helps the model to understand the image content more comprehensively.

[0054] The U-Net network is an encoder-decoder structure with powerful feature extraction and multi-scale information fusion capabilities. Figure 2 A U-Net schematic diagram of the image processing method according to an embodiment of the present application is shown as follows: Figure 2 As shown in the figure, the U-Net network uses a four-fold downsampling and upsampling process. The encoder extracts local and global image features and the decoder gradually restores image details. The encoder uses multiple layers of convolution and downsampling to gradually extract high-level image features. The decoder, through upsampling and skip connections, fuses the encoder features with the decoder output, preserving image details. The U-Net network's flexibility enables the size and depth of the convolution kernel to be adjusted according to specific imaging requirements, adapting to different noise patterns and image content.

[0055] According to some embodiments provided by the present application, the original image is in Bayer channel format, and the image enhancement processing includes demosaicing processing and image filter processing. The image enhancement processing is performed on the reconstructed image features to obtain the target image, which may include the following specific steps: Using dynamic attention method, each unit pixel is demosaiced to obtain demosaic features, and the demosaic features are converted from Bayer channel format to RGB channel format; Image filter processing is performed based on the demosaicing features to obtain the target image.

[0056] In the embodiment provided in the present application, a dynamic attention mechanism is used to perform demosaicing on each pixel. The dynamic attention mechanism can adapt to different image contents and structures, helps to retain and enhance image details such as edges and textures, and can adapt to various shooting conditions and scenes, making the image clearer and more vivid. The focus is dynamically adjusted according to different areas of the image content to accurately reconstruct the color information of each pixel, avoiding artificial priors and being universally adaptable on different hardware devices. The image features in the Bayer format are converted into RGB format, and the color information of each pixel can be interpolated to fill the missing color channels. After the demosaicing is completed, image filter processing is performed based on the demosaiced features in the obtained RGB format to further improve the visual quality of the image.

[0057] For example, the aforementioned Bayer channel formats can include single-channel Bayer and four-channel Bayer formats. The single-channel Bayer format uses monochrome sampling: each pixel records the intensity of only one color, and the missing two colors are restored through interpolation using a demosaicing algorithm. The four-channel Bayer channel format is an extension of the physical limitations of the three RGB channels by adding new pixel types (such as W and IR). However, it should be noted that each pixel only covers a single color filter. The newly added channel can be selected as W (full color), indicating no color filter, capturing all visible light and improving the signal-to-noise ratio. IR (infrared) indicates an extended spectral response range. The demosaicing process for the four-channel Bayer matrix can be adjusted to use the W value to calibrate the interpolation weights of adjacent RGB pixels, which can optimize color reproduction at edges.

[0058] According to the embodiment provided by the present application, in step S105: performing image filter processing based on the demosaicing feature to obtain a target image, the specific steps may include: Based on the predetermined image style information, each channel of the RGB channel format of the demosaiced feature is polished separately to obtain a color correction feature; The color correction feature is nonlinearly transformed by a predetermined gamma value to obtain a target image. The predetermined gamma value is used to adjust the image brightness distribution.

[0059] In the embodiments provided herein, RGB channels can be individually retouched in response to user-defined image style information (e.g., "retro" or "cold tones"), employing a color correction matrix to adjust color mapping relationships, such as enhancing red saturation. Based on the color-corrected image, each pixel value is subjected to a nonlinear transformation, and the brightness distribution is adjusted using a predetermined gamma value. By combining color stylization with adaptive gamma correction, a target image can be generated that both meets the visual needs of the human eye and preserves key details.

[0060] For example, a color correction matrix (CCM) can be used to perform filter processing, combined with prompt (i.e., image style information) and through multi-layer convolution and nonlinear activation functions, the RGB three-channel image can be polished so that the image can output an image style that meets the user's expectations, i.e., the filter function.

[0061] Exemplarily, the above-mentioned filter processing may also be performed using a deep learning-based method, such as using a convolutional neural network (CNN) or a Transformer network to learn a nonlinear color mapping relationship.

[0062] For example, the overall color tone can be fine-tuned by setting a bias vector according to the image style information, for example, by enhancing the blue / cyan channel and weakening the red channel, so that the processed image presents a cool color effect.

[0063] For example, gamma correction is based on the fact that the human eye's perception of brightness is nonlinear, meaning that the human eye is more sensitive to changes in dark areas than changes in bright areas. The gamma value is a parameter of a nonlinear transformation used to describe this nonlinear relationship.

[0064] The predefined gamma value is used to adjust the image's brightness distribution in an exponential manner. If the gamma value is set to <1, the image will appear brighter due to the faster increase in pixel values ​​in dark areas. Similarly, if the gamma value is set to >1, the image will appear darker due to the faster decrease in pixel values ​​in bright areas. By adjusting the gamma value, you can effectively control the overall brightness of the image. An appropriate gamma value can enhance image contrast, making image details more prominent, making it more suitable for display devices or meeting specific visual requirements.

[0065] According to the above exemplary embodiments, the present application also provides an optional implementation manner: Figure 3 A schematic diagram of the image processing method according to an embodiment of the present application is shown in FIG. Figure 3 The image shown includes: image correction (black level combined with white balance correction), denoising network, demosaicing, and color and brightness adjustment.

[0066] In an optional embodiment of the present application, black level correction is a key step in image processing, which is used to eliminate the inherent noise generated by the CMOS sensor under no light conditions. The black level is defined as the level value output by the CMOS sensor when the pixel value is 0. Due to the limitations of the sensor hardware characteristics, even in a completely dark environment, the CMOS sensor will still output a non-zero baseline signal, called the black level. If not corrected, this baseline signal will cause the overall brightness of the image to shift, affecting the accuracy of subsequent processing steps. By introducing learnable black level correction parameters, it is possible to dynamically adapt to the hardware characteristics of different sensors, significantly improving the accuracy and robustness of the correction. In an optional embodiment of the present application, white balance is used to correct color deviations caused by differences in light source color temperature. Different light sources (such as sunlight, fluorescent lighting, and incandescent lighting) have different color temperatures, which can cause unnatural color shifts in images. For example, images captured under incandescent lighting may appear yellowish, while images captured under fluorescent lighting may appear bluish. By dynamically adjusting white balance parameters, the present invention can adapt to color temperature variations under different lighting conditions, ensuring the authenticity and consistency of image colors.

[0067] The network structure of the above-mentioned image correction part can be obtained by using a predetermined combination of a fully connected layer and an activation layer.

[0068] In the denoising network of an optional embodiment of this application, the denoising network is a key module in the image signal processing pipeline, used to separate noise signals from the original Bayer array image while preserving image detail information. The denoising network consists of a U-Net and a wavelet transformer (including wavelet decomposition and wavelet reconstruction), combining the advantages of both the spatial and frequency domains to significantly improve denoising results.

[0069] The U-Net network extracts local and global image features through the encoder and gradually restores image details through the decoder. The encoder uses multiple layers of convolution and downsampling to gradually extract high-level image features. The decoder, through upsampling and skip connections, fuses the encoder features with the decoder output, preserving image details. The U-Net network's flexibility enables it to adjust the size and depth of the convolution kernel according to specific imaging requirements, adapting to different noise patterns and image content.

[0070] To further enhance denoising, the wavelet transform was introduced as a supporting tool. The wavelet transform, a time-to-frequency domain transformation algorithm, decomposes an image into subbands of varying frequencies, thereby separating noise signals from image details. The wavelet transform effectively separates the high-frequency components of an image (typically containing noise and edge information) from the low-frequency components (containing the main image content). By extracting the high-frequency components using the wavelet transform and combining it with the multi-scale feature extraction capabilities of the U-Net network, dual optimization in both the frequency and spatial domains is achieved, significantly improving the accuracy and robustness of denoising.

[0071] In the demosaicing section of this optional embodiment, the demosaicing module is a key step in the image signal processing pipeline, converting the single-channel Bayer array image output by the CMOS sensor into a three-channel RGB image format. Because each pixel in a Bayer array image contains only one color (red, green, or blue), traditional demosaicing methods typically estimate the missing color information through interpolation algorithms. However, this interpolation method is prone to information loss, especially in areas with complex edges and textures, resulting in artifacts and color distortion.

[0072] To address this issue, this optional implementation proposes a demosaicing method based on a convolutional neural network. This method uses convolution operations to deepen channels and extract features, thereby preserving detailed information in the original image. Specifically, the demosaicing network gradually recovers missing color information from the Bayer array image through multiple layers of convolution and nonlinear activation functions. Compared with traditional interpolation methods, convolutional neural networks can better capture the local structure and global context of the image, significantly improving the accuracy and visual quality of color reconstruction.

[0073] In an optional embodiment of this application, a color correction matrix is ​​used to adjust the image's color representation based on ambient lighting conditions and convert it to the standard sRGB color space. Gamma correction uses a nonlinear mapping to equalize image brightness and darkness, enhancing the image's visual quality. These correction modules are optimized through end-to-end training, automatically adapting to different imaging scenarios and improving overall image quality.

[0074] Through the collaborative work of these multiple modules, this optional implementation not only effectively suppresses noise and restores color information, but also preserves image detail, significantly improving the overall quality of images in low-light environments. Combining a U-Net network with a wavelet transform denoising algorithm, this algorithm significantly improves noise suppression through dual optimization in both the frequency and spatial domains. The multi-scale feature extraction capabilities of the U-Net network, combined with the high-frequency component separation capabilities of the wavelet transform, effectively preserve image detail while removing complex noise in low-light environments.

[0075] This paper proposes an end-to-end image processing algorithm that, unlike traditional single denoising networks, integrates and optimizes multiple modules, including black level correction (BLC), white balance (WB), denoising network (DN), demosaicing (DM), color correction matrix (CCM), and gamma correction (GC). Through end-to-end training and learnable parameters, it dynamically adapts to varying hardware characteristics and lighting conditions, improving the overall quality and efficiency of image processing in low-light environments, effectively capturing fine-grained image features, and enhancing color quality.

[0076] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, Figure 4 A schematic diagram of an image processing device according to an embodiment of the present application is shown in FIG. Figure 4 As shown, the embodiment of the present application further provides an image processing device, including: Feature extraction module 401, used to extract global features and local features of the original image; A black level processing module 402 is configured to determine a black level value for each unit pixel in the original image based on the global features and the local features, where the black level value is generated by device noise of a sensor that acquires the original image; Correction module 403, configured to perform image correction based on the black level value of each unit pixel to obtain intermediate image features; A denoising processing module 404 is used to perform denoising based on the intermediate image features and repair the denoised intermediate image features to obtain reconstructed image features; The image enhancement module 405 is used to perform image enhancement processing on the reconstructed image features to obtain a target image.

[0077] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0078] Figure 5 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 5 As shown, the electronic device includes a memory 501 and a processor 502. The memory 501 stores a computer program that can be executed on the processor 502. When the processor 502 executes the computer program, the method of the above embodiment is implemented. The number of memory 501 and processor 502 can be one or more. In a specific implementation, the electronic device may also include a communication interface 503 for communicating with external devices and performing data exchange.

[0079] In a specific implementation, if the memory 501, processor 502, and communication interface 503 are implemented independently, the memory 501, processor 502, and communication interface 503 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0080] Optionally, in a specific implementation, if the memory 501 , the processor 502 , and the communication interface 503 are integrated on a chip, the memory 501 , the processor 502 , and the communication interface 503 may communicate with each other through an internal interface.

[0081] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0082] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0083] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0084] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0085] It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0086] Furthermore, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache memory. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0087] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0088] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0090] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0091] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus or device), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0092] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0093] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0094] The above are merely exemplary embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Extract global features and local features of the original image; Determine a black level value of each unit pixel in the original image based on the global feature and the local feature, where the black level value is generated by device noise of a sensor that acquires the original image; Performing image correction according to the black level value of each unit pixel to obtain an intermediate image feature; Performing denoising based on the intermediate image features, and repairing the denoised intermediate image features to obtain reconstructed image features; Perform image enhancement processing on the reconstructed image features to obtain a target image.

2. The method according to claim 1, characterized in that Determining a black level value of each unit pixel in the original image based on the global feature and the local feature includes: The global feature and the local feature are fused through a fully connected layer to obtain a fused feature; The fused features are input into a black level prediction layer, and the black level value of each unit pixel is determined based on the features of the sensor.

3. The method according to claim 2, characterized in that The performing image correction according to the black level value of each unit pixel to obtain an intermediate image feature includes: Subtracting the corresponding black level value from the original pixel value of each unit pixel to obtain a first correction value of each unit pixel; The intermediate image feature is generated based on the first correction value of each unit pixel.

4. The method according to claim 3, characterized in that The method further comprises: Inputting the fused features into a white balance prediction layer to determine the white balance value of each unit pixel; Performing color temperature correction based on the white balance value of each unit pixel to obtain a second correction value for each unit pixel; The intermediate image feature is obtained based on the first correction value and the second correction value of each unit pixel.

5. The method according to claim 1, wherein The performing denoising based on the intermediate image features and repairing the denoised intermediate image features to obtain reconstructed image features includes: performing wavelet decomposition based on the intermediate image features to obtain multiple components of different frequencies, the multiple components including a first component representing noise and edge information in the original image, and a second component representing image information of the original image; Adding noise to the first component to obtain features after forward diffusion; Perform prediction based on the features after forward diffusion to obtain prediction noise; Performing denoising processing based on the predicted noise to obtain a denoised first component; Wavelet reconstruction is performed based on the denoised first component and the second component to obtain the reconstructed image features.

6. The method according to claim 5, characterized in that The step of performing prediction based on the features after forward diffusion to obtain predicted noise includes: Downsampling the forward-diffused features to obtain a plurality of downsampled features, wherein the plurality of downsampled features correspond to different image scales; Performing upsampling processing on the multiple downsampled features one by one to obtain multiple upsampled features; The multiple up-sampled features are fused with the corresponding down-sampled features to obtain the predicted noise.

7. The method according to any one of claims 1 to 6, characterized in that The original image is in a Bayer channel format, the image enhancement processing includes demosaicing processing and image filter processing, and the image enhancement processing is performed on the reconstructed image features to obtain a target image, including: Performing demosaicing on each unit pixel using a dynamic attention method to obtain a demosaic feature, wherein the demosaic feature is converted from the Bayer channel format to the RGB channel format; Perform image filter processing based on the demosaicing feature to obtain a target image.

8. The method according to claim 7, characterized in that The performing image filter processing based on the demosaicing feature to obtain a target image includes: Based on predetermined image style information, each channel of the RGB channel format of the demosaiced feature is retouched to obtain a color correction feature; The color correction feature is nonlinearly transformed by a predetermined gamma value to obtain the target image, and the predetermined gamma value is used to adjust the image brightness distribution.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program. 10 . A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to claim 1 is implemented.

Images