An imaging method and related equipment based on an RGBW sensor

Abstract

This application discloses an imaging method and related equipment based on an RGBW sensor, applied to an electronic device including the RGBW sensor. The method includes: acquiring an RGBW raw image captured by the RGBW sensor; processing the W channel of the RGBW raw image to obtain a W raw image, and processing the RGB channels of the RGBW raw image to obtain an RGGB raw image; adjusting the signal-to-noise ratio of the RGGB raw image based on the W raw image to generate a first image. Using this embodiment, the signal-to-noise ratio of the image can be improved, thereby improving the image quality captured in low-light environments.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an imaging method and related equipment based on an RGBW sensor. Background Technology

[0002] Traditional color image sensors typically use the RGGB (red, green, green, blue) Bayer pixel arrangement, widely used in cameras of electronic devices such as mobile phone cameras. Under normal lighting conditions, RGGB sensors provide good image quality. However, in low-light environments, due to the weaker light sensitivity of RGGB sensors, image noise increases and the signal-to-noise ratio decreases, leading to reduced image sharpness and affecting basic image quality. A common solution is to increase exposure time to improve sensitivity, but this can result in blurry images when shooting moving scenes. At the hardware level, some mobile phone manufacturers increase the size of the image sensor to improve light sensitivity, but due to the size limitations of mobile devices, it is difficult to accommodate a larger sensor, thus this approach is not very effective.

[0003] In addition, some mobile phone manufacturers have modified the pixel arrangement of image sensors, adopting a new Bayer format image sensor with RGBW (red, green, blue, white). This approach significantly increases the white (W) filter, enabling RGBW sensors to have both color sensitivity and enhanced light sensitivity. However, the increased white channel affects the color balance of the image generated after fusing RGB and W pixel data. Furthermore, the higher the proportion of W pixels, the greater the impact on color fidelity and balance. Therefore, finding a fusion strategy that can improve the light sensitivity of the image sensor while maintaining color balance is a pressing issue. Summary of the Invention

[0004] This application provides an imaging method and related equipment based on an RGBW sensor, which can improve the signal-to-noise ratio of the image to improve the image quality captured in low-light environments.

[0005] In a first aspect, embodiments of this application provide an imaging method based on an RGBW sensor, applied to an electronic device, the electronic device including the RGBW sensor, the method including: acquiring an RGBW raw image collected by the RGBW sensor; processing the W channel of the RGBW raw image to obtain a W raw image (W data stream), and processing the RGB channels of the RGBW raw image to obtain an RGGB raw image (RGGB data stream); adjusting the signal-to-noise ratio of the RGGB raw image based on the W raw image to generate a first image.

[0006] Most existing imaging methods are designed based on RGB (Red-Green-Green-Blue) sensors. When shooting in low-light environments, the weak light sensitivity of RGB sensors leads to increased image noise and a decreased signal-to-noise ratio, resulting in reduced image sharpness and affecting basic image quality. To address this problem, this application proposes an imaging method based on the more sensitive RGBW (Red-Green-Blue-White) sensor. Furthermore, to avoid affecting the color balance of the image generated after fusing RGB and W pixel data, this application provides a fusion strategy based on an RGBW sensor. Specifically, the electronic device acquires the RGBW raw image (RGBW RAW) captured by the RGBW sensor. Since the RGBW sensor adds white (W) pixels compared to the traditional RGGB sensor, it can capture more photons, improving its light sensitivity in low-light environments. Further, the W channel of the RGBW raw image is processed to obtain the W raw image (W RAW), and the RGB channels of the RGBW raw image are processed to obtain the RGB raw image (RGBRAW), facilitating subsequent targeted adjustments to the image's brightness and color information. Furthermore, by adjusting the signal-to-noise ratio of the original RGB image based on the original W image, and utilizing the brightness information of the original W image to adjust the RGB channel data, the subtle textures and details in the original RGB image can be preserved while retaining the color information of the RGB channels as much as possible to avoid color loss or distortion. This generates a first image with a high signal-to-noise ratio, which improves the clarity and realism of the image obtained after visualization processing based on the first image under low-light conditions. This enhances the image quality captured in low-light environments and improves the user's shooting experience.

[0007] In one possible implementation, the pixels of the RGBW sensor include multiple pixel units, each pixel unit consisting of one of the three types of pixels (R, G, B) and a neighboring W pixel. The pixel arrangement of the original RGBW image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor.

[0008] In this embodiment, since the pixels of the RGBW sensor include multiple pixel units composed of one of the three types of pixels (R, G, B) and a neighboring W pixel, and the pixel arrangement of the RGBW original image is consistent with that of the sensor, the pixels in the RGBW original image also have the same pixel units as the RGBW sensor. Furthermore, the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor, and the channel value of each pixel in the RGBW original image corresponds to the pixel of the RGBW sensor. This ensures that the acquired RGBW original image can retain the information of each pixel, which helps to maintain the detail and clarity of the image in subsequent processing stages, thereby improving the realism and quality of the image.

[0009] In one possible implementation, the total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the RGBW sensor.

[0010] In this embodiment, the ratio of RGB pixels to W pixels in each pixel unit of the RGBW sensor can be an integer ratio, such as 1:1, 1:2, 1:3, 1:8, 1:15, etc., or a fractional ratio, such as 4:5, 7:9, etc. Accordingly, based on the ratio of RGB pixels to W pixels in each pixel unit of the RGBW sensor, an RGBW sensor with a suitable resolution (i.e., total pixel value) can be selected to acquire images. Specifically, the smaller the ratio of RGB pixels to W pixels in each pixel unit, the larger the resolution (i.e., total pixel value) of the corresponding RGBW sensor. For example, when the ratio of RGB pixels to W pixels in a pixel unit is 1:8, the resolution (i.e., the total pixel value) of the corresponding RGBW sensor is 108M pixels; when the ratio of RGB pixels to W pixels in a pixel unit is 1:15, the resolution (i.e., the total pixel value) of the corresponding RGBW sensor needs to be approximately 200 million pixels to provide enough pixels to capture spatial details, thereby meeting the needs of different application scenarios for color and brightness information, and helping to maintain the detail and clarity of the image to improve the overall image quality. Since the W pixels in an RGBW sensor are used to capture luminance information and the RGB pixels are used to capture color information, the smaller the ratio of RGB pixels to W pixels in a pixel unit of the RGBW sensor, the fewer RGB pixels there are compared to W pixels in a pixel unit. Therefore, to ensure that the RGBW sensor can acquire sufficient color information, the total pixel value needs to be increased to increase the number of pixel units, so that the RGBW sensor has a sufficient number of RGB pixels to capture enough color information, thereby improving the color accuracy and reproduction of the final generated image. At the same time, increasing the total pixel value of the RGBW sensor can also increase the resolution of the original RGBW image, resulting in improved detail and clarity of the final generated image, thus better improving the image quality when shooting in low light conditions and enhancing the user's shooting experience.

[0011] In one possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; the step of acquiring the RGBW raw image captured by the RGBW sensor may include: acquiring the long-exposure RGBW raw image captured by the RGBW sensor with a long exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor with a short exposure.

[0012] In this embodiment, different types of RGBW raw images can be acquired depending on the exposure time of the RGBW sensor. Specifically, RGBW raw images acquired by the RGBW sensor with long exposure are called long-exposure RGBW raw images, while RGBW raw images acquired with normal exposure time are called normal-exposure RGBW raw images. The normal exposure time can be adjusted according to the actual shooting scene, and can be either a shorter or longer exposure time to achieve the desired exposure effect. Under low-light conditions, because the light-sensing ability of RGB pixels is weak and the number of photons received is small, the noise level increases. Therefore, by appropriately extending the exposure time of the RGBW sensor (i.e., long exposure time), the RGB pixels can accumulate more photon signals, reducing the noise generated by the RGB pixels to a certain extent and improving the signal-to-noise ratio, thereby improving the image quality in low-light environments. The W pixel has a stronger light sensitivity, so under the same lighting conditions, the W pixel can receive more photons than the RGB pixel. Using a normal exposure time can ensure that the brightness information in the image is captured properly, avoiding overexposure caused by too much light received by the W pixel. It can also avoid motion blur or noise caused by long exposure, thus capturing the brightness information of the instantaneous scene more accurately and improving image quality.

[0013] In one possible implementation, obtaining the W original image based on the W channel processing of the RGBW original image may include: calculating the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image; obtaining the W original image based on the average channel value; and the channel value of each W pixel in the W original image being sequentially equal to the average channel value.

[0014] In this embodiment of the application, regarding how to obtain a W original image based on the W channel processing of an RGBW original image, the specific steps include: firstly, calculating the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image; then, using the calculated average channel value as the channel value of each W pixel in the W original image, thereby effectively extracting the brightness information in the RGBW original image to generate the W original image. This ensures that the brightness distribution in the W original image is consistent with that of the RGBW original image, so that the signal-to-noise ratio of the RGBW original image can be adjusted using the W original image, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0015] In one possible implementation, the step of processing the RGB channels of the original RGBW image to obtain the original RGGB image may include: selecting a target pixel from each pixel unit in the long-exposure RGBW original image, wherein the target pixel is a pixel of the R, G, and B channels in the pixel unit; generating the original RGGB image based on the channel values ​​of the target pixel; wherein the channel values ​​of each pixel in the original RGGB image correspond to and are equal to the channel values ​​of the target pixel in sequence.

[0016] In this embodiment, regarding how to obtain an RGGB original image based on the RGB channels of an RGBW original image, the specific steps include: first, selecting pixels (i.e., target pixels) from each pixel unit in the long-exposure RGBW original image, representing the R, G, and B channels. Further, the channel values ​​of the selected target pixels are sequentially used as the channel values ​​of each pixel in the RGGB original image (RGGBRAW), ensuring that the color information of the generated RGGB RAW is consistent with that of the RGBW original image. This improves the color accuracy and fidelity of subsequent adjustments to the RGGB original image, enhancing the overall quality of the captured image and improving the user's shooting experience.

[0017] In one possible implementation, if the RGBW sensor is currently in image capture mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image may include: extracting features from the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image; merging the first feature image and the second feature image to generate a third feature image; and reconstructing a denoised and de-mosaiced RGB image based on the third feature image, wherein the denoised and de-mosaiced RGB image is the first image.

[0018] In this embodiment, when the RGBW sensor is in shooting mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image can specifically include: firstly, performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image, thereby effectively extracting the brightness features in the W original image and the color features in the RGGB original image, so that the image can be adjusted in a targeted manner according to different needs. Further, by merging the first feature image (brightness features of the W original image) and the second feature image (color features of the RGGB original image), a third feature image containing the brightness features of the W original image and the color features of the RGGB original image is generated. This ensures that the key information of the RGBW original image in terms of color and brightness is preserved, while noise generated in the RGGB original image due to insufficient light is suppressed to a certain extent, thereby improving the signal-to-noise ratio of the third feature image. Then, based on the third feature image, a denoised and de-mosaiced RGB image (i.e., the first image) is reconstructed to restore the clarity and detail of the third feature image. While preserving the subtle textures and details in the original RGGB image, the color information of the RGB channels is minimized to improve color accuracy. This results in improved clarity and realism of the image obtained after visualization processing based on the first image under low light conditions, thereby improving the image quality in low light environments and enhancing the user's shooting experience.

[0019] In one possible implementation, if the RGBW sensor is currently in photo capture mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: inputting the W original image and the RGGB original image into a dual-stream fusion algorithm model; adjusting the signal-to-noise ratio of the RGGB original image based on the W original image through the dual-stream fusion algorithm model, and outputting the first image.

[0020] In this embodiment, when the RGBW sensor is in shooting mode, the original W image and the original RGGB image can be input into a dual-stream fusion algorithm model. This model adjusts the signal-to-noise ratio (SNR) of the RGGB image based on the original W image, thereby outputting a first image with a high SNR that retains image details and color information. Furthermore, the dual-stream fusion algorithm model is a pre-trained deep learning network structure model (e.g., a convolutional neural network model). Using this model ensures the stability of the quality of the output image (i.e., the first image) under different conditions, improving the clarity and realism of the image obtained after visualization processing based on the first image under low-light conditions. This enhances the image quality captured in low-light environments and improves the user's shooting experience.

[0021] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image.

[0022] In this embodiment, the dual-stream fusion algorithm model may include a color restoration sub-model and an image fusion sub-model. Further, the original RGGB sample image is input into the color restoration sub-model, and trained using a first loss function to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The original W sample image and the original RGGB sample image are input into the image fusion sub-model, and trained using a second and a third loss function, enabling the image fusion sub-model to fuse the input dual-stream images (i.e., the original W sample image and the original RGGB sample image) to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image. Through this embodiment, the trained color restoration sub-model can be used for preview imaging, allowing users to obtain a rough image effect before taking a photo, thereby improving the user experience. The trained image fusion sub-model can extract detail and color information from the original W sample image and the original RGGB sample image. By fusing the original W sample image and the original RGGB sample image, the noise present in the single original RGGB sample image is reduced, which improves the signal-to-noise ratio of the final high-resolution RGB image. While making the details clearer and more realistic, it can also maintain color balance, thereby improving the image quality of the image taken in low light environment and enhancing the user's shooting experience.

[0023] In one possible implementation, the original RGGB sample image and the original W sample image are obtained by degradation based on a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; and the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0024] In this embodiment, since training the dual-stream fusion algorithm model requires paired low-resolution (LR) and high-resolution (HR) images as training data pairs, and obtaining the low-resolution LR images (i.e., the original RGGB sample images and the original W sample images) is difficult, the original RGGB images (i.e., the original RGGB sample images and the original W sample images) can be generated by degradation from easily obtainable high-resolution RGB images (i.e., high-resolution RGB sample images) to train the dual-stream fusion algorithm model. This reduces the cost of model training, increases the diversity of the training dataset, improves the model's generalization ability and robustness, and enhances the stability of the image quality output by the dual-stream fusion algorithm model under different conditions. Furthermore, for the color restoration sub-model, the color information difference between the low-resolution RGB image and the high-resolution RGB sample image is evaluated using a first loss function, enabling the color restoration sub-model to accurately restore the color information of the original RGGB sample image, thereby improving the color reproduction accuracy of the low-resolution RGB image. For the image fusion sub-model, a second loss function is used to evaluate... The differences in detail and color information between the high-resolution RGB image and the high-resolution RGB sample image ensure that the image fusion sub-model does not lose important details and color information during image fusion, thereby improving the clarity and color reproduction of the high-resolution RGB image output by the image fusion sub-model. Furthermore, the color component differences between the low-resolution RGB image output by the color restoration sub-model and the high-resolution RGB image output by the image fusion sub-model are evaluated using a third loss function. This ensures that the final image generated by the image fusion sub-model (i.e., the high-resolution RGB image) has consistent colors with the low-resolution RGB image generated by the color restoration sub-model for preview, further improving the color reproduction of the image fusion sub-model. In summary, the image fusion sub-model trained through the embodiments of this application, used as a two-stream fusion algorithm model for image fusion, not only improves the clarity and color reproduction of the output first image (i.e., the high-resolution RGB image), but also enhances the stability of the image quality output by the two-stream fusion algorithm model under different conditions, thereby improving the user's shooting experience.

[0025] In one possible implementation, the electronic device further includes a display screen; the method further includes: color correcting the first image to generate a second image in RGB format; and displaying the second image on the display screen.

[0026] In this embodiment, when the RGBW sensor is in shooting mode, after generating a first image (an RGB image after denoising and de-mosaicing) based on the signal-to-noise ratio of the original RGGB image using the original W image, color correction can be performed on this first image. For example, automatic white balance processing can eliminate color temperature deviations in the first image caused by different light sources, ensuring that white objects appear true white in the image. Then, a color transformation matrix is ​​used to convert the color space of the first image to a range conforming to a specific standard, avoiding color differences on different devices due to different color space representations used by different devices and sensors. This results in a more accurate and consistent color in the converted image. Through this embodiment, it can be ensured that the second RGB format image after color correction can present consistent color performance and color accuracy on the displays of different devices, thereby improving the user's shooting experience.

[0027] In one possible implementation, if the RGBW sensor is currently in preview or video recording mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: determining the type and distribution pattern of noise in the RGGB original image by comparing the channel values ​​in the W original image and the corresponding channel values ​​in the RGGB original image; performing noise reduction processing on the RGGB original image according to the type and distribution pattern of noise to generate a high signal-to-noise ratio RGGB original image, wherein the high signal-to-noise ratio RGGB original image is the first image.

[0028] In this embodiment, when the RGBW sensor is currently in preview or video recording mode, adjusting the signal-to-noise ratio (SNR) of the RGGB original image based on the W original image to obtain a first image can specifically include: firstly, by comparing the channel values ​​of the W channel image and the RGGB channel image, determining the noise type and distribution pattern in the RGGB original image, so as to select an appropriate noise reduction method to reduce the noise in the RGGB original image. Further, based on the determined noise type and distribution pattern, targeted noise reduction processing is performed on the RGGB original image, for example, using a specific noise reduction filter or a lightweight neural network model algorithm, to accurately remove noise from the RGGB original image while preserving as much detail and color information as possible, resulting in a high SNR RGGB original image (i.e., the first image). This improves the clarity and realism of the image obtained after visualization processing based on the high SNR RGGB original image (i.e., the first image) under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience. In this embodiment, when the RGBW sensor is in preview or video recording mode, an algorithm with lower computational complexity and memory consumption compared to the algorithm design in photo mode is adopted. The internal computing unit of the sensor first performs preliminary noise reduction processing on the W original image and RGGB original image and outputs continuous high signal-to-noise ratio image frames to reduce the computational burden on the subsequent processor (such as CPU). This improves the image quality when the electronic device is previewing or recording video, and also improves the real-time performance and smoothness of previewing or video recording, thereby enhancing the user's shooting experience.

[0029] In one possible implementation, the method further includes: performing denoising and de-mosaic processing on the first image to obtain a third image in RGB format.

[0030] In this embodiment, when the RGBW sensor is in preview or video recording mode, the details and color information of the first image (i.e., the original RGGB image with high signal-to-noise ratio) are restored by denoising and de-mosaic processing on the first image. This improves the clarity and realism of the generated RGB format third image under low light conditions, thereby improving the image quality generated after subsequent visualization processing based on the third image and enhancing the user's shooting experience.

[0031] In one possible implementation, the step of denoising and de-mosaicing the first image to obtain a third image in RGB format includes: inputting the first image into an original image processing network model; and generating the third image by performing denoising and de-mosaicing processing on the original image processing network model.

[0032] In this embodiment, when the RGBW sensor is in preview or video recording mode, a first image (i.e., a high signal-to-noise ratio RGGB original image) can be input into the original image processing network model. The original image processing network model then performs denoising and de-mosaic processing on the first image and outputs a third image in RGB format, thereby further improving the quality and clarity of the third image. At the same time, the original image processing network model can adaptively adjust according to different scenes and conditions to adapt to different lighting conditions, environmental changes, and other factors, ensuring that the quality of the processed image remains at a high level. This improves the quality of the image generated after subsequent visualization processing based on the third image, thereby enhancing the user's shooting experience.

[0033] In one possible implementation, the electronic device further includes the display screen, and the method further includes: performing color correction on the third image to generate a fourth image in RGB format; and displaying the fourth image on the display screen.

[0034] In this embodiment, when the RGBW sensor is in preview or video recording mode, after denoising and de-mosaicing the first image (i.e., the high signal-to-noise ratio RGGB original image) to obtain a third image in RGB format, color correction can be performed on the third image, and then the color-corrected fourth image in RGB format can be displayed on the screen. For example, automatic white balance processing can eliminate color temperature deviations in the third image caused by different light sources, allowing white objects to appear true white in the image. Furthermore, a color transformation matrix can be used to convert the color space of the third image to a range conforming to a specific standard, avoiding color differences on different devices due to different color spaces used by different devices and sensors. This results in a more accurate and consistent color representation in the converted image. Through this embodiment, it can be ensured that the color-corrected fourth image in RGB format can display consistent color performance and color accuracy on the screens of different devices, thereby improving the user's shooting experience.

[0035] Secondly, this application provides an imaging device based on an RGBW sensor, applied to an electronic device, the electronic device including the RGBW sensor; it may include:

[0036] The acquisition unit is used to acquire the original RGBW image collected by the RGBW sensor;

[0037] The first processing unit is used to process the W channel of the RGBW original image to obtain the W original image, and to process the RGB channel of the RGBW original image to obtain the RGGB original image.

[0038] The adjustment unit adjusts the signal-to-noise ratio of the original RGGB image based on the original W image to generate a first image.

[0039] Most existing imaging methods are designed based on RGB (Red-Green-Green-Blue) sensors. When shooting in low-light environments, the weak light sensitivity of RGB sensors leads to increased image noise and a decreased signal-to-noise ratio, resulting in reduced image sharpness and affecting basic image quality. To address this problem, this application proposes an imaging device based on an RGBW (Red-Green-Blue-White) sensor with better light sensitivity. Furthermore, to avoid affecting the color balance of the image generated after fusing RGB and W pixel data, this application provides a fusion strategy based on an RGBW sensor. Specifically, this application acquires the original RGBW image (RGBWRAW) collected by the RGBW sensor through an acquisition unit. Since the RGBW sensor adds white (W) pixels compared to the traditional RGGB sensor, it can capture more photons, thus improving its light sensitivity in low-light environments. Furthermore, the first processing unit processes the W channel of the RGBW original image to obtain the W original image (W RAW), and processes the RGB channels of the RGBW original image to obtain the RGB original image (RGB RAW), facilitating subsequent targeted adjustments to the brightness and color information of the image. Further, the adjustment unit adjusts the signal-to-noise ratio of the RGB original image based on the W original image. Utilizing the brightness information of the W original image, the RGB channel data is adjusted, preserving subtle textures and details in the RGB original image while retaining as much color information as possible from the RGB channels to avoid color loss or distortion. This generates a first image with a high signal-to-noise ratio, improving the clarity and realism of the image obtained after visualization processing based on this first image under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0040] In one possible implementation, the pixels of the RGBW sensor include multiple pixel units, each pixel unit consisting of one of the three types of pixels (R, G, B) and a neighboring W pixel. The pixel arrangement of the original RGBW image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor.

[0041] In one possible implementation, the total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the RGBW sensor.

[0042] In one possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; the acquisition unit is specifically used for:

[0043] Acquire the long-exposure RGBW raw image captured by the RGBW sensor during long-exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor during regular-exposure.

[0044] In one possible implementation, the first processing unit is specifically used for:

[0045] Calculate the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image;

[0046] The W original image is obtained based on the average channel value; the channel value of each W pixel in the W original image corresponds to the average channel value in sequence.

[0047] In one possible implementation, the first processing unit is specifically used for:

[0048] Select a target pixel from each pixel unit in the long exposure RGBW original image, wherein the target pixel is a pixel of the R, G, and B channels in the pixel unit;

[0049] The RGGB original image is generated based on the channel values ​​of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel.

[0050] In one possible implementation, if the RGBW sensor is currently in photo-taking mode, the adjustment unit is specifically used for:

[0051] Feature extraction is performed on the original W image and the original RGGB image respectively to obtain a first feature image corresponding to the original W image and a second feature image corresponding to the original RGGB image;

[0052] The first feature image and the second feature image are merged to generate a third feature image;

[0053] The RGB image after denoising and de-mosaic processing is reconstructed based on the third feature image, and the RGB image after denoising and de-mosaic processing is the first image.

[0054] In one possible implementation, if the RGBW sensor is currently in photo-taking mode, the adjustment unit is specifically used for:

[0055] The W original image and the RGGB original image are input into the dual-stream fusion algorithm model;

[0056] Using the dual-stream fusion algorithm model, the signal-to-noise ratio of the RGGB original image is adjusted based on the W original image, and the first image is output.

[0057] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image.

[0058] In one possible implementation, the original RGGB sample image and the original W sample image are obtained by degradation based on a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; and the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0059] In one possible implementation, the electronic device further includes a display screen; the imaging device further includes:

[0060] The first color correction unit is used to perform color correction on the first image and generate a second image in RGB format;

[0061] A first display unit is used to display the second image on the display screen.

[0062] In one possible implementation, if the RGBW sensor is currently in preview or video recording mode, the adjustment unit is specifically used for:

[0063] By comparing the channel values ​​in the original W image with the corresponding channel values ​​in the original RGGB image, the type and distribution pattern of noise in the original RGGB image are determined.

[0064] Based on the type and distribution pattern of the noise, the original RGGB image is denoised to generate a high signal-to-noise ratio (SNR) RGGB image, which is the first image.

[0065] In one possible implementation, the imaging device further includes:

[0066] The second processing unit is used to perform denoising and de-mosaic processing on the first image to obtain a third image in RGB format.

[0067] In one possible implementation, the second processing unit is specifically used for:

[0068] The first image is input into the original image processing network model;

[0069] The third image is generated after denoising and depixelation processing using the original image processing network model.

[0070] In one possible implementation, the electronic device further includes the display screen, and the imaging device further includes:

[0071] The second color correction unit is used to perform color correction on the third image and generate a fourth image in RGB format;

[0072] The second display unit is used to display the fourth image on the display screen.

[0073] Thirdly, embodiments of this application provide an electronic device that may include an RGBW sensor, a memory, and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, causing the electronic device to perform the method described in any one of the second aspects above.

[0074] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by the processor to implement the method described in either the first or second aspect above.

[0075] Fifthly, embodiments of this application provide a computer program, the computer program including instructions, the computer program being executed by a computing device to implement the method described in any one of the first or second aspects above. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0077] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0078] Figure 2 This is a schematic diagram of the software structure of the electronic device 100 provided in the embodiments of this application.

[0079] Figure 3 This is a flowchart illustrating an imaging method based on an RGBW sensor provided in an embodiment of this application.

[0080] Figure 4 This is a schematic diagram of the pixel arrangement of some RGBW sensors provided in the embodiments of this application.

[0081] Figure 5A This is a flowchart illustrating an imaging method based on an RGBW sensor in a photo-taking mode, as provided in an embodiment of this application.

[0082] Figure 5B This is a flowchart illustrating another imaging method based on an RGBW sensor in a shooting mode provided in this application embodiment.

[0083] Figure 5C This is a schematic diagram of the network structure of a dual-stream fusion algorithm model provided in an embodiment of this application.

[0084] Figure 6 This is a flowchart illustrating an imaging method based on an RGBW sensor in preview or video recording mode, as provided in an embodiment of this application.

[0085] Figure 7 This is a schematic diagram of the imaging process of an imaging method based on an RGB8W sensor provided in an embodiment of this application.

[0086] Figure 8 This is a schematic diagram of a preview or video recording process for an imaging method based on an RGB8W sensor provided in an embodiment of this application.

[0087] Figure 9 This is a schematic diagram of a user interface for taking photos in low-light environments using an electronic device, as provided in an embodiment of this application.

[0088] Figure 10 This is a schematic diagram of a user interface for a user to shoot videos in a low-light environment using an electronic device, as provided in an embodiment of this application.

[0089] Figure 11 This is a schematic diagram of the structure of an imaging device based on an RGBW sensor provided in an embodiment of this application.

[0090] Figure 12 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0091] The embodiments of this application are described below with reference to the accompanying drawings.

[0092] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of this application. As used in the specification and appended claims of the embodiments of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in the embodiments of this application refers to and includes any or all possible combinations of one or more of the listed items.

[0093] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0094] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0095] First, some terms used in this application will be explained to help those skilled in the art understand the embodiments of this application.

[0096] (1) Signal-to-noise ratio (SNR) refers to the ratio between the strength of the useful signal and the strength of the background noise. This concept is widely used in image processing and other fields, and is one of the important indicators for measuring image quality. Noise is an unwanted random fluctuation in an image introduced by various factors, such as sensor noise, electronic noise, and ambient light noise. A high SNR means that the information in the image is clearer and easier to identify. The SNR is usually calculated using a logarithmic scale and expressed in decibels (dB). In practical applications, improving the SNR is usually a goal in image processing algorithms and system design.

[0097] (2) RAW image: refers to the original image acquired by the image sensor without any processing. RAW image retains the original data of each pixel obtained from the sensor, including brightness, color and other relevant information.

[0098] (3) Convolutional Neural Network (CNN): This is a type of deep learning neural network that can effectively capture and learn hierarchical feature representations, which makes it perform well in fields such as image processing and computer vision. Due to the translation invariance and parameter sharing characteristics of CNNs for image data, they can effectively process data with spatial hierarchical structure. They are mainly used for processing and analyzing data with grid structure, such as image and video recognition, computer vision tasks, etc.

[0099] (4) A feature map is the output of a layer in a convolutional neural network (CNN). When an image is input into a CNN, each convolutional layer processes the image (or the feature map of the previous layer) using its filters (or kernels) to generate new feature maps. These feature maps are essentially representations of the image after being processed by specific filters.

[0100] (5) Deep Learning Network refers to a highly complex and hierarchical machine learning model built based on artificial neural networks. It is a key technology in fields such as artificial intelligence, computer vision, and natural language processing, and is applied to various scenarios such as image recognition, speech processing, and predictive analytics. The construction process of a deep learning network typically includes steps such as multi-layer neuron configuration, weight training, feature learning, and model optimization, aiming to solve complex pattern recognition and data analysis problems by mimicking the way the human brain processes information.

[0101] (6) Loss Function (LF): Defines the difference between the model’s output for a given input and the actual label.

[0102] (7) Mean Squared Error Loss (MSE Loss) function: It is a loss function used to measure the difference between the model's predicted value and the actual value. It is calculated by averaging the squares of the difference between the predicted value and the actual value.

[0103] (8) Perception Loss (PL) Function: This is a loss function based on deep learning, commonly used in tasks such as image generation and image super-resolution. The core idea of ​​the perception loss function is to use a pre-trained deep convolutional neural network (usually a network pre-trained on large-scale image classification tasks, such as VGG, ResNet, etc.) to extract image features and use these features to measure the difference between the generated image and the real image. Compared to pixel-level comparison, feature-based comparison can better capture the high-level semantic information of the image, making the loss function more discriminative and generalizable.

[0104] To facilitate understanding of the embodiments of this application, the exemplary electronic devices provided in the embodiments of this application will be described below.

[0105] Please see Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 100 is a smart terminal device and can be of various types; this embodiment does not limit its specific type. For example, the terminal device can be a mobile phone, and may also include tablet computers, desktop computers, desktop computers with touch-sensitive surfaces or touch panels, laptop computers, handheld computers, smart screens, wearable devices (such as smartwatches, smart bracelets, etc.), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, in-vehicle systems, smart headphones, game consoles, and may also be Internet of Things (IoT) devices or smart home devices such as smart water heaters, smart lights, smart air conditioners, etc. Please see below. Figure 1 The following is combined with Figure 1 A detailed description of each component of the electronic device 100 is provided below:

[0106] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0107] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0108] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0109] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0110] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0111] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0112] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0113] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0114] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0115] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0116] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0117] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0118] The modem processor may include a modulator and a demodulator. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0119] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0120] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0121] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0122] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0123] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0124] The Image Sensor (ISP) processes the raw image data captured by the image sensor in the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (i.e., the image sensor). The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing, transforming it into an image that can be displayed on the screen. Exemplarily, Auto White Balance (AWB) and Color Correction Matrix (CCM) are two important functional modules in the ISP. The AWB module automatically adjusts the white balance in the image by analyzing different areas of the image, detecting the main light source in the scene, and adjusting the color temperature of the image to eliminate color deviations and make whites appear more accurate. The CCM module is used for color correction, adjusting the gain of different color channels in the image through matrix transformation to ensure that the image's color representation matches the real scene, avoiding color distortion caused by the sensor, light source, or other factors, and making the image colors more accurate. Furthermore, the ISP can also perform algorithmic optimization of image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set within the camera 193.

[0125] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element (i.e., an image sensor) through a lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In this embodiment, the photosensitive element (i.e., the image sensor) is an RGBW sensor formed by arranging and combining four types of pixels: red (R), green (G), blue (B), and white (W). RGB pixels are light-receiving elements corresponding to the wavelengths of red, green, and blue colors, while W pixels are light-receiving elements that receive all RGB wavelengths. The RGBW sensor improves the light sensitivity of the image sensor by adding a white channel, allowing it to capture more light in low-light environments, improving image brightness and clarity, reducing noise to improve the signal-to-noise ratio, and thus improving image quality in low-light conditions. Optionally, the ratio of the number of RGB pixels to the number of W pixels in the RGBW sensor can be 1:1, 1:2, 1:3, 1:8, 1:15, etc., or other non-integer ratios, such as 4:5, 7:9, etc. The embodiments of this application do not limit this.

[0126] The optical signal is converted into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0127] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0128] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0129] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0130] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0131] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0132] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0133] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0134] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0135] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0136] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0137] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0138] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0139] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100.

[0140] The 180C barometric pressure sensor is used to measure barometric pressure.

[0141] The magnetic sensor 180D includes a Hall sensor.

[0142] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0143] Distance sensor 180F is used to measure distance.

[0144] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0145] The 180L ambient light sensor is used to detect ambient light intensity.

[0146] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0147] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy.

[0148] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0149] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, the bone conduction sensor 180M can also be incorporated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information from the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0150] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0151] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to different touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0152] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0153] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0154] The software system of electronic device 100 can adopt a layered architecture. Figure 2 This is a schematic diagram of the software structure of the electronic device 100 provided in the embodiments of this application.

[0155] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0156] The application layer may include a series of application packages. In this embodiment, the application package may include a camera, a gallery, etc.

[0157] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, which may include camera management and camera devices. The camera access interface is used to provide application programming interfaces and programming frameworks for camera applications.

[0158] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0159] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include runtime code and data that implement the shooting methods provided in the embodiments of this application.

[0160] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0161] The hardware layer is the physical part of a computer system, comprising various hardware components used to perform calculations and process data. In the field of image processing, the hardware layer typically involves a series of specially designed hardware components, which may include image sensors, image signal processors, digital signal processors, and image processors. An image sensor in the hardware layer corresponds to a camera device in the hardware abstraction layer. For specific descriptions of image sensors, image signal processors, digital signal processors, and image processors, please refer to the above. Figure 1 The relevant descriptions will not be repeated here.

[0162] Furthermore, the camera device driver drives the image sensor (e.g., an RGBW sensor) to acquire images and drives the image signal processor to preprocess the images. The digital signal processor driver drives the digital signal processor to process the images. The image processor driver drives the graphics processor to process the images.

[0163] The following describes, in conjunction with the above software structure, the software and hardware workflows of the embodiments of this application when taking pictures using the electronic device 100.

[0164] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application. This then sends a command to start the camera by calling the camera device (or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver can then activate the corresponding camera's image sensor (e.g., an RGBW sensor) and acquire image light signals through the image sensor. One camera device in the camera hardware abstraction layer corresponds to one image sensor in the hardware layer.

[0165] Then, the camera's image sensor (e.g., an RGBW sensor) can transmit the acquired image light signal to the image signal processor for preprocessing to obtain the image electrical signal, which is the original image (e.g., the RGBW original image), and transmit the original image to the camera hardware abstraction layer through the camera device driver.

[0166] The camera hardware abstraction layer (HIBIL) can send raw images to a camera algorithm library. The camera algorithm library stores program code that implements the imaging method based on an RGBW sensor provided in this application embodiment. Based on a digital signal processor and an image processor, the camera algorithm library executes the above code, enabling the electronic device 100 to perform some or all of the steps of the imaging method based on an RGBW sensor provided in this application embodiment.

[0167] The camera algorithm library can send processed images (such as RGB images) to the camera hardware abstraction layer. The camera hardware abstraction layer can then display the processed images. Simultaneously, the camera algorithm library can perform various image processing tasks, such as noise reduction, color correction, and contrast adjustment, to improve image quality or perform specific computer vision tasks.

[0168] Optionally, through the interface provided by the camera hardware abstraction layer, the original image or the image processed by the camera algorithm library can be stored in a specific storage unit. Furthermore, in response to the user's operation of opening the gallery application, such as clicking the gallery application icon and then clicking to view the image, the gallery application calls the corresponding interface to access the image data in the storage unit, and then calls the image decoding library to decode the stored image data into an image that can be displayed on the screen for display in the application.

[0169] It should be noted that, in the process of taking pictures using the electronic device 100 in this application embodiment, the original image obtained and the image processed by the camera algorithm library can be a static image obtained based on the camera's image sensor or a dynamic video frame. This application embodiment does not limit this.

[0170] Based on the above description of the hardware and software structure of the electronic device 100, the imaging method based on the RGBW sensor provided in the embodiments of this application will be introduced next.

[0171] For example, please see Figure 3 , Figure 3 This application provides a flowchart illustrating an imaging method based on an RGBW sensor, which can be applied to the above-mentioned methods. Figure 1 The hardware structure of electronic devices and Figure 2 The software structure of an electronic device may include, but is not limited to, the following steps S301-S303.

[0172] Step S301: Acquire the original RGBW image captured by the RGBW sensor.

[0173] Specifically, electronic devices acquire RGBW raw images (RGBW RAW) captured by RGBW sensors. Since RGBW sensors have added white (W) pixels compared to traditional RGGB sensors, they can capture more photons, thereby improving their light sensitivity in low-light environments.

[0174] In one possible implementation, the pixels of the RGBW sensor include multiple pixel units, each pixel unit consisting of one of the three types of pixels (R, G, B) and a neighboring W pixel. The pixel arrangement of the original RGBW image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor.

[0175] Specifically, since the pixels of the RGBW sensor consist of multiple pixel units composed of one of the three types of pixels (R, G, B) and a neighboring W pixel, and the pixel arrangement of the RGBW original image is consistent with that of the sensor, the pixels in the RGBW original image also have the same pixel units as the RGBW sensor. Furthermore, the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor, and the channel value of each pixel in the RGBW original image corresponds to the pixel of the RGBW sensor. This ensures that the acquired RGBW original image can retain the information of each pixel, which helps to maintain the detail and clarity of the image in subsequent processing stages, thereby improving the realism and quality of the image.

[0176] In one possible implementation, the total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the RGBW sensor.

[0177] In this RGBW sensor, the ratio of the number of RGB pixels to the number of W pixels in each pixel unit can be an integer ratio, such as 1:1, 1:2, 1:3, 1:8, 1:15, etc., or a fractional ratio, such as 4:5, 7:9, etc. The embodiments of this application do not specifically limit this ratio.

[0178] For example, please see Figure 4 , Figure 4 This is a schematic diagram of the pixel arrangement of some RGBW sensors provided in the embodiments of this application.

[0179] like Figure 4 As shown in (A), this RGB3W sensor is based on a traditional Bayer array RGGB sensor, where each pixel (i.e., RGB pixel) is supplemented with three white W pixels to form a new pixel unit. The ratio of RGB pixels to W pixels in each pixel unit is 1:3. The pixel array of the RGGB original image acquired based on this RGB3W sensor and... Figure 4 The pixel arrangement shown in (A) is consistent, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor. Similarly, as Figure 4 As shown in (B), this RGB8W sensor is based on a traditional Bayer array RGGB sensor, where each pixel (i.e., an RGB pixel) is supplemented with 8 white W pixels to form a new pixel unit. The ratio of RGB pixels to W pixels in each pixel unit is 1:8. The pixel array of the RGGB original image acquired based on this RGB8W sensor and... Figure 4The pixel arrangement shown in (B) is consistent, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor. For example... Figure 4 As shown in (C), this RGB15W sensor is based on a traditional Bayer array RGGB sensor, where each pixel (i.e., RGB pixel) is supplemented with 15 white W pixels to form a new pixel unit. The ratio of RGB pixels to W pixels in each pixel unit is 1:15. The pixel array of the RGGB original image acquired based on this RGB15W sensor and... Figure 4 The pixel arrangement shown in (C) is consistent, and the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor. In summary, the RGBW sensor provided in this application provides that the channel value of each R, G, B, and W pixel in the generated RGBW original image is determined based on the intensity of light of the corresponding wavelength collected by the pixel at the corresponding position of the RGBW sensor, and converted into a digital signal to represent the intensity or brightness value of each pixel in the corresponding color channel. This ensures that the acquired RGBW original image can retain the information of each pixel, which helps maintain the detail and clarity of the image in subsequent processing stages, thereby improving the realism and quality of the image.

[0180] It is understood that the embodiments of this application only exemplarily introduce several pixel arrangement methods of RGBW sensors. In other embodiments, the pixels of RGBW sensors may also present other different arrangement methods, which are not limited in this application.

[0181] Since the W pixels in an RGBW sensor are used to capture luminance information and the RGB pixels are used to capture color information, as the number of white (W) filters (pixels) in each pixel unit of the RGBW sensor increases, the number of RGB pixels compared to the number of W pixels in a pixel unit decreases. While the light sensitivity of the RGBW sensor increases, its color sensitivity decreases, which can easily affect the color balance of the image generated by fusing RGB and W pixel data. Therefore, this embodiment of the application can increase the total pixel value of the RGBW sensor to increase the number of pixel units, so that the RGBW sensor has a sufficient number of RGB pixels to capture enough color information. Specifically, an RGBW sensor with an appropriate resolution (i.e., total pixel value) can be selected to acquire images based on the ratio of RGB pixels to W pixels in each pixel unit of the RGBW sensor. Furthermore, the smaller the ratio of RGB pixels to W pixels in each pixel unit, the larger the resolution (i.e., total pixel value) of the corresponding RGBW sensor. For example, when the ratio of RGB pixels to W pixels in a pixel unit is 1:8, the resolution (i.e., the total pixel value) of the corresponding RGBW sensor is 108M pixels; when the ratio of RGB pixels to W pixels in a pixel unit is 1:15, the resolution (i.e., the total pixel value) of the corresponding RGBW sensor needs to be approximately 200 million pixels to provide enough pixels to capture spatial details, thereby meeting the needs of different application scenarios for color and brightness information, improving the color accuracy and reproduction of the final generated image. At the same time, the increase in the total pixel value of the RGBW sensor can also improve the resolution of the original RGBW image, making the details and clarity of the final generated image better, thus better improving the image quality when shooting in low light conditions, and improving the user's shooting experience.

[0182] In one possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; the step of acquiring the RGBW raw image captured by the RGBW sensor may include: acquiring the long-exposure RGBW raw image captured by the RGBW sensor with a long exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor with a short exposure.

[0183] Specifically, based on the exposure time of the RGBW sensor, RGBW raw images can be divided into two types: long-exposure RGBW raw images acquired by the RGBW sensor with long exposure time, and regular-exposure RGBW raw images acquired with normal exposure time. The regular exposure time can be adjusted according to the actual shooting scene, and can be either a shorter or longer exposure time to achieve the desired exposure effect. In low-light conditions, because the light-sensing ability of RGB pixels is weak and the number of photons received is small, the noise level increases. Therefore, by appropriately extending the exposure time of the RGBW sensor (i.e., long exposure time), the RGB pixels can accumulate more photon signals, reducing the noise generated by the RGB pixels to a certain extent and improving the signal-to-noise ratio, thereby improving image quality in low-light environments. The W pixel has a stronger light sensitivity, so under the same lighting conditions, the W pixel can receive more photons than the RGB pixel. Using a normal exposure time can ensure that the brightness information in the image is captured properly, avoiding overexposure caused by too much light received by the W pixel. It can also avoid motion blur or noise caused by long exposure, thus capturing the brightness information of the instantaneous scene more accurately and improving image quality.

[0184] Step S302: Process the W channel of the RGBW original image to obtain the W original image, and process the RGB channels of the RGBW original image to obtain the RGGB original image.

[0185] Specifically, the W channel in the RGBW original image is processed to obtain the W original image (W RAW), and the RGB channel in the RGBW original image is processed to obtain the RGB original image (RGB RAW), so that the brightness and color information of the image can be adjusted in a targeted manner later.

[0186] In one possible implementation, obtaining the W original image based on the W channel processing of the RGBW original image may include: calculating the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image; obtaining the W original image based on the average channel value; and the channel value of each W pixel in the W original image being sequentially equal to the average channel value.

[0187] Specifically, regarding how to obtain the W original image based on the W channel processing of the RGBW original image, it can be included as follows: First, calculate the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image. Then, use the calculated average channel value as the channel value of each W pixel in the W original image, thereby effectively extracting the brightness information in the RGBW original image to generate the W original image. This ensures that the brightness distribution in the W original image can be consistent with that of the RGBW original image, so that the signal-to-noise ratio of the RGBW original image can be adjusted using the W original image, thereby improving the image quality in low-light environments and enhancing the user's shooting experience.

[0188] In one possible implementation, the step of processing the RGB channels of the original RGBW image to obtain the original RGGB image may include: selecting a target pixel from each pixel unit in the long-exposure RGBW original image, wherein the target pixel is a pixel of the R, G, and B channels in the pixel unit; generating the original RGGB image based on the channel values ​​of the target pixel; wherein the channel values ​​of each pixel in the original RGGB image correspond to and are equal to the channel values ​​of the target pixel in sequence.

[0189] Specifically, the process of obtaining the RGGB raw image from the RGB channels of the RGBW raw image can be summarized as follows: First, select pixels (i.e., target pixels) from each pixel unit in the long-exposure RGBW raw image, representing the R, G, and B channels. Then, use the channel values ​​of the selected target pixels sequentially as the channel values ​​of each pixel in the RGGB raw image (RGGB RAW). This ensures that the color information of the generated RGGB raw image is consistent with that of the RGBW raw image, thereby improving the color accuracy and fidelity of subsequent adjustments to the RGGB raw image, enhancing the overall quality of the captured image, and improving the user's shooting experience.

[0190] Step S303: Adjust the signal-to-noise ratio of the RGGB original image based on the W original image to generate the first image.

[0191] Specifically, by adjusting the signal-to-noise ratio of the original RGB image using the original W image, and by adjusting the RGB channel data using the brightness information of the original W image, the subtle textures and details in the original RGB image can be preserved while retaining the color information of the RGB channels as much as possible to avoid color loss or distortion. This generates a first image with a high signal-to-noise ratio, which improves the clarity and realism of the image obtained after visualization processing based on the first image under low-light conditions. This improves the image quality in low-light environments and enhances the user's shooting experience.

[0192] Optionally, depending on the current shooting mode of the RGBW sensor, different methods can be used to adjust the signal-to-noise ratio of the RGGB original image based on the W original image to generate the first image. In one possible implementation, if the RGBW sensor is currently in shooting mode, a specific implementation of how to adjust the signal-to-noise ratio of the RGGB original image based on the W original image to obtain the first image in step S303 of the above method can be found in [reference needed]. Figure 5A , Figure 5A This application provides an example flowchart of an imaging method based on an RGBW sensor in a shooting mode, which can be applied to the above-mentioned... Figure 1 The hardware structure of electronic devices and Figure 2 In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may also include steps S5A01-S5A05.

[0193] Step S5A01: Perform feature extraction on the original W image and the original RGGB image respectively to obtain the first feature image corresponding to the original W image and the second feature image corresponding to the original RGGB image.

[0194] Specifically, by performing feature extraction on the original W image and the original RGGB image respectively, a first feature image corresponding to the original W image and a second feature image corresponding to the original RGGB image are obtained, thereby effectively extracting the brightness features in the original W image and the color features in the original RGGB image, so that the image can be adjusted in a targeted manner according to different needs in the future.

[0195] Step S5A02: Merge the first feature image and the second feature image to generate the third feature image.

[0196] Specifically, by merging the first feature image (brightness features of the original W image) and the second feature image (color features of the original RGGB image), a third feature image is generated that contains the brightness features of the original W image and the color features of the original RGGB image. This allows the key information of the original RGBW image in terms of color and brightness to be preserved, while the noise generated in the original RGGB image due to insufficient light is also suppressed to a certain extent, thereby improving the signal-to-noise ratio of the third feature image.

[0197] Step S5A03: Reconstruct the RGB image after denoising and de-mosaic processing based on the third feature image.

[0198] Specifically, the RGB image after denoising and de-mosaic processing is the first image. Reconstructing the denoised and de-mosaiced RGB image (i.e., the first image) based on this third feature image restores the clarity and detail of the third feature image. While preserving the subtle textures and details in the original RGGB image, it minimizes the loss of color information in the RGB channels to improve color accuracy. This results in improved clarity and realism in subsequent visualization processing based on the first image under low-light conditions, thereby enhancing the image quality captured in low-light environments and improving the user's shooting experience.

[0199] Step S5A04: Perform color correction on the denoised and de-mosaiced RGB image to generate a second image in RGB format.

[0200] Step S5A05: Display the second image on the screen.

[0201] Specifically, after generating a first image (i.e., an RGB image after denoising and de-mosaicing) based on the original W image by adjusting the signal-to-noise ratio of the original RGGB image, color correction can be performed on this first image (i.e., the RGB image after denoising and de-mosaicing). For example, automatic white balance processing can be performed using the automatic white balance module in the signal processor (ISP) to eliminate color temperature deviations in the first image caused by different light sources, ensuring that white objects appear true white in the image. Then, the color transformation matrix module in the ISP converts the color space of the first image to a range conforming to a specific standard, avoiding color differences on different devices due to different color space representations used by different devices and sensors. This results in a more accurate and consistent color display on the screen for the converted second image. Through the embodiments of this application, it can be ensured that the second RGB format image after color correction can present consistent color performance and color accuracy on the screens of different devices, thereby improving the user's shooting experience.

[0202] Optionally, if the RGBW sensor is currently in photo capture mode, a specific implementation of how to adjust the signal-to-noise ratio of the RGGB original image based on the W original image to obtain the first image in the above method step S303 can also be based on a dual-stream fusion algorithm model.

[0203] Please see Figure 5B , Figure 5B This is an example flowchart of an imaging method based on an RGBW sensor in another shooting mode provided in this application embodiment. This method can be applied to the above... Figure 1 The hardware structure of electronic devices and Figure 2In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may also include steps S5B01-S5B04.

[0204] S5B01: Input the W original image and the RGGB original image into the dual-stream fusion algorithm model.

[0205] S5B02: Using a dual-stream fusion algorithm model, the signal-to-noise ratio of the RGGB original image is adjusted based on the W original image to output the first image.

[0206] Specifically, when the RGBW sensor is in shooting mode, the original W image and the original RGGB image can be input into a dual-stream fusion algorithm model. This model adjusts the signal-to-noise ratio (SNR) of the RGGB image based on the original W image, resulting in a high SNR first image that retains image details and color information. Furthermore, this dual-stream fusion algorithm model is a pre-trained deep learning network model (e.g., a convolutional neural network model). Using this model ensures the stability of the output image (i.e., the first image) quality under different conditions, improving the clarity and realism of the image obtained after visualization processing based on this first image in low-light conditions. This enhances the image quality captured in low-light environments and improves the user's shooting experience.

[0207] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image.

[0208] Specifically, the dual-stream fusion algorithm model can include a color restoration sub-model and an image fusion sub-model. Further, the original RGGB sample image is input into the color restoration sub-model, and trained using a first loss function to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The original W sample image and the original RGGB sample image are input into the image fusion sub-model, and trained using a second and a third loss function. This allows the image fusion sub-model to fuse the input dual-stream images (i.e., the original W sample image and the original RGGB sample image) to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image. Through this embodiment, the trained color restoration sub-model can be used for preview imaging, allowing users to obtain a rough image effect before taking a photo, thereby improving the user experience. The trained image fusion sub-model can extract detail and color information from the original W sample image and the original RGGB sample image. By fusing the original W sample image and the original RGGB sample image, the noise present in the single original RGGB sample image is reduced, which improves the signal-to-noise ratio of the final high-resolution RGB image. While making the details clearer and more realistic, it can also maintain color balance, thereby improving the image quality of the image taken in low light environment and enhancing the user's shooting experience.

[0209] In one possible implementation, the original RGGB sample image and the original W sample image are obtained by degradation based on a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; and the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0210] Specifically, since training the dual-stream fusion algorithm model requires paired low-resolution (LR) and high-resolution (HR) images as training data pairs, and obtaining low-resolution LR images (i.e., the original RGGB sample images and the original W sample images) is difficult, the original RGGB images (i.e., the original RGGB sample images) and the original W images (i.e., the original W sample images) generated by degradation from easily obtainable high-resolution RGB images (i.e., high-resolution RGB sample images) can be used to train the dual-stream fusion algorithm model. This reduces the cost of model training, increases the diversity of the training dataset, improves the model's generalization ability and robustness, and enhances the stability of the image quality output by the dual-stream fusion algorithm model under different conditions. Furthermore, for the color restoration sub-model, the color information difference between the low-resolution RGB image and the high-resolution RGB sample image is evaluated using a first loss function, enabling the color restoration sub-model to accurately restore the color information of the original RGGB sample image, thereby improving the color reproduction accuracy of the image and making the final image more realistic and natural. For the image fusion sub-model, a second loss function is used to evaluate the differences in detail and color information between the high-resolution RGB image and the high-resolution RGB sample image. This ensures that the image fusion sub-model does not lose important details and color information during the image fusion process, thereby improving the clarity and color reproduction of the high-resolution RGB image output by the model. Furthermore, a third loss function is used to evaluate the color component differences between the low-resolution RGB image output by the color restoration sub-model and the high-resolution RGB image output by the image fusion sub-model. This ensures that the final image generated by the image fusion sub-model (i.e., the high-resolution RGB image) has consistent colors with the low-resolution RGB image generated by the color restoration sub-model for preview, further improving the color reproduction of the image fusion sub-model.

[0211] For example, see Figure 5C , Figure 5C This is a schematic diagram of the network structure of a dual-stream fusion algorithm model provided in an embodiment of this application. Figure 5C As shown, the network structure of this two-stream fusion algorithm model mainly includes an M1 module (i.e., the color restoration sub-model) and an M2 module (i.e., the image fusion sub-model). Firstly, based on high-resolution RGB sample images I... GT Degradation is performed, and the resulting original RGGB and W sample images are used as input images to train the two-stream fusion algorithm model. Then, the original RGGB sample images are used as input images to the M1 module (i.e., the color restoration sub-model) to obtain the output I. Ir(That is, a low-resolution RGB image), and then constrain the M1 module (that is, the color restoration sub-model) through loss1 (that is, the first loss function), and evaluate I based on the calculated loss value of loss1 (that is, the first loss function). Ir (That is, low-resolution RGB images) and I GT The color information differences between (i.e., high-resolution RGB sample images) are analyzed, and then the parameters of the M1 module (i.e., the color restoration sub-model) are continuously adjusted through backpropagation until the loss value of loss1 (i.e., the first loss function) is less than a preset threshold. This ensures that the M1 module (i.e., the color restoration sub-model) can accurately restore the color information of the original RGGB sample image. Through the embodiments of this application, the trained M1 module (i.e., the color restoration sub-model) can be used for preview imaging, allowing users to obtain a rough image effect before taking a photo, thereby improving the user experience. It is understood that in some embodiments, loss1 (i.e., the first loss function) can be the mean squared error loss function (MSE Loss) and the perception loss function, or other loss functions; this application embodiment does not specifically limit this.

[0212] Furthermore, by freezing the M1 module (i.e., the color restoration sub-model), that is, stopping the training of the M1 module (i.e., the color restoration sub-model), and calculating the latest output I... Ir (That is, the color component I of a low-resolution RGB image) Ir，CrCb Next, the M2 module (i.e., the image fusion sub-model) is trained. The original images of the RGGB samples and the original images of the W samples are used as input images to the M2 module (i.e., the image fusion sub-model) for image fusion, resulting in the output I. hr (That is, a high-resolution RGB image), and then constrain the M2 module (that is, the image fusion sub-model) through loss2 (that is, the second loss function), and evaluate I based on the calculated loss value of loss2 (that is, the second loss function). hr (That is, a high-resolution RGB image) and I GT The model identifies subtle differences and color information between high-resolution RGB sample images. Then, it uses backpropagation to continuously adjust the parameters of the M2 module (the image fusion sub-model) until the loss function (loss2) is less than a preset threshold. This ensures that the image fusion sub-model does not lose important details and color information during image fusion, thereby improving the output I of the M2 module (the image fusion sub-model). hrThe sharpness and color reproduction of the image (i.e., the high-resolution RGB image) are then calculated. Finally, the latest output I of the M2 module (i.e., the image fusion sub-model) is calculated. hr (That is, the color component I of a high-resolution RGB image) hr，CrCb Then, loss3 (the third loss function) is used to make the color component I... Ir，CrCb and color component I hr，CrCb To maintain consistency and ensure that the I generated by the M2 module (i.e., the image fusion sub-model) remains the same. hr (That is, a high-resolution RGB image) and the I-type image generated by the M1 module (that is, the color restoration sub-model) that can be used for preview. Ir (That is, the low-resolution RGB image) has consistent colors, thereby further improving the color reproduction of the M2 module (that is, the image fusion sub-model). The M2 module (that is, the image fusion sub-model) trained through the embodiments of this application serves as a dual-stream fusion algorithm model, which can not only improve the clarity and color reproduction of the output high-resolution RGB image (that is, the first image), but also improve the stability of the image quality output by the dual-stream fusion algorithm model under different conditions, thereby enhancing the user's shooting experience.

[0213] It is understood that the neural network model trained in the embodiments of this application (e.g., the color restoration sub-model and the image fusion sub-model) can be a neural network with a U-Net architecture, or a neural network with other types of architectures; loss2 (i.e., the second loss function) and loss3 (i.e., the third loss function) can be the perception loss function, or other loss functions, and the embodiments of this application do not specifically limit them.

[0214] Step S5B03: Perform color correction on the first image to generate a second image in RGB format.

[0215] Step S5B04: Display the second image on the display screen.

[0216] Specifically, for a detailed description of steps S5B03-S5B04, please refer to the relevant descriptions of steps S5A04-S5A05 above, which will not be repeated here.

[0217] Optionally, if the RGBW sensor is currently in preview or video recording mode, for a specific implementation of how to adjust the signal-to-noise ratio of the RGBGB original image based on the W original image to obtain the first image in the above method step S303, please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a flowchart illustrating an imaging method based on an RGBW sensor in preview or video recording mode, as provided in an embodiment of this application. This method can be applied to the above-mentioned... Figure 1 The hardware structure of electronic devices and Figure 2 In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may also include steps S601-S605.

[0218] Step S601: By comparing the channel values ​​in the original W image with the corresponding channel values ​​in the original RGGB image, determine the type and distribution pattern of noise in the original RGGB image.

[0219] Specifically, by comparing the channel values ​​of the W channel image and the RGGB channel image, the noise type and distribution pattern in the original RGGB image are determined, so as to select an appropriate noise reduction method to reduce the noise in the original RGGB image.

[0220] Step S602: Based on the type and distribution pattern of noise, perform noise reduction processing on the original RGGB image to generate an original RGGB image with a high signal-to-noise ratio.

[0221] Specifically, the high signal-to-noise ratio (SNR) RGGB original image is the first image. In this embodiment, the RGGB original image can be targeted for noise reduction based on the determined noise type and distribution pattern. For example, a specific noise reduction filter or a lightweight neural network model algorithm can be used to accurately remove noise from the RGGB original image while preserving as much detail and color information as possible, resulting in a high SNR RGGB original image (i.e., the first image). This improves the clarity and realism of the image obtained after visualization processing based on the high SNR RGGB original image (i.e., the first image) under low light conditions, thereby improving the image quality captured in low light environments and enhancing the user's shooting experience. In this embodiment, when the RGBW sensor is in preview or video recording mode, an algorithm with lower computational complexity and memory consumption compared to the algorithm design in photo mode is adopted. The internal computing unit of the sensor first performs preliminary noise reduction processing on the W original image and RGGB original image and outputs continuous high signal-to-noise ratio image frames to reduce the computational burden on the subsequent processor (such as CPU). This improves the image quality when the electronic device is previewing or recording video, and also improves the real-time performance and smoothness of previewing or video recording, thereby enhancing the user's shooting experience.

[0222] Step S603: Denoise and remove mosaic from the first image to obtain a third image in RGB format.

[0223] Furthermore, by performing denoising and de-mosaic processing on the first image, the details and color information of the first image (i.e., the original RGGB image with a high signal-to-noise ratio) are restored, which improves the clarity and realism of the generated RGB format third image under low-light conditions, thereby improving the image quality generated after subsequent visualization processing based on the third image, and enhancing the user's shooting experience.

[0224] In one possible implementation, the step of denoising and de-mosaicing the first image to obtain a third image in RGB format includes: inputting the first image into an original image processing network model; and generating the third image by performing denoising and de-mosaicing processing on the original image processing network model.

[0225] Specifically, in this embodiment, a first image (i.e., a high signal-to-noise ratio RGGB original image) can be input into an original image processing network model. The original image processing network model can then perform denoising and de-mosaic processing on the first image and output a third image in RGB format, thereby further improving the quality and clarity of the third image. At the same time, the original image processing network model can adaptively adjust according to different scenes and conditions to adapt to different lighting conditions, environmental changes, and other factors, ensuring that the quality of the processed image remains at a high level. This improves the quality of the image generated after subsequent visualization processing based on the third image, thereby enhancing the user's shooting experience.

[0226] Step S604: Perform color correction on the third image to generate a fourth image in RGB format.

[0227] Step S605: Display the fourth image on the display screen.

[0228] Specifically, after denoising and de-mosaicing the first image (i.e., the high signal-to-noise ratio RGGB original image) to obtain a third image in RGB format, color correction can be performed on the third image, and then the color-corrected fourth image in RGB format can be displayed on the screen. For example, automatic white balance processing can eliminate color temperature deviations in the third image caused by different light sources, allowing white objects to appear true white in the image. Furthermore, a color transformation matrix can be used to convert the color space of the third image to a range conforming to specific standards, avoiding color differences on different devices' screens due to different color spaces used by different devices and sensors. This results in a more accurate and consistent color display of the fourth image on different devices' screens. Through the embodiments of this application, it can be ensured that the color-corrected fourth image in RGB format can present consistent color performance and color accuracy on different devices' screens, thereby improving the user's shooting experience.

[0229] For example, based on the above Figure 4 The RGB8W sensor described in (B) above, combined with the above Figure 3 and Figures 5A-5C The method embodiments described herein are followed by a flowchart illustrating the application of the above method embodiments in the photo-taking mode provided in this application.

[0230] Please see Figure 7 , Figure 7 This is a schematic diagram of the imaging process of an imaging method based on an RGB8W sensor provided in an embodiment of this application. Figure 7 As shown, the RGB8W sensor may include an array of RGBW pixels (photosensitive elements) with a total pixel value (resolution) of 108M, and a computing unit (logic) disposed within the RGB8W sensor. The arrangement of the RGBW pixel (photosensitive element) array is as described above. Figure 4 The pixel arrangement of the RGB8W sensor described in (B) is the same. In shooting mode, the RGBW raw images at different exposure times (i.e., long exposure time and normal exposure time) are first acquired through the pixel array of the RGB8W sensor. Specifically, the pixel array of the RGBW raw image is the same as... Figure 4 The pixel arrangement shown in (B) is consistent, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor, which is 108M.

[0231] Furthermore, within the computing unit (Logic) of the RGB8W sensor, based on the different exposure times (i.e., long exposure time and normal exposure time) of the pixels (photosensitive elements) of the RGB8W sensor during photography, there are an 8-in-1 average calculation (8-in-1 Binning) module and a color pick (Pick Color) module to process the RGBW raw images captured under different exposure times (i.e., long exposure RGBW raw images and normal exposure RGBW raw images) to generate dual-stream RGB raw RAW images (i.e. W raw images and RGGB raw images). Specifically, the 8-in-1 Binning module calculates the average channel value of the W pixels in each pixel unit of the normal exposure RGBW original image. This average channel value is then used as the channel value for each W pixel in the W original image, resulting in the W original image. The Pick Color module selects pixels with R, G, and B channels (i.e., target pixels) from each pixel unit of the long exposure RGBW original image. The channel values ​​of these selected target pixels are then used as the channel values ​​for each pixel in the RGGB original image, generating the RGGB original image. Each image in the dual-stream RGB original image (i.e., the W original image and the RGGB original image) has a width and height that are 1 / 3 of the RGBW original image (i.e., the long exposure RGBW original image and the normal exposure RGBW original image), therefore their resolutions are 1 / 9 of the RGBW original image (i.e., the long exposure RGBW original image and the normal exposure RGBW original image), meaning both the W original image and the RGGB original image are 12M. It is understood that the conventional exposure time in the embodiments of this application can be adjusted according to the actual shooting scene in order to achieve the ideal exposure effect. It can be a shorter exposure time or a longer exposure time. The embodiments of this application do not limit this.

[0232] Furthermore, the dual-stream RGB original images (i.e., the W original image and the RGGB original image) are transmitted to the 2-Flow Fusion module in the processor (CPU) via data transmission. The 2-Flow Fusion module adjusts the signal-to-noise ratio of the RGGB original image based on the W original image, generating a denoised and de-mosaiced RGB image (i.e., the first image). Specifically, the dual-stream RGB original images (i.e., the W original image and the RGGB original image) can be input into a pre-trained 2-Flow Fusion algorithm model to perform image fusion and output a high signal-to-noise ratio first image that retains image details and color information. This improves the clarity and realism of the RGB image obtained after visualization processing based on this first image under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience. The training method of the 2-Flow Fusion algorithm model can be found above. Figure 5C The relevant descriptions of the method embodiments in the document will not be repeated here.

[0233] Then, the image signal processing (ISP) in the processor performs color correction on the denoised and de-mosaiced RGB image (i.e., the first image), thereby generating an RGB image that can be displayed on the screen (i.e., the second image). Specifically, color correction is performed on the denoised and de-mosaiced RGB image (i.e., the first image) through automatic white balance processing and color transformation matrix (AWB / CCM), so that white objects can appear as true white in the RGB image (i.e., the second image), and consistent color performance and color accuracy can be achieved on the screens of different devices, thereby improving the user's shooting experience.

[0234] It is understandable that the above Figure 7 The described method is merely one possible implementation of the imaging method based on an RGB8W sensor in the photo-taking mode provided in this application embodiment. The specific implementation process may vary depending on the type of RGBW sensor, and this application embodiment does not limit it.

[0235] For example, based on the above Figure 4 The RGB8W sensor described in (B) above, combined with the above Figure 3 and Figure 6 The method embodiments described herein are followed by a flowchart illustrating the application of the above method embodiments in preview or video recording modes provided in this application.

[0236] Please see Figure 8 , Figure 8This is a schematic diagram of a preview or video recording process for an imaging method based on an RGB8W sensor provided in an embodiment of this application. Figure 8 As shown, the RGB8W sensor can include an array of RGBW pixels (photosensitive elements) with a total pixel value (resolution) of 108M, and a computing unit (Logic) located inside the RGB8W sensor. Further, within the computing unit (Logic) of the RGB8W sensor, based on the different exposure times (i.e., long exposure time and normal exposure time) of the RGB8W sensor's pixels (photosensitive elements) during preview or video recording, an 8-in-1 binning module and a color pick module are configured to process the RGBW raw images captured at different exposure times (i.e., long exposure RGBW raw images and normal exposure RGBW raw images), respectively, to generate the RGB8W raw image and the RGB8W raw image. The specific description of the arrangement of the RGBW pixel (photosensitive element) array and how the 8-in-1 binning module and the color pick module process the long exposure RGB8W raw image and the normal exposure RGB8W raw image respectively in preview or video recording modes is consistent with the above. Figure 7 The embodiments of the methods described are similar, and their detailed descriptions can be found in [reference needed]. Figure 7 The relevant descriptions will not be repeated here.

[0237] Furthermore, the RGB8W sensor may also include a preprocessing module equipped with a RAW Fusion algorithm. By comparing the channel values ​​of the W channel image and the RGGB channel image, the noise type and distribution pattern in the RGGB original image are determined, and targeted noise reduction processing is performed on the RGGB original image to generate a high signal-to-noise ratio RGGB original image (i.e., the first image). The RAW Fusion algorithm within this preprocessing module employs an algorithm with lower computational complexity and memory consumption compared to the algorithm design in photo mode. The sensor's internal computing unit first performs preliminary noise reduction processing on the W and RGGB original images and outputs continuous high signal-to-noise ratio image frames. This reduces the computational burden on the subsequent processor (such as the CPU), thereby improving the image quality of the electronic device during previewing or video recording in low-light conditions, while also improving the real-time performance and smoothness of previewing or video recording, thus enhancing the user's shooting experience. It should be noted that the RAW Fusion algorithm in this embodiment may use a specific noise reduction filter or a lightweight neural network model algorithm; this embodiment is not limited in this regard.

[0238] The preprocessed high signal-to-noise ratio (SNR) RGGB raw image (i.e., the first image) is then transmitted to the image signal processing (ISP) module in the processor (CPU) via data transmission. Specifically, the high SNR RGGB raw image (i.e., the first image) is first denoised and de-pixelated using a raw image processing network model (RAW net), outputting an RGB format image (i.e., the third image). Further color correction is performed on this RGB format image (i.e., the third image) to generate an RGB image that can be displayed on the screen (i.e., the fourth image). Specifically, automatic white balance processing and color transformation matrix (AWB / CCM) are used to perform color correction on the high SNR RGGB raw image (i.e., the first image), ensuring that white objects appear true white in the RGB image (i.e., the fourth image), and maintaining consistent color performance and accuracy on different device displays, thereby improving the user's shooting experience.

[0239] It is understandable that the above Figure 8 The described method is merely one possible implementation of the imaging method based on an RGB8W sensor in preview or video recording mode provided in this application embodiment. The specific implementation process may vary depending on the type of RGBW sensor, and this application embodiment does not limit it.

[0240] The methods of the embodiments of this application have been described in detail above. It is understood that each device, in order to achieve the corresponding functions, includes hardware structures and / or software modules for executing each function. Based on the units and steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. The devices provided in the embodiments of this application are described below.

[0241] The above describes the imaging method based on an RGBW sensor provided in the embodiments of this application. Next, we will introduce some user interface diagrams of users taking pictures using electronic devices after applying the imaging method based on an RGBW sensor provided in the embodiments of this application.

[0242] For example, please see Figure 9 , Figure 9 This is a schematic diagram of a user interface for taking photos in low-light environments using an electronic device, as provided in an embodiment of this application. This user interface schematic diagram can be... Figure 1 The hardware structure of electronic devices in Figure 2The software architecture implementation. For example... Figure 9 As shown, when a user turns on an electronic device, the device's screen displays the desktop, i.e., user interface 91. User interface 91 may include icons for at least one application (e.g., weather, calendar, email, settings, app store, notes, gallery, phone, SMS, browser, and camera 901, etc.). The application icons, their names, and their locations can be adjusted according to user preferences; this embodiment does not limit this.

[0243] In user interface 91, the user can click the camera 901 control. In response to the click of the camera 901 control, the display screen of the electronic device can display user interface 92, which is in photo-taking mode. User interface 92 may include a preview display box 902 and a photo-taking control 903. The preview display box 902 can be used to preview the scene image 904 (i.e., the fourth image) being captured in low light environment in real time. The scene image 904 (i.e., the fourth image) is generated by obtaining a high signal-to-noise ratio RGGB original image (i.e., the first image) through the imaging method based on the RGBW sensor in preview mode provided in this application embodiment, followed by denoising, de-mosaic processing, and color correction (e.g., automatic white balance / color conversion matrix) of the first image, resulting in an RGB format fourth image that can be displayed on the screen. In addition, user interface 92 may also include a camera switching control 905 and a photo album control 906. The camera switching control 905 can be used to switch the camera capturing images between the front camera and the rear camera, and the photo album control 906 can be used to view photos or videos captured by the electronic device. Furthermore, in response to the user's click on the camera control 903, the electronic device acquires a low-light environment photo 907 (i.e., the second image) generated by the imaging method based on the RGBW sensor in the camera mode provided in the embodiments of this application. The photo 907 (i.e., the second image) can be a second image in RGB format that can be displayed on the screen after color correction (e.g., automatic white balance / color conversion matrix) of the RGB image (i.e., the first image) output by the dual-stream fusion algorithm model provided in the embodiments of this application after denoising and de-mosaic processing.

[0244] Furthermore, in response to the user's click on the album control 906, the electronic device's display screen presents a user interface 93. The user interface 93 may include an image display box 908, which displays the latest photo 907 (i.e., the second image) taken by the electronic device. Through the embodiments of this application, the clarity and realism of the photo 907 taken under low light conditions can be improved, thereby improving the image quality taken under low light conditions and enhancing the user's shooting experience.

[0245] For example, please see Figure 10 , Figure 10 This is a schematic diagram of a user interface for a user to shoot video in a low-light environment using an electronic device, as provided in an embodiment of this application. Figure 10 The diagram shown illustrates the interface for launching the camera after a user turns on their electronic device, along with related descriptions. (See also:) Figure 9 The descriptions of user interfaces 91 and 92 in the previous section will not be repeated here. User interface 92 also includes a video mode control 909. Further, in response to a user clicking the video mode control 909 or other operations that switch the electronic device to video recording mode, the electronic device's display screen presents a video recording mode user interface 94. This user interface 94 may include a preview display box 910 and a shooting control 911. The preview display box 910 can be used to display the scene image 912 (i.e., the fourth image) being captured in the current low-light environment, allowing the user to preview the captured scene video in real time. A detailed description of the scene image 912 (i.e., the fourth image) can be found above. Figure 9 The description of scene image 904 is omitted here. Furthermore, the user interface 94 may also include a photo album control 913, which can be used to view the latest video captured by the electronic device. Further, in response to the user clicking the shooting control 911, the electronic device can acquire consecutive scene images 912 (i.e., the fourth image) to generate the latest captured video 914, until the electronic device receives another click from the user on the shooting control 911. Further in response to the user clicking the photo album control 913, the electronic device's display screen presents a user interface 95, which may include a video display frame 915. The video display frame 915 displays the latest captured video 914. Each frame of this video 914 is generated by using the imaging method based on an RGBW sensor in the video shooting mode provided in this application embodiment. After obtaining a high signal-to-noise ratio RGGB original image (i.e., the first image), the first image is then denoised, de-mosaiced, and color corrected (e.g., automatic white balance / color conversion matrix) to generate a fourth image in RGB format that can be displayed on the screen. Through the embodiments of this application, the clarity and realism of video 914 shot under low light conditions can be improved, thereby improving the image quality shot in low light environments and enhancing the user's shooting experience.

[0246] It should be noted that, Figures 9-10 The user interface diagram of the electronic device shown is an exemplary illustration of the embodiments of this application. The user interface diagram of the electronic device may also be of other styles. The number and specific functions of the controls shown in the above user interface are merely exemplary descriptions, and the embodiments of this application do not limit them.

[0247] The methods provided in the embodiments of this application have been described in detail above. The devices provided in the embodiments of this application are described below. For example, please refer to... Figure 11 , Figure 11 This is a schematic diagram of an imaging device based on an RGBW sensor provided in an embodiment of this application. The imaging device 1100 can be applied to an electronic device, which includes the RGBW sensor. The imaging device 1100 may include an acquisition unit 1101, a first processing unit 1102, and an adjustment unit 1103. Detailed descriptions of each unit are as follows:

[0248] Acquisition unit 1101 is used to acquire the original RGBW image collected by the RGBW sensor;

[0249] The first processing unit 1102 is used to process the W channel of the RGBW original image to obtain the W original image, and to process the RGB channel of the RGBW original image to obtain the RGGB original image.

[0250] The adjustment unit 1103 adjusts the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

[0251] Most existing imaging methods are designed based on RGB (Red-Green-Green-Blue) sensors. When shooting in low-light environments, the weak light sensitivity of RGB sensors leads to increased image noise and a decreased signal-to-noise ratio, resulting in reduced image sharpness and affecting basic image quality. To address this problem, this application proposes an imaging device based on an RGBW (Red-Green-Blue-White) sensor with better light sensitivity. Furthermore, to avoid affecting the color balance of the image generated after fusing RGB and W pixel data, this application provides a fusion strategy based on an RGBW sensor. Specifically, this application uses an acquisition unit 1101 to acquire the original RGBW image (RGBWRAW) collected by the RGBW sensor. Since the RGBW sensor adds white (W) pixels compared to the traditional RGGB sensor, it can capture more photons, thus improving its light sensitivity in low-light environments. Furthermore, the first processing unit 1102 processes the W channel of the RGBW original image to obtain the W original image (W RAW), and processes the RGB channels of the RGBW original image to obtain the RGB original image (RGB RAW), so that the brightness and color information of the image can be adjusted in a targeted manner later. Further, the adjustment unit 1103 adjusts the signal-to-noise ratio of the RGB original image based on the W original image. By using the brightness information of the W original image to adjust the RGB channel data, the subtle textures and details in the RGB original image can be preserved while retaining the color information of the RGB channels as much as possible to avoid color loss or distortion, generating a first image with a high signal-to-noise ratio. This improves the clarity and realism of the image obtained after visualization processing based on the first image under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0252] In one possible implementation, the pixels of the RGBW sensor include multiple pixel units, each pixel unit consisting of one of the three types of pixels (R, G, B) and a neighboring W pixel. The pixel arrangement of the original RGBW image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor.

[0253] In one possible implementation, the total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the RGBW sensor.

[0254] In one possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; the acquisition unit 1101 is specifically used for:

[0255] Acquire the long-exposure RGBW raw image captured by the RGBW sensor during long-exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor during regular-exposure.

[0256] In one possible implementation, the first processing unit 1102 is specifically used for:

[0257] Calculate the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image;

[0258] The W original image is obtained based on the average channel value; the channel value of each W pixel in the W original image corresponds to the average channel value in sequence.

[0259] In one possible implementation, the first processing unit 1102 is specifically used for:

[0260] Select a target pixel from each pixel unit in the long exposure RGBW original image, wherein the target pixel is a pixel of the R, G, and B channels in the pixel unit;

[0261] The RGGB original image is generated based on the channel values ​​of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel.

[0262] In one possible implementation, if the RGBW sensor is currently in photo-taking mode, the adjustment unit 1103 is specifically used for:

[0263] Feature extraction is performed on the original W image and the original RGGB image respectively to obtain a first feature image corresponding to the original W image and a second feature image corresponding to the original RGGB image;

[0264] The first feature image and the second feature image are merged to generate a third feature image;

[0265] The RGB image after denoising and de-mosaic processing is reconstructed based on the third feature image, and the RGB image after denoising and de-mosaic processing is the first image.

[0266] In one possible implementation, if the RGBW sensor is currently in photo-taking mode, the adjustment unit 1103 is specifically used for:

[0267] The W original image and the RGGB original image are input into the dual-stream fusion algorithm model;

[0268] Using the dual-stream fusion algorithm model, the signal-to-noise ratio of the RGGB original image is adjusted based on the W original image, and the first image is output.

[0269] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image.

[0270] In one possible implementation, the original RGGB sample image and the original W sample image are obtained by degradation based on a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; and the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0271] In one possible implementation, the electronic device further includes a display screen; the imaging device further includes:

[0272] The first color correction unit is used to perform color correction on the first image and generate a second image in RGB format;

[0273] A first display unit is used to display the second image on the display screen.

[0274] In one possible implementation, if the RGBW sensor is currently in preview or video recording mode, the adjustment unit 1103 is specifically used for:

[0275] By comparing the channel values ​​in the original W image with the corresponding channel values ​​in the original RGGB image, the type and distribution pattern of noise in the original RGGB image are determined.

[0276] Based on the type and distribution pattern of the noise, the original RGGB image is denoised to generate a high signal-to-noise ratio (SNR) RGGB image, which is the first image.

[0277] In one possible implementation, the imaging device further includes:

[0278] The second processing unit is used to perform denoising and de-mosaic processing on the first image to obtain a third image in RGB format.

[0279] In one possible implementation, the second processing unit is specifically used for:

[0280] The first image is input into the original image processing network model;

[0281] The third image is generated after denoising and depixelation processing using the original image processing network model.

[0282] In one possible implementation, the electronic device further includes the display screen, and the imaging device further includes:

[0283] The second color correction unit is used to perform color correction on the third image and generate a fourth image in RGB format;

[0284] The second display unit is used to display the fourth image on the display screen.

[0285] It should be noted that the functions of each unit in the imaging device 1100 described in the embodiments of this application can be found in the relevant descriptions of the above method embodiments, and will not be repeated here. It is understood that the devices and methods provided in the embodiments of this application can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of modules or units described above is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0286] For example, please see Figure 12 , Figure 12 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. For example... Figure 12 As shown, the electronic device 1200 includes an RGBW sensor 1201, at least one processor 1202, and a memory 1203. The processor 1202 is coupled to the memory 1203; in this embodiment, the coupling can be a communication connection, an electrical connection, or other forms. Furthermore, the electronic device 1200 provided in this embodiment may also include at least one display screen 1204. Figure 12(Only one is shown). The RGBW sensor 1201, processor 1202, memory 1203, and display screen 1204 can be connected via bus 1205. Specifically, memory 1203 is used to store program instructions. Processor 1202 is used to call the program instructions stored in memory 1203, so that electronic device 1200 can execute the steps in the control method provided in the embodiments of this application. The description of each component and related steps can be referred to above, and will not be repeated here.

[0287] It should be noted that the electronic device 1200 provided in this application embodiment may include more or fewer components than those shown in the figure, or combine some components, or split some components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or any combination of software and hardware.

[0288] This application provides a computer-readable storage medium storing a computer program that is executed by a processor of the aforementioned routing device to implement the steps performed by the routing device in the network recovery method provided in this application; or the computer program is executed by a processor of the aforementioned electronic device to implement the steps performed by the electronic device in the network recovery method provided in this application.

[0289] This application provides a computer program including instructions. The computer program is executed by the router device processor to implement the steps performed by the router device in the network recovery method provided in this application; or the computer program is executed by the electronic device processor to implement the steps performed by the electronic device in the network recovery method provided in this application.

[0290] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0291] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps may be performed in other orders or simultaneously, or some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. It should also be noted that the features and functions of two or more devices according to this disclosure can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0292] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0293] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0294] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. An imaging method based on a red-green-blue-white (RGBW) sensor, characterized in that, Applied to an electronic device, the electronic device including the RGBW sensor, the method includes: Acquire the original RGBW image captured by the RGBW sensor; The W channel of the original RGBW image is processed to obtain the W original image, and the RGB channel of the original RGBW image is processed to obtain the RGGB original image; The signal-to-noise ratio of the original RGGB image is adjusted based on the original W image to generate a first image; If the RGBW sensor is currently in image capture mode, adjusting the signal-to-noise ratio of the RGBGB original image based on the W original image to obtain the first image includes: The W original image and the RGGB original image are input into the dual-stream fusion algorithm model; The dual-stream fusion algorithm model during the training phase includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image. The loss value of the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image, so that the parameters of the color restoration sub-model are continuously adjusted by the backpropagation algorithm during training until the loss value of the first loss function is less than a preset threshold. During the inference phase, the image fusion sub-model in the trained dual-stream fusion algorithm model is used to adjust the signal-to-noise ratio of the RGGB original image based on the W original image, and the first image is output.

2. The method according to claim 1, characterized in that, The pixels of the RGBW sensor include multiple pixel units. Each pixel unit consists of one of the three types of pixels (R, G, B) and a neighboring W pixel. The pixel arrangement of the original RGBW image is consistent with the pixel arrangement of the RGBW sensor. The resolution of the original RGBW image is equal to the total pixel value of the RGBW sensor.

3. The method according to claim 2, characterized in that, The total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the RGBW sensor.

4. The method according to claim 2, characterized in that, The RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; acquiring the RGBW raw image captured by the RGBW sensor includes: Acquire the long-exposure RGBW raw image captured by the RGBW sensor during long-exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor during regular-exposure.

5. The method according to claim 4, characterized in that, The process of processing the W channel of the original RGBW image to obtain the original W image includes: Calculate the average channel value of W pixels in each pixel unit of the normally exposed RGBW original image; The W original image is obtained based on the average channel value; the channel value of each W pixel in the W original image corresponds to the average channel value in sequence.

6. The method according to claim 4 or 5, characterized in that, The process of processing the RGB channels of the original RGBW image to obtain the original RGGB image includes: Select a target pixel from each pixel unit in the long exposure RGBW original image, wherein the target pixel is a pixel of the R, G, and B channels in the pixel unit; The RGGB original image is generated based on the channel values ​​of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel.

7. The method according to any one of claims 1-5, characterized in that, If the RGBW sensor is currently in image capture mode, adjusting the signal-to-noise ratio of the RGBGB original image based on the W original image to obtain the first image includes: Feature extraction is performed on the original W image and the original RGGB image respectively to obtain a first feature image corresponding to the original W image and a second feature image corresponding to the original RGGB image; The first feature image and the second feature image are merged to generate a third feature image; The RGB image after denoising and de-mosaic processing is reconstructed based on the third feature image, and the RGB image after denoising and de-mosaic processing is the first image.

8. The method according to claim 1, characterized in that, The original RGGB sample image and the original W sample image are obtained by degradation based on the high-resolution RGB sample image; the second loss function is used to evaluate the differences in detail and color information between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the differences in color components between the low-resolution RGB image and the high-resolution RGB image.

9. The method according to any one of claims 1-5, characterized in that, The electronic device further includes a display screen; the method further includes: Perform color correction on the first image to generate a second image in RGB format; The second image is displayed on the display screen.

10. The method according to any one of claims 1-5, characterized in that, If the RGBW sensor is currently in preview or video recording mode, adjusting the signal-to-noise ratio of the RGBGB original image based on the W original image to obtain the first image includes: By comparing the channel values ​​in the original W image with the corresponding channel values ​​in the original RGGB image, the type and distribution pattern of noise in the original RGGB image are determined. Based on the type and distribution pattern of the noise, the original RGGB image is denoised to generate a high signal-to-noise ratio (SNR) RGGB image, which is the first image.

11. The method according to claim 10, characterized in that, The method further includes: The first image is denoised and depixelated to obtain a third image in RGB format.

12. The method according to claim 11, characterized in that, The step of denoising and depixelating the first image to obtain a third image in RGB format includes: The first image is input into the original image processing network model; The third image is generated after denoising and depixelation processing using the original image processing network model.

13. The method according to claim 11, characterized in that, The electronic device further includes a display screen, and the method further includes: Perform color correction on the third image to generate a fourth image in RGB format; The fourth image is displayed on the display screen.

14. An imaging device based on an RGBW sensor, characterized in that, Applied to an electronic device, the electronic device including the RGBW sensor; comprising: The acquisition unit is used to acquire the original RGBW image collected by the RGBW sensor; The first processing unit is used to process the W channel of the RGBW original image to obtain the W original image, and to process the RGB channel of the RGBW original image to obtain the RGGB original image. The adjustment unit adjusts the signal-to-noise ratio of the original RGGB image based on the original W image to generate a first image; If the RGBW sensor is currently in photo-taking mode, the adjustment unit is specifically used for: The W original image and the RGGB original image are input into the dual-stream fusion algorithm model; The dual-stream fusion algorithm model during the training phase includes a color restoration sub-model and an image fusion sub-model. The color restoration sub-model is trained by inputting the original RGGB sample image and a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the original RGGB sample image. The image fusion sub-model is trained by inputting the original W sample image and the original RGGB sample image, as well as a second loss function and a third loss function, and is used to fuse the original W sample image and the original RGGB sample image to generate a high-resolution RGB image that restores the details and color information of the original RGGB sample image. The loss value of the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image, so that the parameters of the color restoration sub-model are continuously adjusted by the backpropagation algorithm during training until the loss value of the first loss function is less than a preset threshold. During the inference phase, the image fusion sub-model in the trained dual-stream fusion algorithm model is used to adjust the signal-to-noise ratio of the RGGB original image based on the W original image, and the first image is output.

15. An electronic device, characterized in that, The electronic device includes an RGBW sensor, a memory, and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, causing the electronic device to perform the method according to any one of claims 1-13.

16. A computer-storable medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1-13.

17. A computer program, characterized in that, The computer program includes instructions that are executed by a computing device to implement the method of any one of claims 1-13.

Images