Imaging method based on RGBW sensor and related equipment

Abstract

The invention discloses an imaging method based on an RGBW sensor and related equipment, which are applied to electronic equipment, and the electronic equipment comprises the RGBW sensor. The method comprises the following steps: acquiring an RGBW original image acquired by an RGBW sensor; processing a W channel of the RGBW original image to obtain a W original image, and processing an RGB channel of the RGBW original image to obtain an RGGB original image; and adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image. According to the embodiment of the invention, the signal-to-noise ratio of the image can be improved, so that the quality of the image shot in a low-light environment is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an imaging method and related equipment based on an RGBW sensor. Background Art

[0002] The pixels of traditional color image sensors are usually arranged in the RGGB (red, green, blue) Bayer format and are widely used in cameras of electronic devices, such as mobile phone cameras. Under normal lighting conditions, RGGB sensors can provide good image quality. However, in low-light environments with insufficient light, due to the weak photosensitivity of RGGB sensors, the noise of the captured images increases and the signal-to-noise ratio decreases, resulting in a decrease in image clarity, thus affecting the basic quality of the image. The conventional solution is to increase the exposure time to increase the sensitivity, but this solution may cause the captured images to be blurred when shooting moving scenes. At the hardware level, some mobile phone manufacturers increase the size of image sensors to increase the sensitivity, but due to the size limitations of mobile phone devices, it is difficult to accommodate larger sensors, so the actual effect of this solution is not good.

[0003] In addition, some mobile phone manufacturers have modified the pixel arrangement of image sensors, adopting the new RGBW (red, green, blue, and white) Bayer format. This solution significantly increases the number of white (W) filters, giving RGBW sensors both color sensitivity and enhanced light sensitivity. However, the addition of the white channel affects the color balance of the image generated by fusing RGB and W pixel data. The higher the proportion of W pixels, the greater the impact on image color fidelity and balance. Therefore, finding a fusion strategy that improves the image sensor's light sensitivity while maintaining color balance is an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide an imaging method and related equipment based on an RGBW sensor, which can improve the signal-to-noise ratio of an image to improve the quality of images taken in low-light environments.

[0005] In a first aspect, an embodiment of the present application provides an imaging method based on an RGBW sensor, which is applied to an electronic device, wherein the electronic device includes the RGBW sensor. The method may include: acquiring an RGBW original image captured by the RGBW sensor; processing the W channel of the RGBW original image to obtain a W original image (W data stream), and processing the RGB channels of the RGBW original image to obtain an RGGB original image (RGGB data stream); and adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

[0006] Most existing imaging methods are designed based on red, green, blue, and RGGB sensors. If the camera is taken in a dark environment with insufficient light, the noise of the captured image will increase and the signal-to-noise ratio will decrease due to the weak photosensitivity of the RGGB sensor. Therefore, the clarity of the image will decrease, thereby affecting the basic quality of the image. In response to this technical problem, the embodiment of the present application designs an imaging method based on an RGBW (red, green, blue, and white) sensor with better photosensitivity. At the same time, in order to avoid affecting the color balance of the image generated after fusing RGB and W pixel data, the embodiment of the present application provides a fusion strategy based on an RGBW sensor. Specifically, the electronic device obtains an RGBW raw image (RGBW RAW) collected by the RGBW sensor. Since the RGBW sensor adds white (W) pixels compared to the traditional RGGB sensor, the RGBW sensor can capture more photons to improve the photosensitivity in a dark environment. Furthermore, the W channel in the RGBW raw image is processed to obtain a W raw image (W RAW), and the RGB channels in the RGBW raw image are processed to obtain an RGB raw image (RGBRAW), so as to facilitate subsequent targeted adjustment of the brightness and color information of the image. Furthermore, the signal-to-noise ratio of the RGB original image is adjusted based on the W original image, and the RGB channel data is adjusted using the brightness information of the W original image. This can preserve the subtle textures and details in the RGGB original image while retaining the color information of the RGB channels as much as possible to avoid color loss or distortion, thereby generating a first image with a high signal-to-noise ratio. The image obtained after visualization processing based on the first image has improved clarity and realism under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0007] In one possible implementation, a pixel of the RGBW sensor includes multiple pixel units, each of which is composed of one pixel among R, G, and B pixels and an adjacent W pixel. The pixel arrangement of the RGBW raw image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor.

[0008] In the embodiment of the present application, since the pixels of the RGBW sensor include multiple pixel units consisting of one pixel from the three types of R, G, and B pixels and an adjacent W pixel, and the pixel arrangement of the RGBW raw image is consistent with that of the sensor, the pixels in the RGBW raw image also have the same pixel units as those of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor. The channel value of each pixel in the RGBW raw image corresponds to the pixel of the RGBW sensor, thereby ensuring that the acquired RGBW raw image can retain the information of each pixel, which helps to maintain the details and clarity of the image in the subsequent processing stage, thereby improving the realism and quality of the image.

[0009] In a possible implementation, the total pixel value of the RGBW sensor is determined based on a ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the corresponding total pixel value of the RGBW sensor.

[0010] In the embodiment of the present application, the ratio of the number of RGB pixels to W pixels in each pixel unit of the RGBW sensor can be an integer value, such as 1:1, 1:2, 1:3, 1:8, 1:15, etc., or a fractional value, such as 4:5, 7:9, etc. Accordingly, based on the ratio of the number of RGB pixels to W pixels in each pixel unit of the RGBW sensor, an RGBW sensor with an appropriate resolution (i.e., total pixel value) can be selected to capture an image. Specifically, the smaller the ratio of the number of RGB pixels to W pixels in each pixel unit, the greater the resolution (i.e., total pixel value) of the corresponding RGBW sensor. For example, when the ratio of the number of RGB pixels to W pixels in a pixel unit is 1:8, the resolution of the corresponding RGBW sensor (i.e., the total pixel value) is 108M pixels; when the ratio of the number of RGB pixels to W pixels in a pixel unit is 1:15, the resolution of the corresponding RGBW sensor (i.e., the total pixel value) needs to be approximately 200 million pixels to provide sufficient pixels to capture spatial details, thereby meeting the requirements of different application scenarios for color and brightness information, helping to maintain image details and clarity to improve the overall image quality. Since the W pixels in the RGBW sensor are used to capture brightness information and the RGB pixels are used to capture color information, if the ratio of the number of RGB pixels to the number of W pixels in a pixel unit of the RGBW sensor is smaller, the number of RGB pixels in a pixel unit is smaller than that of W pixels. Therefore, in order to ensure that the RGBW sensor can obtain sufficient color information, it is necessary to increase the total pixel value to increase the number of pixel units, so that the RGBW sensor has a sufficient number of RGB pixels to capture enough color information, thereby improving the color accuracy and restoration of the final image. At the same time, the increase in the total pixel value of the RGBW sensor can also increase the resolution of the RGBW original image, so that the details and clarity of the final image are improved, thereby better improving the image quality shot in low light conditions and enhancing the user's shooting experience.

[0011] In a possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; and acquiring the RGBW raw image captured by the RGBW sensor may include acquiring the long-exposure RGBW raw image captured by the RGBW sensor through long-exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor through short-exposure.

[0012] In the embodiments of the present application, different types of RGBW raw images can be obtained according to the different exposure times of the RGBW sensor. Specifically, the RGBW raw image collected by the RGBW sensor with a long exposure time is a long-exposure RGBW raw image, and the RGBW raw image collected by the regular exposure time is a regular-exposure RGBW raw image, wherein the regular exposure time can be adjusted according to the actual shooting scene, and can be either a shorter exposure time or a longer exposure time, so as to achieve the ideal exposure effect. Under low-light conditions, since the photosensitivity of RGB pixels is weak and the number of received photons is small, resulting in an increase in the noise level, the exposure time of the RGBW sensor can be appropriately extended (that is, a long exposure time) so that the RGB pixels can accumulate more photon signals, thereby reducing the noise generated by the RGB pixels to a certain extent to improve the signal-to-noise ratio, thereby improving the image quality in low-light environments. W pixels have stronger light sensitivity, so under the same lighting conditions, they can receive more photons than RGB pixels. Using a regular exposure time can ensure that the brightness information in the image is properly captured, avoiding overexposure caused by excessive light receiving by the W pixels. It can also avoid motion blur or noise caused by long exposure, thereby more accurately capturing the brightness information of the instantaneous scene and improving image quality.

[0013] In one possible implementation, the processing based on the W channel of the RGBW original image to obtain the W original image may include: calculating an average channel value of the W pixels in each of the pixel units in the conventionally exposed RGBW original image; obtaining the W original image based on the average channel value; and the channel value of each W pixel in the W original image is sequentially equal to the average channel value.

[0014] In an embodiment of the present application, how to obtain a W original image based on the W channel processing of the RGBW original image may specifically include first calculating the average channel value of the W pixels in each pixel unit in the conventionally exposed RGBW original image, and then sequentially using the calculated average channel value as the channel value of each W pixel in the W original image, thereby effectively extracting the brightness information in the RGBW original image to generate the W original image, ensuring that the brightness distribution in the W original image can be consistent with that of the RGBW original image, so that the W original image can be subsequently used to adjust the signal-to-noise ratio of the RGBW original image, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0015] In one possible implementation, the processing of the RGB channels of the RGBW original image to obtain the RGGB original image may include: selecting a target pixel from each pixel unit in the long-exposure RGBW original image, where the target pixel is a pixel of the R, G, and B channels in the pixel unit; generating the RGGB original image based on the channel value of the target pixel; and the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel in sequence.

[0016] In an embodiment of the present application, regarding how to obtain an RGGB raw image based on the RGB channels of an RGBW raw image, the method may specifically include first selecting pixels of the R, G, and B channels (i.e., target pixels) from each pixel unit in the long-exposure RGBW raw image. Furthermore, the channel values ​​of the selected target pixels are sequentially used as the channel values ​​of each pixel in the RGGB raw image (RGGBRAW), so that the color information of the generated RGGB raw image (RGGB RAW) is consistent with that of the RGBW raw image, thereby improving the color accuracy and restoration of subsequent adjustments to the RGGB raw image, improving the overall quality of the captured image, and enhancing the user's shooting experience.

[0017] In one possible implementation, if the RGBW sensor is currently in a photographing mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image may include: performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image; merging the first feature image with the second feature image to generate a third feature image; and reconstructing a denoised and demosaiced RGB image based on the third feature image, where the denoised and demosaiced RGB image is the first image.

[0018] In an embodiment of the present application, when the RGBW sensor is currently in a photo mode, how to adjust the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image may specifically include first performing feature extraction on the W original image and the RGGB original image, respectively, to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image, thereby effectively extracting the brightness features in the W original image and the color features in the RGGB original image, so as to facilitate subsequent targeted adjustment of the image according to different needs. Furthermore, by merging the first feature image (the brightness features of the W original image) and the second feature image (the color features of the RGGB original image), a third feature image containing the brightness features of the W original image and the color features of the RGGB original image is generated. This allows the key information of the RGBW original image in terms of color and brightness to be retained while the noise generated by insufficient light in the RGGB original image is also suppressed to a certain extent, thereby improving the signal-to-noise ratio of the third feature image. Then, based on the third feature image, a denoised and demosaiced RGB image (that is, the first image) is reconstructed to restore the clarity and details of the third feature image, retain the subtle textures and details in the original RGGB image, and avoid the loss of color information of the RGB channels as much as possible to improve color accuracy. The image obtained after subsequent visualization processing based on the first image has improved clarity and realism under low-light conditions, thereby improving the image quality shot in low-light environments and enhancing the user's shooting experience.

[0019] In one possible implementation, if the RGBW sensor is currently in a photo mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: inputting the W original image and the RGGB original image into a dual-stream fusion algorithm model; adjusting the signal-to-noise ratio of the RGGB original image based on the W original image through the dual-stream fusion algorithm model, and outputting the first image.

[0020] In an embodiment of the present application, when the RGBW sensor is currently in a photo mode, the W original image and the RGGB original image can be input into a dual-stream fusion algorithm model. The dual-stream fusion algorithm model can be used to specifically adjust the signal-to-noise ratio of the RGGB original image based on the W original image, thereby outputting a first image with a high signal-to-noise ratio that retains image details and color information. Furthermore, the dual-stream fusion algorithm model is a pre-trained deep learning network structure model (such as a convolutional neural network model). The use of the dual-stream fusion algorithm model can also ensure the stability of the quality of the image output (i.e., the first image) under different conditions, so that the image obtained after visualization processing based on the first image has improved clarity and authenticity under low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0021] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The image fusion sub-model is obtained by inputting an RGGB sample original image and training with a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the image fusion sub-model is obtained by inputting a W sample original image and the RGGB sample original image, and training with a second loss function and a third loss function, and is used to fuse the W sample original image and the RGGB sample original image to generate a high-resolution RGB image that restores the details of the RGGB sample original image and the color information.

[0022] In an embodiment of the present application, the dual-stream fusion algorithm model may include a color restoration sub-model and an image fusion sub-model. Furthermore, the RGGB sample original image is input into the color restoration sub-model, and the color restoration sub-model is trained through the first loss function to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the W sample original image and the RGGB sample original image are input into the image fusion sub-model, and the image fusion sub-model is trained through the second loss function and the third loss function, so that the image fusion sub-model can fuse the input dual-stream images (that is, the W sample original image and the RGGB sample original image) to generate a high-resolution RGB image that restores the details and color information of the RGGB sample original image. Through the embodiment of the present application, the trained color restoration sub-model can be used to preview imaging, so that the user can get a rough image effect before taking a photo, thereby improving the user experience. The trained image fusion sub-model can extract detail information and color information from the W sample original image and the RGGB sample original image. By fusing the W sample original image and the RGGB sample original image, the noise in the single RGGB sample original image is reduced, so that the signal-to-noise ratio of the final high-resolution RGB image is improved, making the details clearer and more realistic while maintaining color balance, thereby improving the image quality shot in low-light environments and enhancing the user's shooting experience.

[0023] In a possible implementation, the RGGB sample original image and the W sample original image are obtained by degrading a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0024] In the embodiment of the present application, since the training of the dual-stream fusion algorithm model requires paired low-resolution (LR) and high-resolution (HR) images as training data pairs, and the low-resolution LR images in the data pair (that is, the RGGB sample original images and the W sample original images) are difficult to obtain, the RGGB original images (that is, the RGGB sample original images) and the W original images (that is, the W sample original images) generated by degradation based on the easily available high-resolution RGB images (that is, the high-resolution RGB sample images) can be used to train the dual-stream fusion algorithm model, thereby reducing the cost of model training, increasing the diversity of the training data set, improving the generalization ability and robustness of the model, and improving the stability of the quality of the images output by the dual-stream fusion algorithm model under different conditions. Furthermore, for the color restoration submodel, the color information difference between the low-resolution RGB image and the high-resolution RGB sample image is evaluated by the first loss function, so that the color restoration submodel can accurately restore the color information of the RGGB sample original image to improve the color restoration of the low-resolution RGB image. For the image fusion submodel, the second loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image, so that the color restoration submodel can accurately restore the color information of the RGGB sample original image to improve the color restoration of the low-resolution RGB image. The detail differences and color information differences between the high-resolution RGB image and the high-resolution RGB sample image ensure that the image fusion sub-model does not lose important details and color information during the image fusion process, so as to improve the clarity and color restoration of the high-resolution RGB image output by the image fusion sub-model. In addition, the color component difference between the low-resolution RGB image output by the color restoration sub-model and the high-resolution RGB image output by the image fusion sub-model is evaluated by the third loss function, so that the final image generated by the image fusion sub-model (that is, the high-resolution RGB image) is consistent with the color of the low-resolution RGB image generated by the color restoration sub-model that can be used for preview, thereby further improving the color restoration of the image fusion sub-model. In summary, the image fusion sub-model trained by the embodiment of the present application is used as a dual-stream fusion algorithm model for image fusion, which can not only improve the clarity and color restoration of the output first image (that is, the high-resolution RGB image), but also improve the stability of the quality of the image output by the dual-stream fusion algorithm model under different conditions, so as to enhance the user's shooting experience.

[0025] In a possible implementation, the electronic device further includes a display screen; the method further includes: performing color correction on the first image to generate a second image in RGB format; and displaying the second image on the display screen.

[0026] In an embodiment of the present application, when the RGBW sensor is currently in a photographing mode, after the first image (the RGB image after denoising and demosaicing) is generated after adjusting the signal-to-noise ratio of the RGGB original image based on the W original image, the first image can be color corrected. For example, the color temperature deviation caused by different light sources in the first image can be eliminated by automatic white balance processing, so that white objects appear true white in the image, and then the color space of the first image is converted to a range that meets specific standards through a color conversion matrix, avoiding the problem of the same image showing color differences on different devices due to the fact that different devices and sensors may use different color space representations, so that the converted image color is more accurate and consistent. Through an embodiment of the present application, it can be ensured that the second image in RGB format after color correction processing can present consistent color performance and color accuracy on the display screens of different devices, thereby enhancing the user's shooting experience.

[0027] In one possible implementation, if the RGBW sensor is currently in a preview or video capture mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: determining the type and distribution pattern of noise in the RGGB original image by comparing channel values ​​in the W original image with corresponding channel values ​​of the RGGB original image; and performing noise reduction processing on the RGGB original image according to the type and distribution pattern of noise to generate an RGGB original image with a high signal-to-noise ratio, where the RGGB original image with a high signal-to-noise ratio is the first image.

[0028] In an embodiment of the present application, when the RGBW sensor is currently in preview or video capture mode, how to adjust the signal-to-noise ratio of the RGGB raw image based on the W raw image to obtain a first image may specifically include firstly comparing the channel values ​​of the W channel image and the RGGB channel image to determine the noise type and distribution pattern in the RGGB raw image, so as to select an appropriate noise reduction method to reduce the noise in the RGGB raw image. Furthermore, based on the determined noise type and distribution pattern, targeted noise reduction processing is performed on the RGGB raw image, such as using a specific noise reduction filter or a lightweight neural network model algorithm to accurately remove the noise in the RGGB raw image while preserving the details and color information of the RGGB raw image as much as possible, thereby obtaining a high signal-to-noise ratio RGGB raw image (i.e., the first image). The image obtained after visualization processing based on the high signal-to-noise ratio RGGB raw image (i.e., the first image) has improved clarity and authenticity in low-light conditions, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience. In the embodiments of the present application, when the RGBW sensor is in preview or video capture mode, an algorithm with lower computational complexity and memory consumption is employed compared to the algorithm used in the photo capture mode. A computing unit within the sensor first performs preliminary noise reduction processing on the W raw image and the RGGB raw image and outputs continuous image frames with a high signal-to-noise ratio, thereby reducing the computational burden on subsequent processors (such as a CPU). This improves the image quality of the electronic device when previewing or capturing videos, while also enhancing the real-time performance and smoothness of the preview or video recording, thereby enhancing the user's shooting experience.

[0029] In a possible implementation, the method further includes: performing denoising and demosaicing processing on the first image to obtain a third image in RGB format.

[0030] In an embodiment of the present application, when the RGBW sensor is currently in preview or video shooting mode, the first image is denoised and demosaiced to restore the details and color information of the first image (that is, the RGGB original image with a high signal-to-noise ratio), so that the clarity and realism of the generated third image in RGB format under low-light conditions are improved, thereby improving the quality of the image generated after subsequent visualization processing based on the third image, thereby enhancing the user's shooting experience.

[0031] In a possible implementation, the denoising and demosaicing processing of the first image to obtain a third image in RGB format includes: inputting the first image into an original image processing network model; and generating the third image after denoising and demosaicing processing through the original image processing network model.

[0032] In an embodiment of the present application, when the RGBW sensor is currently in preview or video shooting mode, the first image (that is, the RGGB original image with a high signal-to-noise ratio) can be input into the original image processing network model, and the original image processing network model can perform denoising and demosaicing on the first image and output a third image in RGB format, thereby further improving the quality and clarity of the third image. At the same time, the original image processing network model can be adaptively adjusted according to different scenes and conditions to adapt to different lighting conditions, environmental changes and other factors, ensuring that the quality of the processed image always remains at a high level, thereby improving the quality of the image generated after subsequent visualization processing based on the third image, thereby enhancing the user's shooting experience.

[0033] In a possible implementation, the electronic device further includes the display screen, and the method further includes: performing color correction on the third image to generate a fourth image in RGB format; and displaying the fourth image on the display screen.

[0034] In an embodiment of the present application, when the RGBW sensor is currently in preview or video shooting mode, after the third image in RGB format is obtained by denoising and demosaicing based on the first image (that is, the RGGB original image with a high signal-to-noise ratio), the third image can be color corrected, and then the fourth image in RGB format after color correction can be displayed on the display screen. For example, automatic white balance processing can be used to eliminate the color temperature deviation caused by different light sources in the third image, so that white objects can appear true white in the image, and then the color space of the third image can be converted to a range that meets specific standards through a color conversion matrix, avoiding the problem that the same image may show color differences on different devices due to different devices and sensors using different color space representations, so that the converted image presents more accurate and consistent colors. Through the embodiment of the present application, it can be ensured that the fourth image in RGB format after color correction can present consistent color performance and color accuracy on the display screens of different devices, thereby improving the user's shooting experience.

[0035] In a second aspect, the present application provides an imaging device based on an RGBW sensor, which is applied to an electronic device. The electronic device includes the RGBW sensor and may include:

[0036] An acquisition unit, configured to acquire an RGBW original image captured by the RGBW sensor;

[0037] A first processing unit is configured to process the W channel of the RGBW original image to obtain a W original image, and to process the RGB channels of the RGBW original image to obtain an RGGB original image;

[0038] An adjustment unit adjusts the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

[0039] Most existing imaging methods are designed based on red, green, blue, and RGGB sensors. If shooting is performed in a dark environment with insufficient light, the noise of the captured image will increase and the signal-to-noise ratio will decrease due to the weak photosensitivity of the RGGB sensor. Therefore, the clarity of the image will decrease, thereby affecting the basic quality of the image. In response to this technical problem, the embodiment of the present application designs an imaging device based on an RGBW (red, green, blue, and white) sensor with better photosensitivity. At the same time, in order to avoid affecting the color balance of the image generated after the fusion of RGB and W pixel data, the embodiment of the present application provides a fusion strategy based on an RGBW sensor. Specifically, the embodiment of the present application obtains the RGBW original image (RGBWRAW) collected by the RGBW sensor through an acquisition unit. Since the RGBW sensor has added white (W) pixels compared to the traditional RGGB sensor, the RGBW sensor can capture more photons to improve the photosensitivity in a dark environment. Furthermore, the first processing unit processes the W channel in the RGBW original image to obtain a W original image (W RAW), and processes the RGB channels in the RGBW original image to obtain an RGB original image (RGB RAW), so as to facilitate subsequent targeted adjustment of the image's brightness and color information. Furthermore, the adjustment unit adjusts the signal-to-noise ratio of the RGB original image based on the W original image, and uses the brightness information of the W original image to adjust the RGB channel data. This can preserve the subtle textures and details in the RGGB original image while preserving the color information of the RGB channels as much as possible to avoid color loss or distortion, thereby generating a first image with a high signal-to-noise ratio. This improves the clarity and realism of the image obtained after visualization processing based on the first image in low-light conditions, thereby improving the quality of images captured in low-light environments and enhancing the user's shooting experience.

[0040] In one possible implementation, a pixel of the RGBW sensor includes multiple pixel units, each of which is composed of one pixel among R, G, and B pixels and an adjacent W pixel. The pixel arrangement of the RGBW raw image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor.

[0041] In a possible implementation, the total pixel value of the RGBW sensor is determined based on a ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the corresponding total pixel value of the RGBW sensor.

[0042] In a possible implementation, the RGBW original image includes a long-exposure RGBW original image and a regular-exposure RGBW original image; and the acquiring unit is specifically configured to:

[0043] The long-exposure RGBW original image acquired by the RGBW sensor through long-time exposure and the regular-exposure RGBW original image acquired by the RGBW sensor through regular-time exposure are acquired.

[0044] In a possible implementation, the first processing unit is specifically configured to:

[0045] Calculating an average channel value of W pixels in each pixel unit in the conventional exposure RGBW original image;

[0046] The W original image is obtained based on the average channel value; the channel value of each W pixel of the W original image is sequentially corresponding to and equal to the average channel value.

[0047] In a possible implementation, the first processing unit is specifically configured to:

[0048] Selecting a target pixel from each pixel unit in the long-exposure RGBW original image, where the target pixel is a pixel of the R, G, and B channels in the pixel unit;

[0049] The RGGB original image is generated based on the channel value of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel in sequence.

[0050] In a possible implementation, if the RGBW sensor is currently in a photographing mode, the adjusting unit is specifically configured to:

[0051] Performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image;

[0052] Merging the first feature image and the second feature image to generate a third feature image;

[0053] A denoised and demosaiced RGB image is reconstructed based on the third feature image, and the denoised and demosaiced RGB image is the first image.

[0054] In a possible implementation, if the RGBW sensor is currently in a photographing mode, the adjusting unit is specifically configured to:

[0055] Inputting the W original image and the RGGB original image into a dual-stream fusion algorithm model;

[0056] The signal-to-noise ratio of the RGGB original image is adjusted based on the W original image through the dual-stream fusion algorithm model, and the first image is output.

[0057] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The image fusion sub-model is obtained by inputting an RGGB sample original image and training with a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the image fusion sub-model is obtained by inputting a W sample original image and the RGGB sample original image, and training with a second loss function and a third loss function, and is used to fuse the W sample original image and the RGGB sample original image to generate a high-resolution RGB image that restores the details of the RGGB sample original image and the color information.

[0058] In a possible implementation, the RGGB sample original image and the W sample original image are obtained by degrading a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0059] In a possible implementation, the electronic device further includes a display screen; and the imaging device further includes:

[0060] a first color correction unit, configured to perform color correction on the first image to generate a second image in RGB format;

[0061] The first display unit is configured to display the second image on the display screen.

[0062] In a possible implementation, if the RGBW sensor is currently in a preview or video capture mode, the adjustment unit is specifically configured to:

[0063] Determine the type and distribution of noise in the RGGB original image by comparing the channel values ​​in the W original image with the corresponding channel values ​​in the RGGB original image;

[0064] According to the type of noise and the distribution law, the RGGB original image is subjected to noise reduction processing to generate an RGGB original image with a high signal-to-noise ratio. The RGGB original image with a high signal-to-noise ratio is the first image.

[0065] In a possible implementation, the imaging device further includes:

[0066] The second processing unit is configured to perform denoising and demosaicing on the first image to obtain a third image in RGB format.

[0067] In a possible implementation, the second processing unit is specifically configured to:

[0068] Inputting the first image into the original image processing network model;

[0069] The third image is generated by performing denoising and demosaicing processing through the original image processing network model.

[0070] In a possible implementation, the electronic device further includes the display screen, and the imaging device further includes:

[0071] a second color correction unit, configured to perform color correction on the third image to generate a fourth image in RGB format;

[0072] The second display unit is configured to display the fourth image on the display screen.

[0073] In a third aspect, an embodiment of the present application provides an electronic device, which may include an RGBW sensor, a memory, and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs any of the methods described in the second aspect above.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by the processor to implement the method described in any one of the first aspect or the second aspect above.

[0075] In a fifth aspect, an embodiment of the present application provides a computer program, which includes instructions and is executed by a computing device to implement the method described in any one of the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0077] Figure 1 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0078] Figure 2 It is a schematic diagram of the software structure of the electronic device 100 provided in an embodiment of the present application.

[0079] Figure 3 This is an example flow chart of an imaging method based on an RGBW sensor provided in an embodiment of the present application.

[0080] Figure 4 Schematic diagram of pixel arrangement of some RGBW sensors provided in embodiments of the present application.

[0081] Figure 5A This is an example flow chart of an imaging method based on an RGBW sensor in a photo mode provided in an embodiment of the present application.

[0082] Figure 5B This is an example flow chart of an imaging method based on an RGBW sensor in another photographing mode provided in an embodiment of the present application.

[0083] Figure 5C This is a schematic diagram of the network structure of a dual-stream fusion algorithm model provided in an embodiment of the present application.

[0084] Figure 6 This is an example flow chart of an imaging method based on an RGBW sensor in a preview or video shooting mode provided in an embodiment of the present application.

[0085] Figure 7 This is a schematic diagram of the photographing process of an imaging method based on an RGB8W sensor provided in an embodiment of the present application.

[0086] Figure 8 This is a schematic diagram of the preview or video shooting process of an imaging method based on an RGB8W sensor provided in an embodiment of the present application.

[0087] Figure 9 This is a schematic diagram of a user interface provided by an embodiment of the present application for a user to use an electronic device to take photos in a low-light environment.

[0088] Figure 10 This is a schematic diagram of a user interface provided by an embodiment of the present application for a user to use an electronic device to shoot a video in a low-light environment.

[0089] Figure 11 This is a schematic structural diagram of an imaging device based on an RGBW sensor provided in an embodiment of the present application.

[0090] Figure 12 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0092] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular expressions "one", "a kind of", "said", "above", "the", and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the embodiments of the present application refers to and includes any or all possible combinations of one or more listed items.

[0093] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0094] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0095] First, some of the terms in this application are explained to help those skilled in the art understand the embodiments of this application.

[0096] (1) Signal-to-Noise Ratio (SNR) refers to the ratio between the useful signal strength and the background noise strength. This concept is widely used in fields such as image processing and is one of the important indicators for measuring image quality. Noise is the unwanted random fluctuations in the image introduced by various factors, such as sensor noise, electronic noise, ambient light noise, etc. A high signal-to-noise ratio means that the information in the image is clearer and easier to recognize. The calculation of the signal-to-noise ratio is usually based on a logarithmic scale and expressed in decibels (dB). In practical applications, improving the signal-to-noise ratio is usually a goal in image processing algorithms and system design.

[0097] (2) RAW image: refers to the original image captured by the image sensor without any processing. The RAW image retains the original data of each pixel obtained from the sensor, including brightness, color and other related information.

[0098] (3) Convolutional Neural Network (CNN): It is a type of deep learning neural network that can effectively capture and learn hierarchical feature representations, which makes it perform well in fields such as image processing and computer vision. Due to the translation invariance and parameter sharing characteristics of CNN for image data, they can effectively process data with spatial hierarchical structures and are mainly used for tasks such as processing and analyzing grid-structured data, such as image and video recognition, and computer vision tasks.

[0099] (4) Feature Map refers to the output of a layer in a convolutional neural network (CNN). When an image is input into a CNN, each convolutional layer uses its filters (or convolution kernels) to process the image (or the feature map of the previous layer) to produce a new feature map. These feature maps are essentially representations of the image after being processed by specific filters.

[0100] (5) Deep Learning Network refers to a highly complex and hierarchical machine learning model built on artificial neural networks. It is a key technology in fields such as artificial intelligence, computer vision, and natural language processing, and is applied in a variety of scenarios such as image recognition, speech processing, and predictive analysis. The construction process of a deep learning network usually includes steps such as multi-layer neuron configuration, weight training, feature learning, and model optimization. It aims to solve complex pattern recognition and data analysis problems by imitating the way the human brain processes information.

[0101] (6) Loss Function (LF): defines the difference between the model’s output for a given input and the actual label.

[0102] (7) Mean Squared Error Loss (MSE Loss) function: It is a loss function used to measure the difference between the model's predicted value and the actual value. Its calculation method is to find the average of the squares of the difference between the predicted value and the actual value.

[0103] (8) Perception Loss (PL) function: This is a deep learning-based loss function commonly used in tasks such as image generation and image super-resolution. The core idea of ​​the perceptual loss function is to use a pre-trained deep convolutional neural network (usually a network pre-trained on large-scale image classification tasks, such as VGG and ResNet) to extract image features and use these features to measure the difference between the generated image and the real image. Compared with pixel-level comparison, feature-based comparison can better capture the high-level semantic information of the image, making the loss function more discriminative and generalizable.

[0104] To facilitate understanding of the embodiments of the present application, the following first introduces an exemplary electronic device provided in the embodiments of the present application.

[0105] See Figure 1 , Figure 1 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 100 is a smart terminal device and can be of various types. The embodiment of the present application does not limit its specific type. For example, the terminal device can be a mobile phone, and can also include a tablet computer, a desktop computer, a desktop computer with a touch-sensitive surface or touch panel, a laptop computer (laptop), a handheld computer, a notebook computer, a smart screen, a wearable device (such as a smart watch, a smart bracelet, etc.), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a car computer, a smart headset, a game console, and can also be an Internet of Things (IOT) device or a smart home device such as a smart water heater, a smart lamp, a smart air conditioner, etc.

[0106] See Figure 1 , combined with Figure 1 The components of the electronic device 100 are described in detail.

[0107] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0108] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0109] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0110] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0111] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0112] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0113] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0114] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0115] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0116] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0117] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0118] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0119] The modem processor may include a modulator and a demodulator. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0120] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0121] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0122] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0123] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0124] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0125] The ISP processes raw image data captured by the image sensor in camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (i.e., the image sensor). The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into an image that can be displayed on the display screen. For example, Auto White Balance (AWB) and Color Correction Matrix (CCM) are two important functional modules in the ISP. The AWB module automatically adjusts the white balance of the image by analyzing different areas of the image, detecting the primary light source in the scene, and adjusting the image's color temperature to eliminate color deviation and make white appear more accurate. The CCM module performs color correction, adjusting the gain of different color channels in the image through matrix transformation to ensure that the image's color representation matches the real scene, avoiding color distortion caused by the sensor, light source, or other factors, and ensuring more accurate color. Furthermore, the ISP can perform algorithmic optimization for image noise and brightness. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located in camera 193.

[0126] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element (i.e., image sensor). The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. In the embodiment of the present application, the photosensitive element (i.e., image sensor) is an RGBW sensor formed by a combination of four pixel arrangements: red (R), green (G), blue (B), and white (W), wherein the RGB pixels are light receiving elements corresponding to wavelengths of red, green, and blue colors, and the W pixels are light receiving elements that receive all RGB wavelengths of light. The RGBW sensor improves the photosensitivity of the image sensor by adding a white channel, can capture more light in low-light environments, improves the brightness and clarity of the image, reduces noise to improve the signal-to-noise ratio of the image, and thus improves the image quality captured in low-light environments. Optionally, the ratio of the number of RGB pixels to the number of W pixels in the RGBW sensor can be 1:1, 1:2, 1:3, 1:8, 1:15, etc., or can be other non-integer ratios, such as 4:5, 7:9, etc., which is not limited in the embodiments of the present application.

[0127] The optical signal is converted into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or other format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than one.

[0128] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0129] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0130] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0131] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0132] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0133] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0134] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0135] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0136] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0137] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0138] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0139] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be located on display screen 194. There are many types of pressure sensors 180A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch location based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, a command to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, a command to create a new short message is executed.

[0140] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100 .

[0141] The air pressure sensor 180C is used to measure air pressure.

[0142] The magnetic sensor 180D includes a Hall sensor.

[0143] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0144] The distance sensor 180F is used to measure distance.

[0145] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0146] The ambient light sensor 180L is used to sense the brightness of the ambient light.

[0147] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0148] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0149] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0150] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0151] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0152] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0153] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0154] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0155] The software system of the electronic device 100 may adopt a layered architecture. Figure 2 It is a schematic diagram of the software structure of the electronic device 100 provided in an embodiment of the present application.

[0156] A layered architecture divides the system into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the system is divided into five layers: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer, from top to bottom.

[0157] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.

[0158] The application framework layer provides an application programming interface (API) and programming framework for the application layer's applications. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, where the camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and programming framework for camera applications.

[0159] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0160] The camera hardware abstraction layer may provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library may include operating codes and data for implementing the shooting method provided in the embodiment of the present application.

[0161] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. These drivers can include camera device drivers, digital signal processor drivers, and image processor drivers.

[0162] The hardware layer is the physical part of a computer system, including various hardware components used to perform calculations and process data. In the field of image processing, the hardware layer usually involves a series of specially designed hardware components, which may include image sensors, image signal processors, digital signal processors, and image processors. An image sensor in the hardware layer corresponds to a camera device in the hardware abstraction layer. For details about image sensors, image signal processors, digital signal processors, and image processors, please refer to the above. Figure 1 The relevant description will not be repeated here.

[0163] Furthermore, the camera device driver is used to drive the camera's image sensor (e.g., RGBW sensor) to capture images and the image signal processor to pre-process the images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0164] In combination with the above software structure, the following exemplarily illustrates the software and hardware workflow when taking photos by using the electronic device 100 in the embodiment of the present application.

[0165] In response to the user's operation of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera device (camera device and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the image sensor (e.g., RGBW sensor) of the corresponding camera and collect image light signals through the image sensor (e.g., RGBW sensor). A camera device in the camera hardware abstraction layer corresponds to an image sensor in the hardware layer.

[0166] Then, the camera's image sensor (such as an RGBW sensor) can transmit the collected image light signal to the image signal processor for preprocessing to obtain an image electrical signal, that is, the original image (such as an RGBW original image), and transmit the original image to the camera hardware abstraction layer through the camera device driver.

[0167] The camera hardware abstraction layer can send the raw image to the camera algorithm library. The camera algorithm library stores the program code that implements the RGBW sensor-based imaging method provided in the embodiments of the present application. Based on the digital signal processor and image processor, the camera algorithm library executes this code, allowing the electronic device 100 to perform some or all of the steps of the RGBW sensor-based imaging method provided in the embodiments of the present application.

[0168] The camera algorithm library sends processed images (e.g., RGB images) to the camera hardware abstraction layer. The camera hardware abstraction layer then displays them. The camera algorithm library also performs various image processing tasks, such as noise reduction, color correction, and contrast adjustment, to improve image quality or perform specific computer vision tasks.

[0169] Optionally, through the interface provided by the camera hardware abstraction layer, the above-mentioned original image or the image processed by the camera algorithm library can be stored in a specific storage unit. Furthermore, in response to the user's operation of opening the gallery application, such as clicking the gallery application icon and then clicking to view the image, the gallery application calls the corresponding interface to access the image data in the storage unit, and then calls the image decoding library to decode the stored image data into an image that can be displayed on the display screen, so as to be displayed in the application.

[0170] It should be noted that, in the embodiment of the present application, when taking photos using the electronic device 100, the original image obtained and the image processed by the camera algorithm library can be a static image obtained based on the image sensor of the camera or a dynamic video frame, and the embodiment of the present application does not limit this.

[0171] In combination with the above description of the hardware structure and software structure of the electronic device 100, the imaging method based on the RGBW sensor provided in the embodiment of the present application is introduced below.

[0172] For example, see Figure 3 , Figure 3 This is a flowchart of an imaging method based on an RGBW sensor provided in an embodiment of the present application. This method can be applied to the above Figure 1 The hardware structure of the electronic equipment and Figure 2 The software structure of the electronic device may include but is not limited to the following steps S301 to S303.

[0173] Step S301: Acquire an RGBW original image captured by an RGBW sensor.

[0174] Specifically, the electronic device obtains an RGBW raw image (RGBW RAW) collected by the RGBW sensor. Since the RGBW sensor has an additional white (W) pixel compared to the traditional RGGB sensor, the RGBW sensor can capture more photons to improve the photosensitivity in dark environments.

[0175] In one possible implementation, a pixel of the RGBW sensor includes multiple pixel units, each of which is composed of one pixel among R, G, and B pixels and an adjacent W pixel. The pixel arrangement of the RGBW raw image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor.

[0176] Specifically, since the pixels of the RGBW sensor include multiple pixel units consisting of one pixel from each of the three R, G, and B pixels and the adjacent W pixel, and the pixel arrangement of the RGBW raw image is consistent with that of the sensor, the pixels in the RGBW raw image also have the same pixel units as the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor. The channel value of each pixel in the RGBW raw image corresponds to the pixel of the RGBW sensor, ensuring that the acquired RGBW raw image can retain the information of each pixel, which helps to maintain the image details and clarity in the subsequent processing stage, thereby improving the realism and quality of the image.

[0177] In a possible implementation, the total pixel value of the RGBW sensor is determined based on a ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the corresponding total pixel value of the RGBW sensor.

[0178] Among them, the ratio of the number of RGB pixels to W pixels in each pixel unit of the RGBW sensor can be an integer ratio, such as 1:1, 1:2, 1:3, 1:8, 1:15, etc., or a fractional ratio, such as 4:5, 7:9, etc., which is not specifically limited in the embodiments of the present application.

[0179] For example, see Figure 4 , Figure 4 Schematic diagram of pixel arrangement of some RGBW sensors provided in embodiments of the present application.

[0180] like Figure 4 As shown in (A), the RGB3W sensor is based on each pixel (that is, RGB pixel) of the traditional Bayer array RGGB sensor, and three white W pixels are added to form a new pixel unit. The ratio of RGB pixels to W pixels in each pixel unit is 1:3. The pixel array of the RGGB original image obtained by the RGB3W sensor is the same as Figure 4 The pixel arrangement shown in (A) is consistent with that in FIG, and the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor. Figure 4 As shown in (B), the RGB8W sensor is based on each pixel (that is, RGB pixel) of the traditional Bayer array RGGB sensor, and 8 white W pixels are added to form a new pixel unit. The ratio of RGB pixels to W pixels in each pixel unit is 1:8. The pixel array of the RGGB original image obtained by the RGB8W sensor is the same as Figure 4The pixel arrangement shown in (B) is consistent with that in FIG, and the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor. Figure 4 As shown in (C), the RGB15W sensor is based on each pixel (that is, RGB pixel) of the traditional Bayer array RGGB sensor, and 15 white W pixels are added to form a new pixel unit. The ratio of the number of RGB pixels to W pixels in each pixel unit is 1:15. The pixel array of the RGGB original image obtained based on the RGB15W sensor is the same as Figure 4 The pixel arrangement shown in (C) is consistent, and the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor. In summary, the RGBW sensor provided in the embodiment of the present application generates a channel value for each R, G, B, and W pixel in the RGBW original image based on the intensity of the corresponding wavelength of light collected by the pixel at the corresponding position of the RGBW sensor, which is converted into a digital signal to represent the intensity or brightness value of each pixel in the corresponding color channel, thereby ensuring that the acquired RGBW original image can retain the information of each pixel, which helps to maintain the details and clarity of the image in the subsequent processing stage, thereby improving the realism and quality of the image.

[0181] It is understandable that the embodiments of the present application only exemplarily introduce several pixel arrangements of RGBW sensors. In other embodiments, the pixels of the RGBW sensor may also present other different arrangements, which are not limited in the embodiments of the present application.

[0182] Since the W pixels in the RGBW sensor are used to capture brightness information and the RGB pixels are used to capture color information, as the number of white (W) filters (pixels) in each pixel unit in the RGBW sensor increases, the number of RGB pixels in a pixel unit is less than that of W pixels, and the photosensitivity of the RGBW sensor increases while the color sensitivity decreases. This can easily affect the color balance of the image generated by the subsequent fusion of RGB and W pixel data. Therefore, the embodiment of the present application can increase the total pixel value of the RGBW sensor to increase the number of pixel units, so that the RGBW sensor has a sufficient number of RGB pixels to capture enough color information. Specifically, according to the ratio of the number of RGB pixels to W pixels in each pixel unit of the RGBW sensor, an RGBW sensor with a suitable resolution (that is, the total pixel value) can be selected to capture images. Furthermore, the smaller the ratio of the number of RGB pixels to W pixels in each pixel unit, the greater the resolution (that is, the total pixel value) of the corresponding RGBW sensor. For example, when the ratio of the number of RGB pixels to W pixels in a pixel unit is 1:8, the resolution of the corresponding RGBW sensor (that is, the total pixel value) is 108M pixels; when the ratio of the number of RGB pixels to W pixels in a pixel unit is 1:15, the resolution of the corresponding RGBW sensor (that is, the total pixel value) needs to be approximately 200 million pixels to provide sufficient pixels to capture spatial details, thereby meeting the requirements of different application scenarios for color and brightness information, and improving the color accuracy and restoration of the final image. At the same time, the increase in the total pixel value of the RGBW sensor can also increase the resolution of the RGBW original image, so that the details and clarity of the final image are improved, thereby better improving the image quality shot in low light conditions, thereby enhancing the user's shooting experience.

[0183] In a possible implementation, the RGBW raw image includes a long-exposure RGBW raw image and a regular-exposure RGBW raw image; and acquiring the RGBW raw image captured by the RGBW sensor may include acquiring the long-exposure RGBW raw image captured by the RGBW sensor through long-exposure, and the regular-exposure RGBW raw image captured by the RGBW sensor through short-exposure.

[0184] Specifically, RGBW raw images can be divided into two types based on the exposure time of the RGBW sensor. RGBW raw images captured by the RGBW sensor using a long exposure time are called long-exposure RGBW raw images, while RGBW raw images captured using a regular exposure time are called regular-exposure RGBW raw images. The regular exposure time can be adjusted based on the actual scene being captured, ranging from shorter to longer exposure times to achieve the desired exposure effect. In low-light conditions, the noise level increases due to the weaker photosensitivity of RGB pixels and the smaller number of received photons. Therefore, by appropriately extending the exposure time of the RGBW sensor (i.e., using a long exposure time), the RGB pixels can accumulate more photon signals, reducing the noise generated by the RGB pixels to a certain extent, improving the signal-to-noise ratio and thus enhancing image quality in low-light environments. W pixels have stronger light sensitivity, so under the same lighting conditions, they can receive more photons than RGB pixels. Using a regular exposure time can ensure that the brightness information in the image is properly captured, avoiding overexposure caused by excessive light receiving by the W pixels. It can also avoid motion blur or noise caused by long exposure, thereby more accurately capturing the brightness information of the instantaneous scene and improving image quality.

[0185] Step S302: Process the W channel of the RGBW original image to obtain a W original image, and process the RGB channels of the RGBW original image to obtain an RGGB original image.

[0186] Specifically, the W channel in the RGBW original image is processed to obtain a W original image (W RAW), and the RGB channels in the RGBW original image are processed to obtain an RGB original image (RGB RAW), so as to facilitate subsequent targeted adjustment of the brightness and color information of the image.

[0187] In one possible implementation, the processing based on the W channel of the RGBW original image to obtain the W original image may include: calculating an average channel value of the W pixels in each of the pixel units in the conventionally exposed RGBW original image; obtaining the W original image based on the average channel value; and the channel value of each W pixel in the W original image is sequentially equal to the average channel value.

[0188] Specifically, regarding how to obtain a W original image based on the W channel processing of the RGBW original image, it can specifically include first calculating the average channel value of the W pixels in each pixel unit in the conventional exposure RGBW original image, and then using the calculated average channel value as the channel value of each W pixel in the W original image in sequence, thereby effectively extracting the brightness information in the RGBW original image to generate the W original image, ensuring that the brightness distribution in the W original image can be consistent with the RGBW original image, so that the W original image can be subsequently used to adjust the signal-to-noise ratio of the RGBW original image, thereby improving the image quality captured in low-light environments and enhancing the user's shooting experience.

[0189] In one possible implementation, the processing of the RGB channels of the RGBW original image to obtain the RGGB original image may include: selecting a target pixel from each pixel unit in the long-exposure RGBW original image, where the target pixel is a pixel of the R, G, and B channels in the pixel unit; generating the RGGB original image based on the channel value of the target pixel; and the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel in sequence.

[0190] Specifically, the method of processing the RGB channels of an RGBW raw image to generate an RGGB raw image may include first selecting pixels of the R, G, and B channels (i.e., target pixels) from each pixel unit in the long-exposure RGBW raw image. Furthermore, the channel values ​​of the selected target pixels are sequentially used as the channel values ​​of each pixel in the RGGB raw image (RGGB RAW). This ensures that the color information of the generated RGGB raw image (RGGB RAW) is consistent with that of the RGBW raw image, thereby improving the color accuracy and restoration of subsequent adjustments to the RGGB raw image, enhancing the overall quality of the captured image, and enhancing the user's photography experience.

[0191] Step S303: adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

[0192] Specifically, the signal-to-noise ratio of the RGB original image is adjusted through the W original image, and the RGB channel data is adjusted using the brightness information of the W original image. This can preserve the subtle textures and details in the RGGB original image while retaining the color information of the RGB channels as much as possible to avoid color loss or distortion, thereby generating a first image with a high signal-to-noise ratio. The image obtained after visualization processing based on the first image has improved clarity and realism under low-light conditions, thereby improving the image quality shot in low-light environments and enhancing the user's shooting experience.

[0193] Optionally, depending on the current shooting mode of the RGBW sensor, different methods can be used to adjust the signal-to-noise ratio of the RGGB raw image based on the W raw image to generate the first image. In one possible implementation, if the RGBW sensor is currently in the shooting mode, a specific implementation of how to adjust the signal-to-noise ratio of the RGGB raw image based on the W raw image in step S303 of the above method to obtain the first image can be found in [1]. Figure 5A , Figure 5A This is an example flow chart of an imaging method based on an RGBW sensor in a photographing mode provided in an embodiment of the present application. This method can be applied to the above Figure 1 The hardware structure of the electronic equipment and Figure 2 In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may further include steps S5A01-step S5A05.

[0194] Step S5A01: performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image.

[0195] Specifically, by performing feature extraction on the W original image and the RGGB original image respectively, a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image are obtained, thereby effectively extracting the brightness features in the W original image and the color features in the RGGB original image, so as to facilitate subsequent targeted adjustment of the image according to different needs.

[0196] Step S5A02: Merge the first feature image and the second feature image to generate a third feature image.

[0197] Specifically, by merging the first feature image (the brightness feature of the W original image) and the second feature image (the color feature of the RGGB original image), a third feature image containing the brightness feature of the W original image and the color feature of the RGGB original image is generated. In this way, the key information of the RGBW original image in terms of color and brightness can be retained, and the noise caused by insufficient light in the RGGB original image will also be suppressed to a certain extent, thereby improving the signal-to-noise ratio of the third feature image.

[0198] Step S5A03: reconstructing a denoised and demosaiced RGB image based on the third feature image.

[0199] Specifically, the denoised and demosaiced RGB image is the first image. Reconstructing the denoised and demosaiced RGB image (also known as the first image) based on the third feature image can restore the clarity and detail of the third feature image, preserving the subtle textures and details in the original RGGB image while minimizing the loss of color information in the RGB channels to improve color accuracy. This allows subsequent visualization processing based on the first image to produce images with improved clarity and fidelity in low-light conditions, thereby improving the quality of images captured in low-light environments and enhancing the user's photography experience.

[0200] Step S5A04: performing color correction on the denoised and demosaiced RGB image to generate a second image in RGB format.

[0201] Step S5A05: Display the second image on the display screen.

[0202] Specifically, after the first image (that is, the RGB image after denoising and demosaicing) is generated after adjusting the signal-to-noise ratio of the RGGB original image based on the W original image (that is, the RGB image after denoising and demosaicing), the first image (that is, the RGB image after denoising and demosaicing) can be color corrected. Exemplarily, the automatic white balance module in the signal processor ISP can be used to perform automatic white balance processing to eliminate the color temperature deviation caused by different light sources in the first image, so that white objects appear to be true white in the image, and then the color conversion matrix module in the ISP is used to convert the color space of the first image to a range that meets specific standards, so as to avoid the problem that the same image may show color differences on different devices due to different devices and sensors using different color space representations, so that the color of the converted second image presented on the display screen is more accurate and consistent. Through the embodiment of the present application, it can be ensured that the second image in RGB format after color correction processing can present consistent color performance and color accuracy on the display screens of different devices, thereby improving the user's shooting experience.

[0203] Optionally, if the RGBW sensor is currently in a photographing mode, a specific implementation of how to adjust the signal-to-noise ratio of the RGGB original image based on the W original image in step S303 of the above method to obtain the first image may also be implemented based on a dual-stream fusion algorithm model.

[0204] See Figure 5B , Figure 5B This is an example flow chart of an imaging method based on an RGBW sensor in another photographing mode provided in an embodiment of the present application. This method can be applied to the above Figure 1 The hardware structure of the electronic equipment and Figure 2In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may further include steps S5B01-step S5B04.

[0205] S5B01: Input the W original image and the RGGB original image into the dual-stream fusion algorithm model.

[0206] S5B02: Using the dual-stream fusion algorithm model, adjust the signal-to-noise ratio of the RGGB original image based on the W original image and output the first image.

[0207] Specifically, when the RGBW sensor is currently in photo mode, the W original image and the RGGB original image can be input into a dual-stream fusion algorithm model. This dual-stream fusion algorithm model then specifically adjusts the signal-to-noise ratio of the RGGB original image based on the W original image, thereby outputting a high signal-to-noise ratio first image that preserves image detail and color information. Furthermore, the dual-stream fusion algorithm model is a pre-trained deep learning network model (e.g., a convolutional neural network model). Using this dual-stream fusion algorithm model can also ensure the stability of the quality of the output image (i.e., the first image) under different conditions. This improves the clarity and realism of the image obtained after visualization processing based on the first image in low-light conditions, thereby improving the quality of images captured in low-light environments and enhancing the user's shooting experience.

[0208] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The image fusion sub-model is obtained by inputting an RGGB sample original image and training with a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the image fusion sub-model is obtained by inputting a W sample original image and the RGGB sample original image, and training with a second loss function and a third loss function, and is used to fuse the W sample original image and the RGGB sample original image to generate a high-resolution RGB image that restores the details of the RGGB sample original image and the color information.

[0209] Specifically, the dual-stream fusion algorithm model may include a color restoration sub-model and an image fusion sub-model. Furthermore, the RGGB sample original image is input into the color restoration sub-model, and the color restoration sub-model is trained through the first loss function to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the W sample original image and the RGGB sample original image are input into the image fusion sub-model, and the image fusion sub-model is trained through the second loss function and the third loss function, so that the image fusion sub-model can fuse the input dual-stream images (that is, the W sample original image and the RGGB sample original image) to generate a high-resolution RGB image that restores the details and color information of the RGGB sample original image. Through the embodiment of the present application, the trained color restoration sub-model can be used to preview imaging, so that the user can get a rough image effect before taking a photo, thereby improving the user experience. The trained image fusion sub-model can extract detail information and color information from the W sample original image and the RGGB sample original image. By fusing the W sample original image and the RGGB sample original image, the noise in the single RGGB sample original image is reduced, so that the signal-to-noise ratio of the final high-resolution RGB image is improved, making the details clearer and more realistic while maintaining color balance, thereby improving the image quality shot in low-light environments and enhancing the user's shooting experience.

[0210] In a possible implementation, the RGGB sample original image and the W sample original image are obtained by degrading a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0211] Specifically, since training the two-stream fusion algorithm model requires paired low-resolution (LR) images and high-resolution (HR) images as training data pairs, and low-resolution LR images (i.e., RGGB sample original images and W sample original images) are difficult to obtain, the two-stream fusion algorithm model can be trained using RGGB original images (i.e., RGGB sample original images) and W original images (i.e., W sample original images) generated by degradation based on readily available high-resolution RGB images (i.e., high-resolution RGB sample images). This reduces the cost of model training, increases the diversity of the training dataset, improves the generalization ability and robustness of the model, and improves the stability of the quality of images output by the two-stream fusion algorithm model under different conditions. Furthermore, for the color restoration sub-model, the color information difference between the low-resolution RGB image and the high-resolution RGB sample image is evaluated through a first loss function, so that the color restoration sub-model can accurately restore the color information of the RGGB sample original image, thereby improving the color restoration of the image and making the final image more realistic and natural. As for the image fusion sub-model, the second loss function is used to evaluate the detail differences and color information differences between the high-resolution RGB image and the high-resolution RGB sample image, ensuring that the image fusion sub-model does not lose important details and color information during the image fusion process, thereby improving the clarity and color reproduction of the high-resolution RGB image output by the model. In addition, the third loss function is used to evaluate the color component differences between the low-resolution RGB image output by the color restoration sub-model and the high-resolution RGB image output by the image fusion sub-model, so that the final image generated by the image fusion sub-model (that is, the high-resolution RGB image) is consistent in color with the low-resolution RGB image generated by the color restoration sub-model and can be used for preview, thereby further improving the color reproduction of the image fusion sub-model.

[0212] For example, see Figure 5C , Figure 5C This is a schematic diagram of the network structure of a dual-stream fusion algorithm model provided in an embodiment of the present application. Figure 5C As shown in the figure, the network structure of the dual-stream fusion algorithm model mainly includes the M1 module (also known as the color restoration sub-model) and the M2 module (also known as the image fusion sub-model). First, based on the high-resolution RGB sample image I GT The RGGB sample original image and the W sample original image are used as the input images for training the dual-stream fusion algorithm model. Then the RGGB sample original image is used as the input image of the M1 module (that is, the color restoration sub-model) to obtain the output I Ir(that is, a low-resolution RGB image), and then constrain the M1 module (that is, the color restoration sub-model) through loss1 (that is, the first loss function), and evaluate I according to the loss value of the calculated loss1 (that is, the first loss function) Ir (that is, low-resolution RGB images) and I GT (that is, the high-resolution RGB sample image), and then continuously adjust the parameters of the M1 module (that is, the color restoration sub-model) through the back-propagation algorithm until the loss value of loss1 (that is, the first loss function) is less than the preset threshold value to ensure that the M1 module (that is, the color restoration sub-model) can accurately restore the color information of the original image of the RGGB sample. Through the embodiment of the present application, the trained M1 module (that is, the color restoration sub-model) can be used to preview imaging, so that the user can get a rough image effect before taking a photo, thereby improving the user experience. It can be understood that in some embodiments, loss1 (that is, the first loss function) can be the mean square error loss function (MSE Loss) and the perceptual loss function (Perception Loss), or it can be other loss functions, and the embodiment of the present application does not make specific limitations on this.

[0213] Furthermore, by freezing the M1 module (ie, the color restoration sub-model), that is, stopping the training of the M1 module (ie, the color restoration sub-model), and calculating the latest output I Ir (that is, the color component I of the low-resolution RGB image) Ir，CrCb Then, the M2 module (also known as the image fusion sub-model) is trained. The RGGB sample original image and the W sample original image are used as input images and input into the M2 module (also known as the image fusion sub-model) for image fusion, and the output I hr (that is, a high-resolution RGB image), and then constrain the M2 module (that is, the image fusion sub-model) through loss2 (that is, the second loss function), and evaluate I according to the loss value of the calculated loss2 (that is, the second loss function). hr (that is, high-resolution RGB images) and I GT (that is, high-resolution RGB sample images), and then continuously adjust the parameters of the M2 module (that is, the image fusion sub-model) through the back-propagation algorithm until the loss value of loss2 (that is, the second loss function) is less than the preset threshold to ensure that the image fusion sub-model does not lose important details and color information during the image fusion process, so as to improve the I output of the M2 module (that is, the image fusion sub-model). hr(that is, high-resolution RGB images) clarity and color reproduction. Finally, calculate the latest output I of the M2 module (that is, the image fusion sub-model) hr (that is, high-resolution RGB image) color component I hr，CrCb After that, the color component I is made Ir，CrCb and color component I hr，CrCb Keep it consistent to ensure that the I generated by the M2 module (that is, the image fusion sub-model) hr (that is, high-resolution RGB image) and the I image generated by the M1 module (that is, color restoration sub-model) that can be used for preview Ir (that is, the low-resolution RGB image) is consistent in color, thereby further improving the color reproduction of the M2 module (that is, the image fusion sub-model). The M2 module (that is, the image fusion sub-model) trained by the embodiment of the present application is used as a dual-stream fusion algorithm model, which can not only improve the clarity and color reproduction of the output high-resolution RGB image (that is, the first image), but also improve the stability of the quality of the image output by the dual-stream fusion algorithm model under different conditions, thereby enhancing the user's shooting experience.

[0214] It can be understood that the neural network model trained in the embodiment of the present application (such as the color restoration sub-model and the image fusion sub-model) can be a neural network of the U-Net architecture or a neural network of other types of architectures; loss2 (that is, the second loss function) and loss3 (that is, the third loss function) can be a perceptual loss function (PerceptionLoss) or other loss functions, and the embodiment of the present application does not make specific limitations on this.

[0215] Step S5B03: Perform color correction on the first image to generate a second image in RGB format.

[0216] Step S5B04: Display the second image on the display screen.

[0217] Specifically, for the detailed description of steps S5B03 to S5B04, please refer to the relevant description of steps S5A04 to S5A05 above, which will not be repeated here.

[0218] Optionally, if the RGBW sensor is currently in preview or video mode, a specific implementation of how to adjust the signal-to-noise ratio of the RGGB original image based on the W original image in step S303 of the above method to obtain the first image can be found in Figure 6 , Figure 6 This is an example flow chart of an imaging method based on an RGBW sensor in a preview or video shooting mode provided in an embodiment of the present application. This method can be applied to the above Figure 1 The hardware structure of the electronic equipment and Figure 2 In the software structure of the electronic device, the method may include the above steps S301-S302, and after the above steps S301-S302, it may further include steps S601-step S605.

[0219] Step S601: Determine the type and distribution of noise in the RGGB original image by comparing the channel values ​​in the W original image with the corresponding channel values ​​in the RGGB original image.

[0220] Specifically, by comparing the channel values ​​of the W channel image and the RGGB channel image, the noise type and distribution law in the RGGB original image are determined, so as to select an appropriate denoising method to reduce the noise in the RGGB original image.

[0221] Step S602: performing noise reduction processing on the RGGB original image according to the type and distribution of noise to generate an RGGB original image with a high signal-to-noise ratio.

[0222] Specifically, the RGGB original image with a high signal-to-noise ratio is the first image. The embodiment of the present application can perform noise reduction processing on the RGGB original image in a targeted manner according to the determined noise type and distribution pattern. For example, a specific noise reduction filter or a lightweight neural network model algorithm is used to accurately remove the noise in the RGGB original image while retaining the details and color information of the RGGB original image as much as possible, thereby obtaining an RGGB original image with a high signal-to-noise ratio (that is, the first image). The image obtained after visualization processing based on the RGGB original image with a high signal-to-noise ratio (that is, the first image) has improved clarity and authenticity under low-light conditions, thereby improving the image quality shot in a low-light environment, thereby enhancing the user's shooting experience. In the embodiments of the present application, when the RGBW sensor is in preview or video capture mode, an algorithm with lower computational complexity and memory consumption is employed compared to the algorithm used in the photo capture mode. A computing unit within the sensor first performs preliminary noise reduction processing on the W raw image and the RGGB raw image and outputs continuous image frames with a high signal-to-noise ratio, thereby reducing the computational burden on subsequent processors (such as a CPU). This improves the image quality of the electronic device when previewing or capturing videos, while also enhancing the real-time performance and smoothness of the preview or video recording, thereby enhancing the user's shooting experience.

[0223] Step S603: performing denoising and demosaicing processing on the first image to obtain a third image in RGB format.

[0224] Furthermore, by denoising and demosaicing the first image, the details and color information of the first image (that is, the original RGGB image with a high signal-to-noise ratio) are restored, so that the clarity and realism of the generated third image in RGB format under low-light conditions are improved, thereby improving the quality of the image generated after subsequent visualization processing based on the third image, thereby enhancing the user's shooting experience.

[0225] In a possible implementation, the denoising and demosaicing processing of the first image to obtain a third image in RGB format includes: inputting the first image into an original image processing network model; and generating the third image after denoising and demosaicing processing through the original image processing network model.

[0226] Specifically, the embodiment of the present application can further improve the quality and clarity of the third image by inputting the first image (that is, the RGGB original image with a high signal-to-noise ratio) into the original image processing network model, and outputting a third image in RGB format after denoising and demosaicing the first image through the original image processing network model. At the same time, the original image processing network model can be adaptively adjusted according to different scenes and conditions to adapt to different lighting conditions, environmental changes and other factors, ensuring that the quality of the processed image always remains at a high level, thereby improving the quality of the image generated after subsequent visualization processing based on the third image, so as to enhance the user's shooting experience.

[0227] Step S604: performing color correction on the third image to generate a fourth image in RGB format.

[0228] Step S605: Display the fourth image on the display screen.

[0229] Specifically, after the third image in RGB format is obtained by denoising and demosaicing based on the first image (that is, the original RGGB image with a high signal-to-noise ratio), the third image can be color corrected, and then the fourth image in RGB format after color correction can be displayed on the display screen. For example, automatic white balance processing can be used to eliminate the color temperature deviation caused by different light sources in the third image, so that white objects can appear true white in the image, and then the color space of the third image can be converted to a range that meets specific standards through a color conversion matrix, avoiding the problem that the same image may show color differences on the display screens of different devices due to different devices and sensors using different color space representations, so that the colors presented by the converted fourth image on the display screens of different devices are more accurate and consistent. Through the embodiment of the present application, it can be ensured that the fourth image in RGB format after color correction can present consistent color performance and color accuracy on the display screens of different devices, thereby improving the user's shooting experience.

[0230] For example, based on the above Figure 4 The RGB8W sensor described in (B) is combined with the above Figure 3 and Figures 5A-5C The method embodiment described above is now described. Next, a flow chart of applying the above method embodiment in the photo taking mode provided by an embodiment of the present application is introduced.

[0231] See Figure 7 , Figure 7 This is a schematic diagram of a photographing process of an imaging method based on an RGB8W sensor provided in an embodiment of the present application. Figure 7 As shown, the RGB8W sensor (RGB8W Senser) may include an RGBW pixel (photosensitive element) array (pixel analog) with a total pixel value (resolution) of 108M, and a calculation unit (Logic) set inside the RGB8W sensor. The arrangement of the RGBW pixel (photosensitive element) array is the same as the above Figure 4 The pixel arrangement of the RGB8W sensor described in (B) is the same as that of the RGB8W sensor. In the photo mode, the RGBW original image under different exposure times (i.e., long exposure time and normal exposure time) is first obtained through the pixel array (pixel analog) of the RGB8W sensor. Specifically, the pixel array of the RGBW original image is the same as that of the RGB8W sensor. Figure 4 The pixel arrangement is consistent with that shown in (B) in FIG, and the resolution of the RGBW original image is equal to the total pixel value of the RGBW sensor, that is, 108M.

[0232] Furthermore, in the calculation unit (Logic) inside the RGB8W sensor, based on the different exposure times of the pixels (photosensitive elements) of the RGB8W sensor when taking pictures (i.e., long exposure time and regular exposure time), an 8-in-1 average calculation (8-in-1 Binning) module and a color selection (Pick Color) module are provided to process the RGBW raw images captured under different exposure times (i.e., long exposure RGBW raw images and regular exposure RGBW raw images) respectively, to generate dual-stream RGB raw RAW images (i.e., W raw images and RGGB raw images). Specifically, the 8-in-1 average calculation (8-in-1 Binning) module calculates the average channel value of the W pixel in each pixel unit in the regular exposure RGBW raw image. The calculated average channel value is then used as the channel value of each W pixel in the W raw image, resulting in the W raw image. The Pick Color module selects the R, G, and B channel pixels (i.e., target pixels) from each pixel unit in the long exposure RGBW raw image. The channel values ​​of the selected target pixels are then used as the channel values ​​of each pixel in the RGGB raw image, resulting in the RGGB raw image. The length and width of each image in the dual-stream RGB raw image (i.e., the W raw image and the RGGB raw image) are respectively 1 / 3 of those of the RGBW raw image (i.e., the long exposure RGBW raw image and the regular exposure RGBW raw image). Therefore, their resolutions are respectively 1 / 9 of those of the RGBW raw images (i.e., the long exposure RGBW raw image and the regular exposure RGBW raw image), i.e., both the W raw image and the RGGB raw image are 12M. It can be understood that the conventional exposure time in the embodiment of the present application can be adjusted according to the actual shooting scene to achieve the ideal exposure effect. It can be a shorter exposure time or a longer exposure time. The embodiment of the present application does not limit this.

[0233] Furthermore, the dual-stream RGB original image (that is, the W original image and the RGGB original image) is transmitted to the dual-stream fusion algorithm (2-Flow Fusion) module in the processor (CPU) through data transmission, and the dual-stream fusion algorithm (2-Flow Fusion) module implements the processing process of adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to generate a denoised and demosaiced RGB image (that is, the first image). Specifically, the dual-stream RGB original image (that is, the W original image and the RGGB original image) can be input into a pre-trained dual-stream fusion algorithm model to perform image fusion and output a first image with a high signal-to-noise ratio that retains image details and color information, so that the clarity and authenticity of the RGB image obtained after visualization processing based on the first image under low-light conditions are improved, thereby improving the image quality shot in low-light environments, so as to enhance the user's shooting experience. Among them, the relevant description of the training method of the dual-stream fusion algorithm (2-Flow Fusion) model can be found in the above Figure 5C The relevant description of the method embodiment in will not be repeated here.

[0234] The image signal processing (ISP) in the processor then performs color correction on the denoised and demosaiced RGB image (i.e., the first image) to generate an RGB image that can be displayed on the display screen (i.e., the second image). Specifically, the denoised and demosaiced RGB image (i.e., the first image) can be color corrected through automatic white balance processing and color conversion matrix (AWB / CCM) so that white objects can appear true white in the RGB image (i.e., the second image), and can have consistent color performance and color accuracy on the displays of different devices, thereby improving the user's shooting experience.

[0235] It is understandable that the above Figure 7 What has been described is only one possible implementation process of the imaging method based on the RGB8W sensor in the photo mode provided in the embodiment of the present application. The specific implementation process may vary based on different types of RGBW sensors, and the embodiment of the present application is not limited to this.

[0236] For example, based on the above Figure 4 The RGB8W sensor described in (B) is combined with the above Figure 3 and Figure 6 The method embodiment described above is now described. Next, a flow chart of applying the above method embodiment in the preview or video shooting mode provided by the embodiment of the present application is introduced.

[0237] See Figure 8 , Figure 8This is a schematic diagram of a preview or video shooting process of an imaging method based on an RGB8W sensor provided in an embodiment of the present application. Figure 8 As shown, the RGB8W sensor (RGB8W Senser) may include an RGBW pixel (photosensitive element) array (pixel analog) with a total pixel value (resolution) of 108M, and a computing unit (Logic) disposed within the RGB8W sensor. Furthermore, within the computing unit (Logic) within the RGB8W sensor, an 8-in-1 average calculation (8-in-1 Binning) module and a color picking (Pick Color) module are provided to process the RGBW raw images captured at different exposure times (i.e., long exposure time and regular exposure time) based on the different exposure times of the RGB8W sensor's pixels (photosensitive elements) when previewing or shooting videos, respectively, to generate W raw images and RGGB raw images. The following are the detailed descriptions of the arrangement of the RGBW pixel (photosensitive element) array and how the 8-in-1 average calculation (8-in-1 Binning) module and the color selection (Pick Color) module process long-exposure RGBW raw images and regular-exposure RGBW raw images in preview or video mode, respectively. Figure 7 The method embodiment described is similar, and its detailed description can be found in Figure 7 The relevant description will not be repeated here.

[0238] Furthermore, the RGB8W sensor (RGB8W Senser) may also include a pre-processing module equipped with a RAW Fusion algorithm. By comparing the channel values ​​of the W channel image and the RGGB channel image, the noise type and distribution pattern in the RGGB original image are determined, and the RGGB original image is subjected to targeted noise reduction processing to generate a high signal-to-noise ratio RGGB original image (also known as the first image). Among them, the RAW Fusion algorithm inside the pre-processing module of the original image adopts an algorithm with lower computational complexity and memory consumption than the algorithm design in the photo mode. The computing unit inside the sensor first performs preliminary noise reduction processing on the W original image and the RGGB original image and outputs continuous high signal-to-noise ratio image frames to reduce the computational burden of the subsequent processor (such as the CPU), thereby improving the image quality of the electronic device when previewing or shooting videos under low light conditions, and also improving the real-time and smoothness of the preview or video recording to enhance the user's shooting experience. It should be noted that the RAW Fusion algorithm in the embodiment of the present application can use a specific noise reduction filter or a lightweight neural network model algorithm, and the embodiment of the present application is not limited to this.

[0239] The high signal-to-noise ratio RGGB original image (that is, the first image) obtained by preprocessing is then transmitted to the image signal processing module ISP in the processor (CPU) through data transmission. Specifically, the high signal-to-noise ratio RGGB original image (that is, the first image) is first denoised and demosaiced through the raw image processing network model (RAW net), and an RGB format image (that is, the third image) is output. The RGB format image (that is, the third image) is further color corrected to generate an RGB image that can be displayed on the display screen (that is, the fourth image). Specifically, the high signal-to-noise ratio RGGB original image (that is, the first image) can be color corrected through automatic white balance processing and color conversion matrix (AWB / CCM), so that white objects can appear true white in the RGB image (that is, the fourth image), and can have consistent color performance and color accuracy on the display screens of different devices, thereby improving the user's shooting experience.

[0240] It is understandable that the above Figure 8 What has been described is only one possible implementation process of the imaging method based on the RGB8W sensor in the preview or video shooting mode provided in the embodiment of the present application. The specific implementation process may vary based on different types of RGBW sensors, and the embodiment of the present application is not limited to this.

[0241] The above detailed description of the method of the embodiment of the present application, it can be understood that, in order to realize the corresponding functions mentioned above, each device includes a hardware structure and / or software module corresponding to the execution of each function. In combination with the units and steps of the examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The following introduces the equipment provided in the embodiment of the present application.

[0242] The above introduces the imaging method based on the RGBW sensor provided in the embodiment of the present application. Next, some user interface schematic diagrams of the user using the electronic device to take pictures after applying the imaging method based on the RGBW sensor provided in the embodiment of the present application are introduced.

[0243] For example, see Figure 9 , Figure 9 This is a user interface diagram of a user using an electronic device to take photos in a low-light environment, provided by an embodiment of the present application. Figure 1 The hardware structure of the electronic equipment and Figure 2The software structure is implemented. Figure 9 As shown, the user turns on the electronic device, so that the display screen of the electronic device displays the desktop of the electronic device, that is, the user interface 91. The user interface 91 may include an icon of at least one application (for example, weather, calendar, email, settings, app store, notes, gallery, phone, short message, browser, and camera 901, etc.). The application icon and the name and position of the corresponding application can be adjusted according to the user's preferences, and this embodiment of the application is not limited to this.

[0244] In the user interface 91, the user can click the camera 901 control. In response to the operation of clicking the camera 901 control, the display screen of the electronic device can display the user interface 92 of the electronic device in the photo mode. The user interface 92 may include a preview display box 902 and a photo control 903. The preview display box 902 can be used to preview the scene image 904 (also known as the fourth image) currently being captured in a low-light environment in real time. The scene image 904 (also known as the fourth image) is obtained by the imaging method based on the RGBW sensor in the preview mode provided by the embodiment of the present application. After obtaining the RGGB original image (also known as the first image) with a high signal-to-noise ratio, the first image is subjected to denoising and demosaicing processing and color correction (such as automatic white balance / color conversion matrix) to generate the fourth image in RGB format that can be displayed on the display screen. In addition, the user interface 92 may also include a camera conversion control 905 and an album control 906. The camera conversion control 905 can be used to switch the camera that captures the image between the front camera and the rear camera. The album control 906 can be used to view photos or videos captured by the electronic device. Furthermore, in response to the user clicking the photo control 903, the electronic device obtains a photo 907 (also known as a second image) of a low-light environment generated according to the imaging method based on the RGBW sensor in the photo mode provided in an embodiment of the present application. The photo 907 (also known as the second image) can be a second image in RGB format that can be displayed on a display screen after color correction (for example, automatic white balance / color conversion matrix) is performed on the RGB image (also known as the first image) after denoising and demosaicing processing output by the dual-stream fusion algorithm model provided in an embodiment of the present application.

[0245] Furthermore, in response to the user clicking on the album control 906, the display screen of the electronic device presents a user interface 93, in which an image display box 908 may be included. The image display box 908 displays the latest photo 907 (that is, the second image) taken by the electronic device. Through the embodiment of the present application, the clarity and authenticity of the photo 907 taken under low-light conditions can be improved, thereby improving the image quality taken under low-light environments and enhancing the user's shooting experience.

[0246] For example, see Figure 10 , Figure 10 This is a user interface diagram of a user using an electronic device to shoot a video in a low-light environment, provided by an embodiment of the present application. Figure 10 As shown in the figure, the user starts the camera interface diagram and related description after turning on the electronic device, please refer to Figure 9 The relevant descriptions of the user interface 91 and the user interface 92 in the figure are not repeated here. The user interface 92 also includes a video mode control 909. Furthermore, in response to the user clicking the video mode control 909 or other operations that switch the electronic device to the video shooting mode, the display screen of the electronic device presents a user interface 94 for the video shooting mode. The user interface 94 may include a preview display box 910 and a shooting control 911, wherein the preview display box 910 can be used to display a scene image 912 (also known as the fourth image) being shot in the current low-light environment, so that the user can preview the shot scene video in real time, wherein the specific description of the scene image 912 (also known as the fourth image) can be found in the above Figure 9 The relevant description about the scene image 904 in the video will not be repeated here. In addition, the user interface 94 may also include an album control 913, which can be used to view the latest video shot by the electronic device. Further, in response to the user clicking the shooting control 911, the electronic device can obtain continuous scene images 912 (that is, the fourth image) to generate the latest video 914, until the electronic device receives the operation of the user clicking the shooting control 911 again and stops acquiring. Further in response to the user clicking the album control 913, the display screen of the electronic device presents a user interface 95, in which the user interface 95 may include a video display box 915, in which the latest video 914 is displayed. Each frame of the video 914 is obtained by the imaging method based on the RGBW sensor in the video shooting mode provided by the embodiment of the present application. After obtaining the RGGB original image with a high signal-to-noise ratio (that is, the first image), the first image is denoised and demosaiced and color corrected (for example, automatic white balance / color conversion matrix), and the fourth image in RGB format that can be displayed on the display is generated. Through the embodiments of the present application, the clarity and realism of the video 914 shot under low-light conditions can be improved, thereby improving the image quality shot in a low-light environment and enhancing the user's shooting experience.

[0247] It should be noted that Figure 9-10 The user interface diagram of the electronic device shown is an exemplary display of an embodiment of the present application. The interface diagram of the electronic device can also be in other styles. The number and specific functions of the controls displayed in the above user interface are merely exemplary descriptions and are not limited to the embodiments of the present application.

[0248] The above details the method provided by the embodiment of the present application. The following describes the device provided by the embodiment of the present application. For example, see Figure 11 , Figure 11 1100 is a schematic structural diagram of an imaging device based on an RGBW sensor provided in an embodiment of the present application. The imaging device 1100 can be applied to an electronic device including the RGBW sensor. The imaging device 1100 can include an acquisition unit 1101, a first processing unit 1102, and an adjustment unit 1103. A detailed description of each unit is as follows:

[0249] An acquisition unit 1101 is configured to acquire an RGBW original image captured by the RGBW sensor;

[0250] A first processing unit 1102 is configured to process the W channel of the RGBW original image to obtain a W original image, and to process the RGB channels of the RGBW original image to obtain an RGGB original image;

[0251] The adjustment unit 1103 adjusts the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

[0252] Most existing imaging methods are designed based on red, green, blue, and RGGB sensors. If shooting is performed in a dark environment with insufficient light, the noise of the captured image increases and the signal-to-noise ratio decreases due to the weak photosensitivity of the RGGB sensor. Therefore, the clarity of the image decreases, thereby affecting the basic quality of the image. In response to this technical problem, an embodiment of the present application designs an imaging device based on an RGBW (red, green, blue, and white) sensor with better photosensitivity. At the same time, in order to avoid affecting the color balance of the image generated after the fusion of RGB and W pixel data, an embodiment of the present application provides a fusion strategy based on an RGBW sensor. Specifically, the embodiment of the present application obtains the RGBW original image (RGBWRAW) collected by the RGBW sensor through the acquisition unit 1101. Since the RGBW sensor adds white (W) pixels compared to the traditional RGGB sensor, the RGBW sensor can capture more photons to improve the photosensitivity performance in a dark environment. Furthermore, the first processing unit 1102 processes the W channel of the RGBW original image to obtain a W original image (W RAW), and processes the RGB channels of the RGBW original image to obtain an RGB original image (RGB RAW), so as to facilitate subsequent targeted adjustment of the brightness and color information of the image. Furthermore, the adjustment unit 1103 adjusts the signal-to-noise ratio of the RGB original image based on the W original image, and uses the brightness information of the W original image to adjust the RGB channel data. This can preserve the subtle texture and details in the RGB original image while preserving the color information of the RGB channels as much as possible to avoid color loss or distortion, thereby generating a first image with a high signal-to-noise ratio. This improves the clarity and realism of the image obtained after visualization processing based on the first image in low-light conditions, thereby improving the quality of images captured in low-light environments and enhancing the user's shooting experience.

[0253] In one possible implementation, a pixel of the RGBW sensor includes multiple pixel units, each of which is composed of one pixel among R, G, and B pixels and an adjacent W pixel. The pixel arrangement of the RGBW raw image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor.

[0254] In a possible implementation, the total pixel value of the RGBW sensor is determined based on a ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the corresponding total pixel value of the RGBW sensor.

[0255] In a possible implementation, the RGBW original image includes a long-exposure RGBW original image and a normal-exposure RGBW original image; and the acquiring unit 1101 is specifically configured to:

[0256] The long-exposure RGBW original image acquired by the RGBW sensor through long-time exposure and the regular-exposure RGBW original image acquired by the RGBW sensor through regular-time exposure are acquired.

[0257] In a possible implementation, the first processing unit 1102 is specifically configured to:

[0258] Calculating an average channel value of W pixels in each pixel unit in the conventional exposure RGBW original image;

[0259] The W original image is obtained based on the average channel value; the channel value of each W pixel of the W original image is sequentially corresponding to and equal to the average channel value.

[0260] In a possible implementation, the first processing unit 1102 is specifically configured to:

[0261] Selecting a target pixel from each pixel unit in the long-exposure RGBW original image, where the target pixel is a pixel of the R, G, and B channels in the pixel unit;

[0262] The RGGB original image is generated based on the channel value of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel in sequence.

[0263] In a possible implementation, if the RGBW sensor is currently in a photographing mode, the adjusting unit 1103 is specifically configured to:

[0264] Performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image;

[0265] Merging the first feature image and the second feature image to generate a third feature image;

[0266] A denoised and demosaiced RGB image is reconstructed based on the third feature image, and the denoised and demosaiced RGB image is the first image.

[0267] In a possible implementation, if the RGBW sensor is currently in a photographing mode, the adjusting unit 1103 is specifically configured to:

[0268] Inputting the W original image and the RGGB original image into a dual-stream fusion algorithm model;

[0269] The signal-to-noise ratio of the RGGB original image is adjusted based on the W original image through the dual-stream fusion algorithm model, and the first image is output.

[0270] In one possible implementation, the dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The image fusion sub-model is obtained by inputting an RGGB sample original image and training with a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the image fusion sub-model is obtained by inputting a W sample original image and the RGGB sample original image, and training with a second loss function and a third loss function, and is used to fuse the W sample original image and the RGGB sample original image to generate a high-resolution RGB image that restores the details of the RGGB sample original image and the color information.

[0271] In a possible implementation, the RGGB sample original image and the W sample original image are obtained by degrading a high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

[0272] In a possible implementation, the electronic device further includes a display screen; and the imaging device further includes:

[0273] a first color correction unit, configured to perform color correction on the first image to generate a second image in RGB format;

[0274] The first display unit is configured to display the second image on the display screen.

[0275] In a possible implementation, if the RGBW sensor is currently in a preview or video capture mode, the adjustment unit 1103 is specifically configured to:

[0276] Determine the type and distribution of noise in the RGGB original image by comparing the channel values ​​in the W original image with the corresponding channel values ​​in the RGGB original image;

[0277] According to the type of noise and the distribution law, the RGGB original image is subjected to noise reduction processing to generate an RGGB original image with a high signal-to-noise ratio. The RGGB original image with a high signal-to-noise ratio is the first image.

[0278] In a possible implementation, the imaging device further includes:

[0279] The second processing unit is configured to perform denoising and demosaicing on the first image to obtain a third image in RGB format.

[0280] In a possible implementation, the second processing unit is specifically configured to:

[0281] Inputting the first image into the original image processing network model;

[0282] The third image is generated by performing denoising and demosaicing processing through the original image processing network model.

[0283] In a possible implementation, the electronic device further includes the display screen, and the imaging device further includes:

[0284] a second color correction unit, configured to perform color correction on the third image to generate a fourth image in RGB format;

[0285] The second display unit is configured to display the fourth image on the display screen.

[0286] It should be noted that the functions of the various units in the imaging device 1100 described in the embodiment of the present application can refer to the relevant description of the above-mentioned method embodiment, and will not be repeated here. It is understandable that the device and method provided in the embodiment of the present application can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the division of the above-mentioned modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0287] For example, see Figure 12 , Figure 12 Schematic diagram of the hardware structure of another electronic device provided in the embodiment of the present application. Figure 12 As shown, the electronic device 1200 includes an RGBW sensor 1201, at least one processor 1202, and a memory 1203. The processor 1202 is coupled to the memory 1203. The coupling in the embodiment of the present application may be a communication connection, an electrical connection, or other forms. In addition, the electronic device 1200 provided in the embodiment of the present application may also include: at least one display screen 1204 ( Figure 12Only one is shown. RGBW sensor 1201, processor 1202, memory 1203, and display screen 1204 may be connected via bus 1205. Specifically, memory 1203 is used to store program instructions. Processor 1202 is used to invoke the program instructions stored in memory 1203, so that electronic device 1200 can execute the steps of the control method provided in the embodiments of the present application. The description of each component and related steps can be found above and will not be repeated here.

[0288] It should be noted that the electronic device 1200 provided in the embodiment of the present application may include more or fewer components than shown in the figure, or combine some components, separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or any combination of software and hardware.

[0289] An embodiment of the present application provides a computer-readable storage medium storing a computer program. The computer program is executed by a processor of the routing device to implement the steps performed by the routing device in the method for restoring a network provided in the embodiment of the present application; or the computer program is executed by a processor of the electronic device to implement the steps performed by the electronic device in the method for restoring a network provided in the embodiment of the present application.

[0290] An embodiment of the present application provides a computer program, which includes instructions. The computer program is executed by the above-mentioned routing device processor to implement the steps performed by the routing device in the method for restoring a network provided in the above-mentioned embodiment of the present application; or the computer program is executed by the above-mentioned electronic device processor to implement the steps performed by the electronic device in the method for restoring a network provided in the above-mentioned embodiment of the present application.

[0291] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0292] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described order of actions, because according to this application, certain steps may be performed in other orders or simultaneously, or certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application. It should also be noted that the features and functions of two or more devices disclosed herein can be concretized in one device. Conversely, the features and functions of a device described above can be further divided into being concretized by multiple devices.

[0293] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0294] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0295] In short, the above description is only an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of this application should be included in the scope of protection of this application.

Claims

1. An imaging method based on a red, green, blue, and white (RGBW) sensor, characterized in that: Applied to an electronic device, the electronic device includes the RGBW sensor, and the method includes: Obtaining an RGBW original image captured by the RGBW sensor; Processing the W channel of the RGBW original image to obtain a W original image, and processing the RGB channels of the RGBW original image to obtain an RGGB original image; The signal-to-noise ratio of the RGGB original image is adjusted based on the W original image to generate a first image.

2. The method according to claim 1, characterized in that The pixels of the RGBW sensor include multiple pixel units, each of which is composed of one pixel among R, G, and B pixels and an adjacent W pixel. The pixel arrangement of the RGBW raw image is consistent with the pixel arrangement of the RGBW sensor, and the resolution of the RGBW raw image is equal to the total pixel value of the RGBW sensor.

3. The method according to claim 1 or 2, characterized in that The total pixel value of the RGBW sensor is determined based on the ratio of the number of RGB pixels to the number of W pixels in the pixel unit; wherein, the smaller the ratio, the larger the total pixel value of the corresponding RGBW sensor.

4. The method according to any one of claims 1 to 3, characterized in that The RGBW original image includes a long-exposure RGBW original image and a regular-exposure RGBW original image; and obtaining the RGBW original image collected by the RGBW sensor includes: The long-exposure RGBW original image acquired by the RGBW sensor through long-time exposure and the regular-exposure RGBW original image acquired by the RGBW sensor through regular-time exposure are acquired.

5. The method according to claim 4, characterized in that The processing based on the W channel of the RGBW original image to obtain the W original image includes: Calculating an average channel value of W pixels in each pixel unit in the conventional exposure RGBW original image; The W original image is obtained based on the average channel value; the channel value of each W pixel of the W original image is sequentially corresponding to and equal to the average channel value.

6. The method according to claim 4 or 5, characterized in that The processing based on the RGB channels of the RGBW original image to obtain the RGGB original image includes: Selecting a target pixel from each pixel unit in the long-exposure RGBW original image, where the target pixel is a pixel of the R, G, and B channels in the pixel unit; The RGGB original image is generated based on the channel value of the target pixel; the channel value of each pixel in the RGGB original image corresponds to and is equal to the channel value of the target pixel in sequence.

7. The method according to any one of claims 1 to 6, characterized in that If the RGBW sensor is currently in a photographing mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: Performing feature extraction on the W original image and the RGGB original image respectively to obtain a first feature image corresponding to the W original image and a second feature image corresponding to the RGGB original image; Merging the first feature image and the second feature image to generate a third feature image; A denoised and demosaiced RGB image is reconstructed based on the third feature image, and the denoised and demosaiced RGB image is the first image.

8. The method according to any one of claims 1 to 7, characterized in that If the RGBW sensor is currently in a photographing mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: Inputting the W original image and the RGGB original image into a dual-stream fusion algorithm model; The signal-to-noise ratio of the RGGB original image is adjusted based on the W original image through the dual-stream fusion algorithm model, and the first image is output.

9. The method according to claim 8, characterized in that The dual-stream fusion algorithm model includes a color restoration sub-model and an image fusion sub-model. The image fusion sub-model is obtained by inputting the RGGB sample original image and training with a first loss function, and is used to generate a low-resolution RGB image that restores the color information of the RGGB sample original image; the image fusion sub-model is obtained by inputting the W sample original image and the RGGB sample original image, and training with a second loss function and a third loss function, and is used to fuse the W sample original image and the RGGB sample original image to generate a high-resolution RGB image that restores the details of the RGGB sample original image and the color information.

10. The method according to claim 9, characterized in that The RGGB sample original image and the W sample original image are obtained by degrading the high-resolution RGB sample image; the first loss function is used to evaluate the color information difference between the low-resolution RGB image and the high-resolution RGB sample image; the second loss function is used to evaluate the detail difference and color information difference between the high-resolution RGB image and the high-resolution RGB sample image; the third loss function is used to evaluate the color component difference between the low-resolution RGB image and the high-resolution RGB image.

11. The method according to any one of claims 1 to 10, characterized in that The electronic device further includes a display screen; and the method further includes: Performing color correction on the first image to generate a second image in RGB format; The second image is displayed on the display screen.

12. The method according to any one of claims 1 to 11, characterized in that If the RGBW sensor is currently in a preview or video capture mode, adjusting the signal-to-noise ratio of the RGGB original image based on the W original image to obtain a first image includes: Determine the type and distribution of noise in the RGGB original image by comparing the channel values ​​in the W original image with the corresponding channel values ​​in the RGGB original image; According to the type of noise and the distribution law, the RGGB original image is subjected to noise reduction processing to generate an RGGB original image with a high signal-to-noise ratio. The RGGB original image with a high signal-to-noise ratio is the first image.

13. The method according to claim 12, characterized in that The method further comprises: De-noising and demosaicing are performed on the first image to obtain a third image in RGB format.

14. The method according to claim 13, characterized in that The performing denoising and demosaicing on the first image to obtain a third image in RGB format includes: Inputting the first image into the original image processing network model; The third image is generated by performing denoising and demosaicing processing through the original image processing network model.

15. The method according to claim 13 or 14, characterized in that The electronic device further includes the display screen, and the method further includes: performing color correction on the third image to generate a fourth image in RGB format; The fourth image is displayed on the display screen.

16. An imaging device based on an RGBW sensor, characterized in that: Applied to electronic equipment, the electronic equipment includes the RGBW sensor; including: An acquisition unit, configured to acquire an RGBW original image captured by the RGBW sensor; A first processing unit is configured to process the W channel of the RGBW original image to obtain a W original image, and to process the RGB channels of the RGBW original image to obtain an RGGB original image; An adjustment unit adjusts the signal-to-noise ratio of the RGGB original image based on the W original image to generate a first image.

17. An electronic device, characterized in that: The electronic device includes an RGBW sensor, a memory, and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 15.

18. A computer storable medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 15.

19. A computer program, characterized in that The computer program comprises instructions, and the computer program is executed by a computing device to implement the method according to any one of claims 1 to 15.

Images