Image processing method, image processing device and chip

Abstract

The embodiment of the present application provides an image processing method, an image processing device and a chip for processing an RGBW image acquired by an image sensor. The method includes: sampling the W channel in the RGBW image to obtain a first sampling map, and sampling each color channel in the RGBW image to obtain a second sampling map corresponding to each color channel; interpolating the first sampling map into a full-resolution full-color image; using the full-color image for guidance, performing a first interpolation process on the second sampling map corresponding to each color channel to obtain a first interpolation map corresponding to each color channel; using the full-color image for guidance, performing a second interpolation process on the first interpolation map corresponding to each color channel to obtain a full-resolution color image corresponding to each color channel; merging the full-resolution color images corresponding to each color channel to generate an RGB image. Based on the above scheme, the image quality of the final RGB image can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of images, and in particular to an image processing method, an image processing device, and a chip. Background Art

[0002] With the development of camera technology for electronic devices such as mobile phones, the requirements for dark light scenes are getting higher and higher, so RGBWCFA with a larger amount of light intake compared to the Bayer type color filter array (Color Filter Array, CFA) has emerged. Since the W channel has a higher signal-to-noise ratio than the color channel (specifically, the R channel, the G channel, and the B channel), it can bring in more light intake, so the RGBWCFA has higher sensitivity, especially in dark light scenes. It has better imaging performance. However, the introduction of the W channel makes the distribution of R pixels, G pixels, and B pixels more sparse, resulting in the degradation of the image quality of the RGB image obtained after the RGBWCFA image is demosaiced. Summary of the invention

[0003] In view of this, embodiments of the present application provide an image processing method, an image processing device, and a chip to at least partially solve the above problems.

[0004] According to a first aspect of an embodiment of the present application, an image processing method is provided for processing an RGBW image acquired by an image sensor, wherein a W channel and R, G, and B color channels in the RGBW image are arranged alternately in a horizontal direction and a vertical direction, and the method comprises:

[0005] Sampling the W channel in the RGBW image to obtain a first sampling image, and sampling each color channel in the RGBW image to obtain a second sampling image corresponding to each color channel;

[0006] interpolating the first sampling image into a full-resolution full-color image;

[0007] Using the full-color image as a guide, performing first interpolation processing on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel;

[0008] Using the full-color image as a guide, performing second interpolation processing on the first interpolation images corresponding to the respective color channels, to obtain full-resolution color images corresponding to the respective color channels;

[0009] The full-resolution color images corresponding to the color channels are merged to generate an RGB image.

[0010] According to a second aspect of an embodiment of the present application, an image processing device is provided for processing an RGBW image acquired by an image sensor, wherein color pixels of a W channel and color pixels of each of R, G, and B color channels in the RGBW image are arranged alternately in the horizontal direction and the vertical direction, and the device comprises:

[0011] A sampling module, configured to sample the W channel in the RGBW image to obtain a first sampling map, and to sample each color channel in the RGBW image to obtain a second sampling map corresponding to each color channel;

[0012] W interpolation module, used for interpolating the first sampling image into a full-resolution full-color image;

[0013] A first interpolation module is used to use the full-color image as a guide to perform a first interpolation process on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel;

[0014] A second interpolation module is used to use the full-color image as a guide to perform a second interpolation process on the first interpolation images corresponding to the color channels to obtain a full-resolution color image corresponding to each color channel;

[0015] The generating module is used to generate an RGB image by performing a merging process based on the full-resolution color images corresponding to the color channels.

[0016] According to a third aspect of an embodiment of the present application, a chip is provided, comprising: a processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the image processing method as described in the first aspect.

[0017] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the image processing method as described in the first aspect.

[0018] For the above scheme, based on the characteristics that W pixels have the largest number and are evenly distributed, the W channel in the RGBW image is first interpolated to obtain more texture detail information in the RGBW image, and then the full-color image obtained by interpolating the W channel is used as a guide to interpolate each color channel in the RGBW image. The full-resolution color image corresponding to each color channel obtained by interpolation can have a higher color resolution, thereby improving the image quality of the final RGB image. In addition, the second interpolation process can correct the error introduced in the first interpolation process, better restore the color information of each color channel, and further improve the image quality of the final RGB image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 A schematic diagram of an RGBWCFA provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the architecture of an image processing process provided in an embodiment of the present application;

[0023] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a second sampling graph, a guide graph, and a color difference graph corresponding to an exemplary G channel provided in an embodiment of the present application;

[0025] Figure 6a and Figure 6b The embodiments of the present application are respectively Figure 5 Schematic diagram of corresponding exemplary interpolation directions and interpolation methods;

[0026] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of an image processing method provided in an embodiment of the present application;

[0028] Figure 9a and Figure 9b The embodiments of the present application are respectively Figure 8 A corresponding schematic diagram of an exemplary interpolation process;

[0029] Fig.10 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0030] Fig.11 The embodiments of the present application provide Fig.10 a corresponding schematic diagram of an exemplary interpolation method;

[0031] Fig.12 The embodiments of the present application provide Fig.10 a corresponding schematic diagram of an exemplary interpolation method;

[0032] Fig.13 A flowchart of an image processing method provided in an embodiment of the present application;

[0033] Fig.14 A schematic diagram of an exemplary first interpolation graph provided in an embodiment of the present application;

[0034] Fig.15 A flowchart of an image processing method provided in an embodiment of the present application;

[0035] Fig.16 A flowchart of an image processing method provided in an embodiment of the present application;

[0036] Fig.17 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0037] Fig.18 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0038] Fig.19 A schematic diagram of the structure of a noise suppression unit provided in an embodiment of the present application;

[0039] Fig. 20 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0041] The image processing method provided in the embodiment of the present application is suitable for processing RGBW images collected by an image sensor. In order to facilitate understanding of the technical solution provided by the present application, the image sensor (or pixel array) is first briefly described below.

[0042] The structure of each pixel in the pixel array is similar. Generally, the structure of each pixel includes a microlens, a filter, and a photosensitive element. The microlens is located above the filter, and the filter is located above the photosensitive element. The light returned by the imaging object is focused by the microlens and then emitted, and then filtered by the filter and then emitted into the photosensitive element, and the photosensitive element converts the light signal into an electrical signal. Depending on the type of light that can be transmitted by different filters, the pixel array may include red pixels (hereinafter referred to as R pixels), blue pixels (hereinafter referred to as B pixels), and green pixels (hereinafter referred to as G pixels). For example, an R pixel refers to only red light that is filtered by a filter and input into the photosensitive element, which is converted into an electrical signal by the photosensitive element. The principles of B pixels and G pixels are similar to those of R pixels, and will not be repeated here.

[0043] Since each pixel in the pixel array can only convert one type of light signal into an electrical signal, it is necessary to perform interpolation operations with the light signals collected by other types of pixels around it to restore the image color of the area collected by the pixel. This process is called demosaicing, which is usually completed in the image processor at the back end of the image sensor or the processor of the electronic device containing the image sensor (hereinafter collectively referred to as the processor).

[0044] In order to capture color images, it is necessary to set filters with specific color arrangements in the pixel array, which can also be called a color filter array (CFA). At present, the common pixel array is mainly based on the RGB array in the form of Bayer CFA, which is characterized by its minimum repeating unit being a 2×2 four-pixel array, including 1 R pixel, 1 B pixel and 2 G pixels, where the R pixel is diagonally adjacent to the B pixel, and the 2 G pixels are diagonally adjacent. Each pixel obtains one color information in RGB, and the RGB information of each pixel is restored by using a demosaicing algorithm.

[0045] This Bayer CFA makes each pixel sensitive to one type of light signal, reducing the amount of light reaching each pixel, thereby reducing the light sensitivity of each pixel, resulting in poor image quality in dark light scenes. With the development of camera technology for electronic devices such as mobile phones, the requirements for dark light scenes are getting higher and higher, so a pixel array based on the RGBWCFA format has emerged, which introduces full-color pixels (hereinafter referred to as W pixels) that allow all wavelengths of light into the RGB array. Since W pixels have a wider spectral response and can receive more photons, RGBWCFA has a higher amount of light input and thus a higher sensitivity than Bayer CFA, which improves the imaging performance in dark light scenes to a certain extent.

[0046] In order to further improve the imaging performance in dark light scenes, a related art has been proposed. Figure 1The RGBWCFA shown includes 8×8 minimum repeating units, each of which includes: a 4×4 square BW cluster formed by alternating 8 blue (B) filter units and 8 full-color (W) filter units, a 4×4 square RW cluster formed by alternating 8 red filter units and 8 full-color filter units, a first 4×4 square GW cluster formed by alternating 8 green (G) filter units and 8 full-color filter units, and a second 4×4 square GW cluster formed by alternating 8 G filter units and 8 W filter units, wherein the 4×4 square RW cluster is diagonally adjacent to the 4×4 square BW cluster, the first 4×4 square GW cluster is horizontally adjacent to the 4×4 square BW cluster, and the second 4×4 square GW cluster is diagonally adjacent to the first 4×4 square GW cluster.

[0047] In the image sensor, since the photosensitive units in the photosensitive unit array correspond to the filter units in the filter unit array one by one, the photosensitive units corresponding to the R filter units in the photosensitive unit array output R pixels, the photosensitive units corresponding to the G filter units output G pixels, the photosensitive units corresponding to the B filter units output B pixels, and the photosensitive units corresponding to the W filter units output W pixels, that is, the image sensor can be used to obtain the same Figure 1 The RGBWCFA image (hereinafter referred to as RGBW image) corresponding to the RGBWCFA shown.

[0048] The RGBW image contains four channels, namely, a full-color W channel, a red R channel, a green G channel, and a blue B channel. The W channel is composed of full-color W pixels, the R channel is composed of red R pixels, the G channel is composed of green G pixels, and the B channel is composed of blue B pixels. In the RGBW image, W pixels are arranged alternately with R pixels, G pixels, and B pixels in the horizontal and vertical directions. For example, Figure 1 Corresponding to the RGBWCFA shown, the RGBW image includes an 8×8 minimum repeating unit, each of which includes: a 4×4 square BW cluster formed by alternating 8 B pixels and 8 W pixels, a 4×4 square RW cluster formed by alternating 8 R pixels and 8 W pixels, a first 4×4 square GW cluster formed by alternating 8 G pixels and 8 W pixels, and a second 4×4 square GW cluster formed by alternating 8 G pixels and 8 W pixels, the 4×4 square RW cluster is diagonally adjacent to the 4×4 square BW cluster, the first 4×4 square GW cluster is horizontally adjacent to the 4×4 square BW cluster, and the second 4×4 square GW cluster is diagonally adjacent to the first 4×4 square GW cluster.

[0049] Since 50% of W pixels are introduced into the RGBW image, the W pixels and the color pixels of each color channel (e.g., R pixels, G pixels, and B pixels) are arranged alternately in the horizontal and vertical directions, making the distribution of R pixels, G pixels, and B pixels more sparse. Compared with the Bayer-type RGB image, the RGBW image is not symmetrical and has a larger period (8×8), which makes the mainstream de-mosaicing algorithm based on symmetrical RGB images with a smaller period (2×2) inapplicable.

[0050] In addition, the prior art mentions a de-mosaicing algorithm for RGBW images, in which a low-resolution partial color image is generated based on the color pixels in the RGBW image, a high-resolution full-color image is first interpolated based on the W pixels in the image, and then a low-resolution full-color image with the same resolution as the low-resolution partial color image is generated based on the high-resolution full-color image, and then a low-resolution color difference is generated based on the low-resolution partial color image and the low-resolution full-color image, and then a high-resolution color difference is generated based on the low-resolution color difference to estimate the final color. Since some color detail information has been lost in the process of generating low-resolution color information, this part of detail information cannot be restored in subsequent processing. Moreover, due to Figure 1 The period of the minimum repeating unit in the RGBW image shown is larger, and the low-resolution color information generated thereby will lose more serious information, which will result in worse quality of the final restored RGB image.

[0051] To this end, an embodiment of the present application provides an image processing method, which utilizes the advantages of uniform distribution and highest sampling rate of the W channel in the RGBW image, and guides the color interpolation process of the RGB three channels based on the effective information of the W channel, thereby improving the image quality of the final restored RGB image.

[0052] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0053] Figure 2 The following is a flow chart of an image processing method provided in an embodiment of the present application. The image processing method is used to process an RGBW image acquired by an image sensor. The image sensor may be Figure 1 The image processing method can be performed by an image processor at the back end of the image sensor or other processors of an electronic device including the image sensor (hereinafter collectively referred to as processors).

[0054] Reference Figure 2 , the image processing method comprises:

[0055] S201, sampling the W channel in the RGBW image to obtain a first sampling image, and sampling each color channel in the RGBW image respectively to obtain a second sampling image corresponding to each color channel;

[0056] In this embodiment, the image sensor may send the collected RGBW image to the processor for execution. Figure 3 The architecture of the image processing flow of the processor executing the embodiment of the present application is shown. Specifically, the processor samples the position of the W pixel in the acquired RGBW image, that is, samples the W channel in the RGBW image, and obtains the first sampling image W1 corresponding to the W channel in the RGBW image. In the first sampling image W1, the value of the W channel is retained, and the values ​​of the R, G, and B color channels are removed.

[0057] In addition, the processor also samples the positions of the R pixels, B pixels, and G pixels in the acquired RGBW image, that is, samples the R channel, B channel, and G channel in the RGBW image to obtain the second sampling map R1 corresponding to the R channel, the second sampling map B1 corresponding to the B channel, and the second sampling map G1 corresponding to the G channel in the RGBW image, collectively referred to as the second sampling map corresponding to each color channel. The value of the corresponding color channel is retained in the second sampling map corresponding to each color channel. In other words, the value of the R channel is retained in the second sampling map corresponding to the R channel. The value of the B channel is retained in the second sampling map corresponding to the B channel. The value of the G channel is retained in the second sampling map corresponding to the G channel. In order to simplify the view, Figure 3 The second sampling graph R1 corresponding to the R channel, the second sampling graph B1 corresponding to the B channel, and the second sampling graph G1 corresponding to the G channel are combined into a graph I1 for display.

[0058] S202, interpolating the first sampling image into a full-resolution full-color image;

[0059] Specifically, the full resolution is the same as the resolution of the image captured by the image sensor (i.e., the RGBW image). Interpolating the first sampling image into a full-color image with full resolution refers to performing W channel interpolation in the first sampling image, that is, inserting the W channel value at the position corresponding to the R channel, G channel, and B channel in the RGBW image in the first sampling image to obtain a full-color image W2 with full resolution. The specific interpolation method applicable to S202 may be a bilinear interpolation method, a nearest neighbor interpolation method, a cubic interpolation method, etc., which is not limited in the embodiment of the present application.

[0060] In a specific implementation, interpolating the first sampling image into a full-color image may include:

[0061] Step A: determining a target position to be interpolated in the first sampling graph.

[0062] The target interpolation position corresponds to the position of the R channel, the G channel and the B channel in the RGBW image;

[0063] Step B, determining the horizontal gradient and vertical gradient of the target position to be interpolated;

[0064] Step C, calculating three predicted values ​​of the target interpolation position in the horizontal direction, the vertical direction and the homogeneous area;

[0065] Step D: Based on the horizontal gradient and the vertical gradient of the target position to be interpolated, one of the three predicted values ​​is determined as the W channel value of the target position to be interpolated.

[0066] The following takes the target interpolation position as C and the coordinates of C as (i, j) as an example to explain the above steps B to D in detail.

[0067] In step B, the horizontal gradient and vertical gradient of the target interpolation position C are determined according to formula (1).

[0068] gradH=|W i，j-1 -W i，j+1 |+|W i，j-3 -W i，j-1 |+|W i，j+1 -W i，j+3 |

[0069] gradV=|W i-1，j -W i+1，j |+|W i-3，j -W i-1，j |+|W i+1，j -W i+3，j | (1)

[0070] Among them, gradH and gradV represent the horizontal gradient and vertical gradient of the target interpolation position C, respectively, and W i，j-3 , W i，j-1 , W i，j+1 and W i，j+3 They represent the W channel values ​​of the four adjacent W channel positions (also called W pixels) of the target interpolation position C in the horizontal direction, W i-3，j , W i-1，j , W i+1，j and W i+3，j They respectively represent the W channel values ​​of the four adjacent W channel positions of the target interpolation position C in the vertical direction.

[0071] In step C, three predicted values ​​of the target interpolation position C in the horizontal direction, vertical direction and homogeneous area are determined according to formula (2).

[0072] H_pred=(W i，j-1 +W i，j+1 ) / 2

[0073] V_pred=(W i-1，j +W i+1，j ) / 2

[0074] N-pred=(H pred +V_pred) / 2 (2)

[0075] Among them, H_pred, V pred and N_pred are the three predicted values ​​of the target interpolation position C in the horizontal direction, vertical direction and homogeneous area, respectively. i，j-1 , W i，j+1 They represent the W channel values ​​of the two adjacent W channel positions of the target interpolation position C in the horizontal direction, W i-1，j , W i+1，j They respectively represent the W channel values ​​of two adjacent W channel positions of the target interpolation position C in the vertical direction.

[0076] In step D, based on the horizontal gradient and the vertical gradient of the target position to be interpolated, one of the three predicted values ​​is determined as the W channel value of the target position to be interpolated.

[0077] Specifically, a first threshold THRESH1 and a second threshold THRESH2 may be set, wherein the first threshold THRESH1 and the second threshold THRESH2 may be selected based on actual needs, experimental data or experience.

[0078] If gradH<THRESH1 and gradV<THRESH1, the W channel value of the target position C to be interpolated is determined as N_pred; if (gradH-gradV)>THRESH2, the W channel value of the target position C to be interpolated is determined as V_pred; if (gradV-gradH)>THRESH2, the W channel value of the target position C to be interpolated is determined as H_pred; otherwise, the W channel value of the target position C to be interpolated is determined as N_pred.

[0079] In this way, the first sampling graph is traversed, and the W channel value is estimated and inserted at each position corresponding to the R channel, the G channel, and the B channel in the RGBW image in the first sampling graph, thereby obtaining a full-color image with full resolution. It should be understood that the interpolation algorithm here is only an example, and other interpolation algorithms can be selected as needed in actual applications, and this embodiment does not limit this.

[0080] In this embodiment, since the sampling rate of W pixels is the highest and the distribution of W pixels is relatively uniform in the RGBW image, the W channel of the RGBW image is first interpolated in the RGBW image, so that better texture detail information in the RGBW image can be obtained, which is beneficial to the subsequent interpolation of other color channels.

[0081] S203 , using the full-color image as a guide, perform a first interpolation process on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel.

[0082] Using the full-color image as a guide, the first interpolation processing is performed on the second sampling images corresponding to each color channel. It can be understood that the first interpolation processing is performed on the second sampling images corresponding to each color channel using the W channel information provided by the full-color image. Since the full-color image contains good texture detail information in the RGBW image, when the first interpolation processing is performed on the second sampling images corresponding to each color channel using the W channel information provided by the full-color image, the first interpolation images corresponding to each color channel obtained by interpolation have higher color resolution and can better reflect the objective color of the photographed object.

[0083] In one implementation of the present application, step S203 includes: taking each color channel as a target color channel, using the full-color image for guidance, interpolating the first interpolation position in the second sampling map corresponding to the target color channel, and obtaining a first interpolation map corresponding to the target color channel. The first interpolation position is a position in the R channel, the G channel, and the B channel of the RGBW image except the target color channel.

[0084] Specifically, taking the G channel as the target color channel, the full-color image is used for guidance, and the G channel value is estimated and inserted at the position corresponding to the R channel and the B channel in the RGBW image in the second sampling map corresponding to the G channel, so as to obtain the first interpolation map corresponding to the G channel. In the first interpolation map corresponding to the G channel, the positions corresponding to the R channel, the B channel and the G channel in the RGBW image all have G channel values, while the position corresponding to the W channel in the RGBW image has not yet interpolated the G channel value. Taking the R channel as the target color channel, the full-color image is used for guidance, and the R channel value is estimated and inserted at the position corresponding to the B channel and the G channel in the RGBW image in the second sampling map corresponding to the R channel, so as to obtain the first interpolation map corresponding to the R channel. In the first interpolation map corresponding to the R channel, the positions corresponding to the R channel, the B channel and the G channel in the RGBW image all have R channel values, while the position corresponding to the W channel in the RGBW image has not yet interpolated the R channel value. Taking the B channel as the target color channel, the full-color image is used for guidance, and the B channel value is estimated and inserted at the position corresponding to the R channel and the G channel in the RGBW image in the second sampling map corresponding to the B channel. In the first interpolation map corresponding to the B channel, positions corresponding to the R channel, the B channel, and the G channel in the RGBW image all have B channel values, while positions corresponding to the W channel in the RGBW image have not yet interpolated B channel values.

[0085] To simplify the view, Figure 3The first interpolation graphs corresponding to the color channels are combined and displayed as FIG12. In FIG12, the positions corresponding to the R channel, the G channel, and the B channel in the RGBW image all have R channel values, B channel values, and G channel values, while the positions corresponding to the W channel in the RGBW image have not yet interpolated R channel values, B channel values, and G channel values.

[0086] S204 , using the full-color image as a guide, perform second interpolation processing on the first interpolation images corresponding to each color channel to obtain a full-resolution color image corresponding to each color channel.

[0087] Since errors are inevitably introduced in the process of performing the first interpolation processing on the second sampling images corresponding to each color channel, when performing the second interpolation processing on the first interpolation images obtained by interpolating the second sampling images corresponding to each color channel, the full-color image is further used for guidance, so that the errors introduced in the first interpolation processing can be corrected, and the color information of each color channel can be better restored, thereby further improving the image quality of the final RGB image.

[0088] In a specific implementation, step S204 specifically includes: taking each color channel as the target color channel, using the full-color image for guidance, interpolating the second interpolation position in the first interpolation map, and obtaining a full-resolution color image corresponding to the target color channel, wherein the second interpolation position corresponds to the W channel position of the RGBW image.

[0089] For example, with the G channel as the target color channel, the position corresponding to the R channel and the B channel of the RGBW image in the first interpolation map corresponding to the G channel has been interpolated to obtain the G channel value. At this time, the G channel value is interpolated for the position corresponding to the W channel of the RGBW image in the first interpolation map using the guidance of the image again, and the full-resolution color image corresponding to the G channel can be obtained. When the G channel value is interpolated for the position corresponding to the W channel of the RGBW image in the first interpolation map, the position corresponding to the R channel and the B channel of the RGBW image in the first interpolation map is used to interpolate the G channel value. Since the first interpolation process inevitably introduces errors, at this time, the W channel information provided by the full-color image is further used to guide the interpolation of the G channel value for the position corresponding to the W channel of the RGBW image in the first interpolation map, so that the error introduced in the first interpolation process can be corrected, so that the G channel value of the position corresponding to the W channel of the RGBW image in the first interpolation map corresponding to the G channel can be more accurately restored.

[0090] In a similar manner, the second interpolation processing is performed on the second interpolation map corresponding to the R channel and the second interpolation map corresponding to the B channel, so that the R channel value of the position corresponding to the W channel of the RGBW image in the first interpolation map corresponding to the R channel and the B channel value of the position corresponding to the W channel of the RGBW image in the first interpolation map corresponding to the B channel can be restored more accurately, so that the full-resolution color image corresponding to each color channel can better restore the color information of each color channel, thereby further improving the image quality of the final RGB image.

[0091] To simplify the view, Figure 3 In FIG. 3 , the full-resolution color images corresponding to each color channel are combined and displayed as one. Figure 3 As shown, each pixel position in Figure I3 has an R channel value, a B channel value and a G channel value.

[0092] S205 , merging the full-resolution color images corresponding to each color channel to generate an RGB color image.

[0093] In the embodiment of the present application, based on the characteristics that the number of W pixels is the largest and the distribution is uniform, the W channel in the RGBW image is interpolated first, so that more texture detail information in the RGBW image can be obtained, and then the full-color image obtained by interpolating the W channel is used for guidance, and each color channel in the RGBW image is interpolated respectively, so that the full-resolution color image corresponding to each color channel obtained by interpolation has a higher color resolution, thereby improving the image quality of the final RGB image. In addition, when interpolating each color channel pair, the full-color image is first used for guidance, and the first interpolation processing is performed on the second sampling map corresponding to each color channel to obtain the first interpolation map corresponding to each color channel, and then the full-color image is used for guidance again, and the second interpolation processing is performed on the first interpolation map corresponding to each color channel to obtain the full-resolution color image corresponding to each color channel. Since the full-color image is further used for guidance in the second interpolation processing, the error introduced in the first interpolation processing can be corrected, and the color information of each color channel can be better restored, thereby further improving the image quality of the final RGB image.

[0094] based on Figure 2 The embodiment shown, as Figure 4 As shown, in one implementation of the present application, step S203 may include:

[0095] S2031a. Determine, in a second sampling map corresponding to a target color channel, a plurality of first candidate positions for interpolating a first interpolation position.

[0096] Depending on the distribution of the color pixels of the RGBW image and the target color channel, the multiple first candidate positions are different.

[0097] For example, in one implementation, the target color channel is the G channel; accordingly, step S2031a may include: determining four G channel positions in the second sampling image corresponding to the G channel and adjacent to the first interpolation position in the horizontal direction and the vertical direction as multiple first candidate positions. In this implementation, four first candidate positions are selected when interpolating the first interpolation position, so that more reference points can be obtained when interpolating the first interpolation position, thereby making the interpolation pixel value of the first interpolation position obtained by interpolation more accurate.

[0098] S2032a, determining weights corresponding to the plurality of first candidate positions according to the W channel value corresponding to the first interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of first candidate positions.

[0099] In one implementation, the weights corresponding to the multiple first candidate positions may be determined according to the difference between the W channel value corresponding to the first interpolation position in the full-color image and the W channel values ​​corresponding to the multiple first candidate positions. For example, the weight corresponding to each first candidate position may be proportional to the absolute value of the reciprocal of the above corresponding difference, and the smaller the difference, the larger the corresponding weight.

[0100] S2033a. Based on the weights corresponding to the multiple first candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple first candidate positions to obtain the color difference value corresponding to the first interpolation position.

[0101] The color difference value of the first candidate position is the difference between the target color channel value corresponding to the first candidate position and the W channel value. For example, if the target color channel is the G channel, the color difference value of the first candidate position is the difference between the G channel value corresponding to the first candidate position in the second sampling image corresponding to the G channel and the W channel value corresponding to the first candidate position in the full-color image.

[0102] In a specific implementation manner, in step S2033a, based on the weights corresponding to the multiple first candidate positions, the color difference values ​​corresponding to the multiple first candidate positions may be weighted summed to obtain the color difference value corresponding to the first interpolation position.

[0103] S2034a. Based on the W channel value corresponding to the first interpolation position and the color difference value corresponding to the first interpolation position, interpolate the first interpolation position to obtain a first interpolation map corresponding to the target color channel.

[0104] In a specific implementation manner, the W channel value corresponding to the first interpolation position and the color difference value corresponding to the first interpolation position may be summed, and the summed result is used as the interpolation pixel value of the first interpolation position.

[0105] In the above manner, each first interpolation position in the second sampling map corresponding to the target color channel is traversed to perform the above operation, so as to obtain the first interpolation map corresponding to the target color channel.

[0106] For ease of understanding, the following Figure 1 The RGBW image corresponding to the RGBWCFA shown, taking the G channel as the target color channel as an example, the process of steps S2031a to S2034a is described in detail. When the G channel is the target color channel, the first interpolation position in the second sampling map corresponding to the G channel corresponds to the positions of the B channel and the R channel in RGBW. Since the interpolation process of the R channel position is similar to that of the B channel position, only the specific interpolation process of interpolating the G channel value at the B channel position is given here. In addition, since the RGBW image is composed of multiple minimum repeating units, each minimum repeating unit is composed of 4×4 blocks (also called clusters) with different colors. For the convenience of illustration, in this example, the interpolation of the B channel position in the B channel block in the second sampling map corresponding to the G channel is taken as an example for explanation.

[0107] Figure 5 The position marked with B in (a) is the first interpolation position in the second sampling image corresponding to the G channel, and the position marked with G is the four G channel blocks adjacent to the B channel block above, below, left and right. Figure 5 (b) shows the full-color image with Figure 5 The full color value at the corresponding position of the G channel block in (a), Figure 5 (c) shows the Figure 5 The color difference value at the corresponding position of the G channel block in (a).

[0108] To explain more clearly, Figure 6a and Figure 6b They show the Figure 5 An exemplary interpolation direction and interpolation method for interpolating the B channel position in the B channel block in FIG. Figure 6a As shown, for each B channel position in the B channel block, the color difference values ​​of the four G channel positions above, below, left and right of the B channel position are weightedly fused to obtain the color difference value of the channel position, and then based on the color difference value of the channel position and the W channel value, the G channel value of the position is obtained. Figure 6bAs shown, the first interpolation positions (i.e., the 8 B channel positions to be interpolated) are respectively represented as k1-k8, the multiple first candidate positions for interpolating the B channel position k1 are positions marked with a1, b1, c1 and d1, the multiple first candidate positions for interpolating the B channel position k2 are positions marked with a2, b1, c1 and d2, the multiple first candidate positions for interpolating the B channel position k3 are positions marked with a3, b2, c2 and d3, and the multiple first candidate positions for interpolating the B channel position k4 are positions marked with a4, b2, c2 and d4. The multiple first candidate positions for interpolating the B channel position k5 are positions marked with a1, b3, c3 and d1, the multiple first candidate positions for interpolating the B channel position k6 are positions marked with a2, b3, c3 and d2, the multiple first candidate positions for interpolating the B channel position k7 are positions marked with a3, b4, c4 and d3, and the multiple first candidate positions for interpolating the B channel position k8 are positions represented by a4, b4, c4 and d4. The G channel values ​​of positions k1-k8 can be obtained by formula (3).

[0109]

[0110] Among them, G k1 , G k2 ,……,G k3 Respectively represent the G channel values ​​at positions k1, k2, ..., k8, W k1 , W k2 ,……,W k8 Respectively represent the W channel values ​​at positions k1, k2, ..., k8, G a1 , G a2 ,……,G d4 Respectively represent the G channel values ​​at positions a1, a2, ..., d4, W a1 , W a2 , ... Wa4 represent the W channel values ​​at positions a1, a2, ..., d4 respectively, w 11 ,w 12 ,w 13 ,w 14 …w 81 ,w 82 ,w 83 ,w 84 Represents the weight corresponding to the first candidate position.

[0111] Among them, the weight w 11 ,w 12 ,w 13 ,w 14 It can be calculated by formula (4), and the remaining weights can be deduced in the same way.

[0112]

[0113] By traversing each B channel block in the second sampling map corresponding to the G channel, the G channel value of each B channel position in the second sampling map corresponding to the G channel can be obtained in the above manner. It should be understood that in a similar manner, the G channel value of each R channel position in the second sampling map corresponding to the G channel can be interpolated to obtain the first interpolation map corresponding to the G channel, that is, Figure 3 The checkerboard image shown in I2.

[0114] It should be understood that this embodiment only provides an exemplary implementation method for interpolating the second sampling images corresponding to each color channel into the first interpolation images corresponding to each color channel through the first interpolation processing. In other embodiments, the second sampling images corresponding to each color channel are interpolated into the first interpolation images corresponding to each color channel in other suitable ways, and this embodiment does not limit this.

[0115] based on Figure 2 The embodiment shown, as Figure 7 As shown, in another implementation of the present application, the first interpolation position includes a first target interpolation position and a second target interpolation position, and the first target interpolation position and the second target interpolation position correspond to different color channels in the RGBW image. For example, the target color channel is the R channel, the first target interpolation position corresponds to the B channel position in the RGBW image, and the second target interpolation position corresponds to the G channel position in the RGBW image. For another example, the target color channel is the B channel, the first target interpolation position corresponds to the R channel position in the RGBW image, and the second target interpolation position corresponds to the G channel position in the RGBW image.

[0116] Accordingly, step S203 includes:

[0117] S2031b, using the full-color image as a guide, interpolate the first target interpolation position in the second sampling image corresponding to the target color channel to obtain an intermediate interpolation image corresponding to the target color channel.

[0118] S2032b, using the full-color image as a guide, interpolate the second target interpolation position in the intermediate interpolation map corresponding to the target color channel to obtain a first interpolation map corresponding to the target color channel.

[0119] Refer to the following Figures 8 to 12 Steps S2031b and S2032b are described.

[0120] like Figure 8 As shown, in one implementation, step S2031b may include:

[0121] S801: Determine, in a second sampling map corresponding to a target color channel, a plurality of second candidate positions for interpolating a first target interpolation position.

[0122] Depending on the distribution of the color pixels of the RGBW image and the target color channel, the plurality of second candidate positions may be different.

[0123] In one implementation of the present application, the target color channel is the R channel, and the first target interpolation position is the same as the position of the B channel in the RGBW image. Accordingly, step S801 includes:

[0124] Four R channel positions in the second sampling image corresponding to the R channel that are adjacent to the first target interpolation position in the diagonal direction and the anti-diagonal direction are determined as multiple second candidate positions.

[0125] In another implementation of the present application, the target color channel is the B channel, and the first target interpolation position is the same as the position of the R channel in the RGBW image; accordingly, step S801 includes:

[0126] Four B channel positions in the second sampling image corresponding to the B channel that are adjacent to the first target interpolation position in the diagonal direction and the anti-diagonal direction are determined as a plurality of second candidate positions.

[0127] In the above two implementations, four second candidate positions are selected when interpolating the first target interpolation position, which enables more reference points to be obtained when interpolating the first target interpolation position, thereby making the interpolated pixel value of the first target interpolation position more accurate.

[0128] S802 : Determine weights corresponding to the plurality of second candidate positions according to the W channel value corresponding to the first target interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of second candidate positions.

[0129] In one implementation, the weights corresponding to the plurality of second candidate positions may be determined according to the difference between the W channel value corresponding to the first target interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of second candidate positions. For example, the weight corresponding to each second candidate position may be proportional to the absolute value of the inverse of the corresponding difference, and the smaller the difference, the larger the corresponding weight.

[0130] S803 . Based on the weights corresponding to the multiple second candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple second candidate positions to obtain the color difference value corresponding to the first target interpolation position.

[0131] The color difference value of the second candidate position is the difference between the target color channel value corresponding to the second candidate position and the W channel value. For example, if the target color channel is the R channel, the color difference value of the first candidate position is the difference between the G channel value corresponding to the second candidate position in the second sampling image corresponding to the R channel and the W channel value corresponding to the second candidate position in the full-color image.

[0132] In one implementation, in step S803, based on the weights corresponding to the multiple second candidate positions, the color difference values ​​corresponding to the multiple second candidate positions may be weighted summed to obtain the color difference value corresponding to the first target interpolation position.

[0133] S804: interpolate the first target interpolation position based on the W channel value corresponding to the first target interpolation position and the color difference value corresponding to the first target interpolation position to obtain an intermediate interpolation map corresponding to the target color channel.

[0134] In an implementation, the W channel value corresponding to the first target interpolation position and the color difference value corresponding to the first target interpolation position may be summed, and the summed result may be used as the interpolation pixel value of the first target interpolation position.

[0135] In the above manner, each first target interpolation position in the second sampling map corresponding to the target color channel is traversed to perform the above operation, so as to obtain a first interpolation map corresponding to the target color channel.

[0136] For ease of understanding, the following Figure 1 The RGBW image corresponding to the RGBWCFA shown in the figure takes the R channel as the target color channel as an example, and the process of steps S801 to S804 is described in detail. When the R channel is the target color channel, the first target interpolation position in the second sampling map corresponding to the R channel corresponds to the position of the B channel in the RGBW. For the convenience of illustration, in this example, the interpolation of the B channel position in the B channel block in the second sampling map corresponding to the R channel is described as an example.

[0137] like Figure 9a As shown, Figure 9a The color difference values ​​of the corresponding positions in the adjacent 4 R channel blocks used for interpolation of the B channel position in the B channel block are shown. Among them, the blank block in the center is the B channel block, and each B channel position in the B channel block is the first target interpolation position. The position where the box marked with RW is located is used for multiple second candidate positions for interpolation of the 8 B channel positions in the B channel block.

[0138] like Figure 9bAs shown, for each B pixel position, the color difference value (RW) of the nearest R pixel position in the diagonal and anti-diagonal directions is used for weighted fusion estimation to obtain the color difference value of the B pixel position, and the w channel value of the B pixel position is added to obtain the final R channel value.

[0139] The eight B channel positions as the first target interpolation positions are respectively indicated as k1-k8, and the multiple second candidate positions for interpolating the eight B pixel positions are respectively indicated as a1-a3, b1-b4, c1-c4 and d1-d3. Among them, the multiple second candidate positions for interpolating the B channel position k1 are positions marked with a1, b1, c1 and d1, the multiple second candidate positions for interpolating the B channel position k2 are positions marked with a2, b2, c2 and d2, the multiple second candidate positions for interpolating the B channel position k3 are positions marked with a1, b2, c2 and d1, and the multiple second candidate positions for interpolating the B channel position k4 are positions marked with a2, b3, c3 and d2. The multiple second candidate positions for interpolating the B channel position k5 are positions marked with a3, b2, c2 and d3, the multiple second candidate positions for interpolating the B channel position k6 are positions marked with a1, b3, c3 and d1, the multiple second candidate positions for interpolating the B channel position k7 are positions marked with a3, b3, c3 and d3, and the multiple second candidate positions for interpolating the B channel position k8 are positions represented by a1, b4, c4 and d1. The R channel values ​​of positions k1-k8 can be obtained by formula (5).

[0140]

[0141] Among them, R k1 , R k2 ,……,R k3 Respectively represent the R channel values ​​at positions k1, k2, ..., k8, W k1 , W k2 ,……,W k8 Respectively represent the W channel values ​​at positions k1, k2, ..., k8, R a1 , R a2 ,……R d3 Respectively represent the R channel values ​​at positions a1, a2, ..., d3, W a1 , W a2 , ...W d3 Respectively represent the W channel values ​​at positions a1, a2, ..., d3, w 11 ,w 12 ,w 13 , w 14 …w 81 , w 82 ,w83 ,w 84 represents the weight corresponding to the second candidate position. The calculation process of each weight can refer to the weight calculation process of formula (4), which will not be repeated here.

[0142] In addition, when the target color channel is the B channel, and the first target interpolation position in the second sampling map corresponding to the B channel corresponds to the position of the R channel in RGBW, the calculation process is similar to the above calculation process. In order to avoid repetition, it will not be repeated here.

[0143] like Fig.10 As shown, in one implementation of the present application, step S2032b includes:

[0144] S1001. Determine, in an intermediate interpolation map corresponding to a target color channel, a plurality of third candidate positions for interpolating a second target interpolation position.

[0145] Depending on the distribution of the color pixels of the RGBW image and the target color channel, the plurality of third candidate positions may be different.

[0146] In one implementation of the present application, the target color channel is an R channel or a B channel, and the second target interpolation position is the same as the position of the G channel in the RGBW image. Accordingly, step S1001 includes: determining two R or B channel positions adjacent to the second target interpolation position in the first direction and two B or R channel positions in the second direction in the intermediate interpolation image corresponding to the R or B channel as the multiple third candidate positions.

[0147] Among them, if the left and right sides of the G channel block in the RGBW image are B channel blocks and the top and bottom are R channel blocks, the first direction is the vertical direction and the second direction is the horizontal direction, that is, when the target color channel is the B channel, the two R channel positions in the intermediate interpolation image corresponding to B that are adjacent to the second target interpolation position in the vertical direction and the two B channel positions in the horizontal direction are determined as multiple third candidate positions; when the target color channel is the R channel, the two B channel positions in the intermediate interpolation image corresponding to R that are adjacent to the second target interpolation position in the vertical direction and the two R channel positions in the horizontal direction are determined as multiple third candidate positions.

[0148] In the opposite case, if the left and right sides of the G channel block in the RGBW image are R channel blocks and the top and bottom are B channel blocks, the first direction is the horizontal direction and the second direction is the vertical direction, that is, when the target color channel is the B channel, the two R channel positions in the intermediate interpolation image corresponding to B that are adjacent to the second target interpolation position in the horizontal direction and the two B channel positions in the vertical direction are determined as multiple third candidate positions; when the target color channel is the R channel, the two B channel positions in the intermediate interpolation image corresponding to R that are adjacent to the second target interpolation position in the horizontal direction and the two R channel positions in the vertical direction are determined as multiple third candidate positions.

[0149] In this implementation, four third candidate positions are selected when interpolating the second target interpolation position, which enables more reference points to be obtained when interpolating the second target interpolation position, thereby making the interpolated pixel value of the second target interpolation position more accurate.

[0150] S1002 : Determine weights corresponding to the plurality of third candidate positions according to the W channel value corresponding to the second target interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of third candidate positions.

[0151] In one implementation, the weights corresponding to the plurality of third candidate positions may be determined according to the difference between the W channel value corresponding to the second target interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of third candidate positions. For example, the weight corresponding to each third candidate position may be proportional to the absolute value of the inverse of the corresponding difference, and the smaller the difference, the larger the corresponding weight.

[0152] S1003 . Based on the weights corresponding to the multiple third candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple third candidate positions to obtain the color difference value corresponding to the second target interpolation position.

[0153] The color difference value of the third candidate position is the difference between the target color channel value and the W channel value corresponding to the third candidate position.

[0154] Specifically, based on the weights corresponding to the multiple third candidate positions, the color difference values ​​corresponding to the multiple third candidate positions may be weighted summed to obtain the color difference value corresponding to the second target interpolation position.

[0155] S1004 . Interpolate the second target interpolation position based on the W channel value corresponding to the second target interpolation position and the color difference value corresponding to the second target interpolation position to obtain a first interpolation map corresponding to the target color channel.

[0156] In this implementation, the W channel value corresponding to the second target interpolation position and the color difference value corresponding to the second target interpolation position may be summed, and the summed result may be used as the interpolation pixel value of the second target interpolation position.

[0157] In the above manner, each second target interpolation position in the intermediate interpolation map corresponding to the target color channel is traversed to perform the above operation, so as to obtain the intermediate interpolation map corresponding to the target color channel.

[0158] For ease of understanding, the following Fig.11 and Fig.12 ,against Figure 1 The RGBW image corresponding to the RGBWCFA shown in the figure illustrates the process of steps S1001 to S1004 through two examples. In the RGBW image, the left and right blocks of the G channel block are B channel blocks and the upper and lower blocks are R channel blocks.

[0159] First example

[0160] In this example, the process of interpolating the R channel value at the second target interpolation position in the intermediate interpolation map corresponding to the R channel is described. When the R channel is the target color channel, the first target interpolation position in the second sampling map corresponding to the R channel corresponds to the position of the B channel in RGBW, and the second target interpolation position in the second sampling map corresponding to the R channel corresponds to the position of the G channel in RGBW.

[0161] Fig.11 (a) in the figure is the second sampling diagram corresponding to the R channel, wherein the R channel position is represented by a dense grid pattern filling method, the first target interpolation position is represented by a sparse grid pattern filled box, and the second target interpolation position is represented by a gray filled square. In this example, the interpolation of the second target interpolation position (represented by k1-k8) in the central block is taken as an example for explanation. Fig.11 (b) in the figure shows a process of performing a first interpolation process on the second sampling image corresponding to the R channel by adopting steps S802 to S804 to obtain an intermediate interpolation image corresponding to the R channel. Fig.11 (c) in FIG. 1 shows a process of interpolating the second target interpolation position represented by k1-k8 in the intermediate interpolation map corresponding to the R channel using steps S1001 to S1004. The R channel value of each G channel position represented by k1-k8 is interpolated using the R channel values ​​of two R channel positions in the horizontal direction and the R channel values ​​of two B channel positions in the vertical direction.

[0162] In this way, each G channel position in the intermediate interpolation map corresponding to the R channel is traversed, and the R channel value is interpolated at each G channel position to obtain the intermediate interpolation map corresponding to the R channel, that is, the checkerboard image corresponding to the R channel.

[0163] Second example

[0164] In this example, the process of interpolating the B channel value at the second target interpolation position in the intermediate interpolation map corresponding to the B channel is described. When the B channel is the target color channel, the first target interpolation position in the second sampling map corresponding to the B channel corresponds to the position of the R channel in RGBW, and the second target interpolation position in the second sampling map corresponding to the B channel corresponds to the position of the G channel in RGBW.

[0165] Fig.12 (a) in the figure is the second sampling image corresponding to the B channel, wherein the B channel position is represented by a sparse grid pattern filling method, the first target interpolation position is represented by a box filled with a dense grid pattern, and the second target interpolation position is represented by a gray-filled square. In this example, the interpolation of the second target interpolation position (represented by k1-k8) in the central block is taken as an example for explanation. Fig.12 (b) in FIG. 1 shows a process of performing a first interpolation process on the second sampling image corresponding to the B channel by adopting steps S802 to S804 to obtain an intermediate interpolation image corresponding to the B channel. Fig.11 (c) in FIG. 1 shows a process of interpolating the second target interpolation position represented by k1-k8 in the intermediate interpolation map corresponding to the B channel using steps S1001 to S1004. Specifically, the B channel value of the G channel position represented by k1-k8 is interpolated using the B channel values ​​of the two B channel positions in the vertical direction and the B channel values ​​of the two R channel positions in the horizontal direction.

[0166] In this way, each G channel position in the intermediate interpolation map corresponding to the B channel is traversed, and the B channel value is interpolated at each G channel position to obtain the intermediate interpolation map corresponding to the B channel, that is, the checkerboard image corresponding to the B channel.

[0167] According to the interpolation method using steps S2031a to 2034a, steps S801 to S804 and steps S1001 to S1002, a checkerboard image corresponding to each color channel of R, G, and B can be obtained. In each image, only the W channel position lacks RGB color information.

[0168] based on Figure 2 The embodiment shown, as Fig.13 As shown, in one implementation of the present application, step S204 includes:

[0169] S2041. Determine, in a first interpolation map corresponding to a target color channel, a plurality of fourth candidate positions for interpolating a second interpolation position.

[0170] Specifically, in the first interpolation maps corresponding to the R channel, the G channel, and the B channel, only the W channel position is missing the R channel value, the G channel value, and the B channel value. Therefore, the second interpolation position in the first interpolation map is the same as the W channel position in the RGBW image.

[0171] Depending on the different gradient information of the second interpolation position in the horizontal direction and the vertical direction, the specific positions and the number of the second candidate positions are different.

[0172] In one implementation, step S2041 includes:

[0173] Step A: Calculate the horizontal gradient and the vertical gradient of the second interpolation position according to the W channel values ​​corresponding to a plurality of W channel positions adjacent to the second interpolation position in the full-color image.

[0174] Step B determines the region type of the first region where the second interpolation position is located according to the horizontal gradient and the vertical gradient.

[0175] Among them, the region types may include flat regions, vertical edges, horizontal edges, and isotropic regions.

[0176] Specifically, if both the horizontal gradient and the vertical gradient are small, the first region is determined to be a flat region. If the horizontal gradient is much larger than the vertical gradient, the first region is determined to be a vertical edge. If the horizontal gradient is much smaller than the vertical gradient, the first region is determined to be a horizontal edge. Otherwise, the first region is determined to be considered an isotropic region.

[0177] Step C: selecting a plurality of fourth candidate positions in the first interpolation map corresponding to the target color channel according to the region type.

[0178] For a flat area or an isotropic area, multiple pixel positions within a larger area centered on the second interpolation position may be selected as multiple fourth candidate positions. For a horizontal edge, two pixel positions adjacent to the second interpolation position in the horizontal direction may be selected as multiple fourth candidate positions. For a vertical edge, two pixel positions adjacent to the second interpolation position in the vertical direction may be selected as multiple fourth candidate positions.

[0179] S2042: Determine weights corresponding to the plurality of second candidate positions according to the W channel value corresponding to the second interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of fourth candidate positions.

[0180] For example, the weights corresponding to the plurality of second candidate positions are determined according to the difference between the W channel value corresponding to the second interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of second candidate positions. For example, the weight corresponding to each second candidate position may be proportional to the absolute value of the inverse of the corresponding difference, and the smaller the difference, the larger the corresponding weight.

[0181] S2043. Perform weighted fusion on the color difference values ​​corresponding to the multiple fourth candidate positions based on the weights corresponding to the multiple fourth candidate positions to obtain the color difference value corresponding to the second interpolation position.

[0182] The color difference value corresponding to the fourth candidate position is the difference between the target color channel value and the W channel value of the fourth candidate position.

[0183] Specifically, based on the weights corresponding to the multiple fourth candidate positions, the color difference values ​​corresponding to the multiple fourth candidate positions may be weighted summed to obtain the color difference value corresponding to the second interpolation position.

[0184] S2044: interpolate the second interpolation position according to the W channel value of the second interpolation position and the color difference value corresponding to the second interpolation position to obtain a full-resolution color image corresponding to the target color channel.

[0185] The color difference value corresponding to the second interpolation position and the W channel value corresponding to the second interpolation position are summed, and the sum result is used as the interpolation pixel value corresponding to the second interpolation position.

[0186] In a sequential manner, each second interpolation position in the first interpolation map corresponding to the target color channel is traversed to perform the above operation to obtain a full-resolution color image corresponding to the target color channel.

[0187] Combine the following Fig.14 The process of the above steps S2032a to S2032d is explained.

[0188] like Fig.14 As shown, the interpolation process of determining the second candidate position is described by taking the second interpolation position as K and the coordinates of K as (i, j) as an example.

[0189] First, according to formula (6), based on the W channel values ​​corresponding to multiple W channel positions adjacent to the second interpolation position K in the full-color image, the horizontal gradient and vertical gradient of the second interpolation position K are determined.

[0190] gradH=|W i，j-2 -W i，j |+|W i，j+2 -W i，j |+|W i-1，j-1 -W i-1，j+1 |+|W i+1，j-1 -W i+1，j+1 |#(6)

[0191] gradV=|W i-2，j -W i，j |+|W i+2，j -W i，j |+|W i-1，j-1 -Wi+1，j-1 |+|W i-1，j+1 -W i+1，j+1 |#

[0192] Where gradH and gradV represent the horizontal gradient and vertical gradient of the second interpolation position K, respectively. i，j , W i，j-2 , W i，j+2 , W i-1，j-1 , W i-1，j+1 , W i+1，j-1 , W i+1，j+1 , W i-2，j , W i+2，j , W i+1，j-1 , W i-1，j+1 Respectively represent the W channel values ​​of each color channel position in the neighborhood of the second position to be interpolated.

[0193] Next, the first region where the second interpolation position K is located is classified according to the calculated horizontal and vertical gradient information.

[0194] Specifically, if both the horizontal gradient and the vertical gradient are small, the first region is determined to be a flat region. If the horizontal gradient is much larger than the vertical gradient, the first region is determined to be a vertical edge. If the horizontal gradient is much smaller than the vertical gradient, the first region is determined to be a horizontal edge. Otherwise, the first region is determined to be considered an isotropic region.

[0195] If the first area is determined to be a flat area or an isotropic area, multiple pixel positions within the 5x5 neighborhood of the second interpolation position K are determined as multiple fourth candidate positions. If the first area is determined to be a horizontal edge, two pixel positions adjacent to the second interpolation position k in the horizontal direction are determined as multiple fourth candidate positions. If the first area is determined to be a vertical edge, two pixel positions adjacent to the second interpolation position k in the vertical direction are determined as multiple fourth candidate positions.

[0196] After determining a plurality of fourth candidate positions for interpolating the second interpolation position k, the weighted coefficients of the plurality of fourth candidate positions for interpolating the second interpolation position k can be calculated using the full-color image, and then a weighted fusion process can be performed according to the weights corresponding to the plurality of fourth candidate positions K and the color difference values ​​corresponding to the plurality of fourth candidate positions to obtain the color difference value corresponding to the second interpolation position K, and a fusion process is performed according to the color difference value corresponding to the second interpolation position K and the W channel value corresponding to the second interpolation position to obtain the interpolation pixel value corresponding to the second interpolation position K. Specifically, the second interpolation position K can be interpolated according to formula (7).

[0197]

[0198] Among them, C i，jrepresents the interpolation pixel value corresponding to the second interpolation position K, C i，j It can represent one of the R channel value, B channel value and G channel value. i，j Indicates the W channel value of the second interpolation position K, W i+p，j+q Represents the W channel values ​​of multiple fourth candidate positions within the second interpolation position K field, (p, q) is the coordinate offset, and N is the normalization term.

[0199] For flat or isotropic regions, the corresponding (p, q) range is larger, and p, q) satisfies:

[0200]

[0201] That is, the multiple second candidate positions are multiple pixel positions within the 5x5 neighborhood range of the second interpolation position K.

[0202] For horizontal edges, (p,q) satisfies:

[0203] (p,q)={(0,-1),(0,1)

[0204] That is, the plurality of second candidate positions are two pixel positions adjacent to the second interpolation position in the horizontal direction.

[0205] For vertical edges, (p,q) satisfies:

[0206] (p,q)={(-1,0),(1,0),

[0207] That is, the plurality of second candidate positions are two pixel positions adjacent to the second interpolation position in the vertical direction.

[0208] In the above manner, each W channel position (i.e., W pixel position) in the first interpolation map is traversed, and each W channel position in the first interpolation map is interpolated to obtain a full-resolution color image corresponding to the target color channel value. The R, G, and B channels are respectively used as the target color channels, thereby obtaining full-resolution color images corresponding to the R, G, and B channels.

[0209] It should be understood that the method of performing the second interpolation processing on the second interpolation position in the first interpolation map based on the full-color image given here is only an example. In other implementations, other suitable interpolation methods may also be used, and the embodiments of the present application are not limited to this.

[0210] In this implementation, when the second interpolation processing is performed on the first interpolation map corresponding to each color channel, the full-color image is further used for guidance, so that the error introduced when the first interpolation processing is performed on the second sampling map corresponding to each color channel can be corrected, and the color information of each color channel can be better restored, thereby further improving the image quality of the final RGB image.

[0211] Based on the above embodiments, Fig.15 This is a schematic flow chart of another image processing method provided in an embodiment of the present application. Compared with the image processing method provided in the previous embodiment, step S205 in this embodiment includes:

[0212] S2051, using the full-color image as a guide, respectively denoising the full-resolution color image corresponding to each color channel to generate a denoised color image corresponding to each color channel;

[0213] S2052: Merge the denoised color images corresponding to each color channel to generate an RGB image.

[0214] As mentioned above, since the full-resolution full-color image obtained in S202 includes more texture feature information in the RGBW image, after the full-resolution color image corresponding to each color channel is obtained in S204, the W channel information provided by the full-color image can be used to perform denoising on the full-resolution color image to suppress color noise, thereby significantly improving the image quality of the final generated RGB image and enhancing the visual effect. The higher the noise, the greater the impact of the denoising process on the final visual effect.

[0215] In one implementation of the present application, Fig.16 As shown, step S2051 may include:

[0216] S2051a. Generate color difference maps corresponding to each color channel based on the full-color image and the full-resolution color images corresponding to each color channel.

[0217] In a specific implementation, the color difference map corresponding to each color channel can be obtained by subtracting the color image of the full color rate corresponding to each color channel from the corresponding position of the full color image.

[0218] S2051b. Determine multiple target filter kernels for color difference images corresponding to each color channel according to the full-color image.

[0219] In one implementation, S2051b may include: for each color difference map corresponding to each color channel, determining multiple target areas to be filtered in the color difference map corresponding to the color channel; for each target area in the color difference map corresponding to each color channel, determining the gradient information of the target area according to the W channel value corresponding to the target area in the full-color image; for each target area in the color difference map corresponding to each color channel, determining the target filter kernel for the target area from multiple preset filter kernels according to the gradient information of the target area. The multiple preset filter kernels may be filter kernels pre-designed according to experiments and / or experience for the situation where the local area of ​​the image is a flat area, an edge area, etc.

[0220] For example, the aforementioned gradient calculation method can be used to calculate the gradient information of each pixel in the target area in the full-color image, and the target filter kernel for the target area can be adaptively estimated based on the distribution of the gradient information of each pixel in the target area. If the gradient information indicates that the target area is a flat area, a stronger filter kernel can be selected from a plurality of preset filter kernels as the target filter kernel, and if the gradient information indicates that the target area is an edge area, a weaker filter kernel can be selected from a plurality of preset filter kernels as the target filter kernel. A stronger filter kernel indicates that the weights for each pixel in the target area are substantially equal, and a weaker filter kernel indicates that the weights for pixels near the center of the target area are larger, and the weights for pixels far from the center of the target area are smaller.

[0221] S2051c, using the multiple target filter cores to perform filtering processing on the color difference images corresponding to each color channel, to obtain a filtered color difference image corresponding to each color channel;

[0222] Since the local target filter kernel of the image is adaptively estimated based on the gradient information of the full-color image, the target filter kernel is a stronger filter kernel in the flat area and a weaker filter kernel in the boundary area. Therefore, when the color difference map is weighted filtered according to the determined target filter kernel, the noise can be suppressed while retaining the boundary of the image.

[0223] S2051d, fusing the full-color image and the filtered color difference images corresponding to each color channel to obtain a denoised color image corresponding to each color channel.

[0224] For example, the corresponding positions of the full-color image and the filtered color difference image corresponding to each color channel can be added to obtain the denoised color image corresponding to each color channel. After obtaining the denoised color image corresponding to each color channel, the denoised color image corresponding to each color channel is merged to generate an RGB image.

[0225] Since the interpolation process of the RGB channels is inevitably affected by noise, the embodiment of the present application uses the relatively clean and clear texture information provided by the full-color image to suppress the noise on the RGB channels, further improving the image quality of the final RGB image and enhancing the visual effect of the image.

[0226] based on Fig.15 In the embodiment shown, in one implementation of the present application, before the full-color image and the filtered color difference images corresponding to each color channel are fused, the process may further include: performing low-pass filtering on the full-color image, thereby further reducing the noise residue of the full-color image, that is, reducing the noise residue on the W channel. In turn, the noise residue on the color image obtained after the fusion process is further reduced. In this embodiment, any suitable image denoising algorithm can be used to denoise the W image, and this application does not limit this.

[0227] Reference Fig.17 , shows a schematic diagram of the structure of an image processing device according to an embodiment of the present application. The device 170 is used to process an RGBW image acquired by an image sensor, in which the W channel and the R, G, and B color channels are arranged alternately in the horizontal and vertical directions. Fig.17 As shown, the device 170 includes:

[0228] The sampling module 1701 is used to sample the W channel in the RGBW image to obtain a first sampling map, and to sample each color channel in the RGBW image to obtain a second sampling map corresponding to each color channel;

[0229] W interpolation module 1702, used to interpolate the first sampling image into a full-resolution full-color image;

[0230] A first interpolation module 1703 is used to perform a first interpolation process on the second sampling images corresponding to each color channel using the full-color image as a guide, so as to obtain a first interpolation image corresponding to each color channel;

[0231] The first interpolation module 1704 is used to use the full-color image as a guide to perform a second interpolation process on the first interpolation map corresponding to each color channel to obtain a full-resolution color image corresponding to each color channel.

[0232] The generating module 1705 is used to merge the full-resolution color images corresponding to each color channel to generate an RGB image.

[0233] In one implementation of the present application, the first interpolation module 1703 is further used to:

[0234] Each color channel is used as a target color channel respectively, and the full-color image is used as a guide to interpolate a first interpolation position in a second sampling map corresponding to the target color channel to obtain a first interpolation map corresponding to the target color channel, wherein the first interpolation position is a position in the R channel, the G channel, and the B channel of the RGBW image except the target color channel.

[0235] In one implementation of the present application, the first interpolation module 1703 is further used to:

[0236] Determining, in a second sampling map corresponding to the target color channel, a plurality of first candidate positions for interpolating the first interpolation position;

[0237] Determining weights corresponding to the plurality of first candidate positions according to the W channel value corresponding to the first interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of first candidate positions;

[0238] Based on the weights corresponding to the multiple first candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple first candidate positions to obtain the color difference value corresponding to the first interpolation position, wherein the color difference value of the first candidate position is the difference between the target color channel value and the W channel value corresponding to the first candidate position;

[0239] Based on the W channel value corresponding to the first interpolation position and the color difference value corresponding to the first interpolation position, the first interpolation position is interpolated to obtain a first interpolation map corresponding to the target color channel.

[0240] In an implementation of the present application, the target color channel is a G channel; the first interpolation module 1703 is further used to:

[0241] Four G channel positions in the second sampling image corresponding to the G channel that are adjacent to the first interpolation position in the horizontal direction and the vertical direction are determined as the multiple first candidate positions.

[0242] In one implementation of the present application, the first interpolation position includes a first target interpolation position and a second target interpolation position, and the first target interpolation position and the second target interpolation position correspond to different color channels in the RGBW image; the first interpolation module 1703 is further used to:

[0243] Using the full-color image as a guide, interpolating the first target interpolation position in the second sampling image corresponding to the target color channel to obtain an intermediate interpolation image corresponding to the target color channel;

[0244] The second target interpolation position in the intermediate interpolation map corresponding to the target color channel is interpolated using the full-color image as a guide to obtain a first interpolation map corresponding to the target color channel.

[0245] In one implementation of the present application, the first interpolation module 1703 is further used to:

[0246] In a second sampling map corresponding to the target color channel, determining a plurality of second candidate positions for interpolating the first target interpolation position;

[0247] Determining weights corresponding to the plurality of second candidate positions according to a W channel value corresponding to the first target interpolation position in the full-color image and W channel values ​​corresponding to the plurality of second candidate positions;

[0248] Based on the weights corresponding to the multiple second candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple second candidate positions to obtain the color difference value corresponding to the first target interpolation position, wherein the color difference value of the second candidate position is the difference between the target color channel value corresponding to the second candidate position and the W channel value;

[0249] Based on the W channel value corresponding to the first target interpolation position and the color difference value corresponding to the first target interpolation position, the first target interpolation position is interpolated to obtain an intermediate interpolation map corresponding to the target color channel.

[0250] In one implementation of the present application, the target color channel is the R channel, and the first target interpolation position is the same as the position of the B channel in the RGBW image; the first interpolation module 1703 is further used to:

[0251] Determine four R channel positions in the second sampling graph corresponding to the R channel that are adjacent to the first target interpolation position in a diagonal direction and a counter-diagonal direction as the multiple second candidate positions;

[0252] In one implementation of the present application, the target color channel is a B channel, and the first target interpolation position is the same as the position of the R channel in the RGBW image; the first interpolation module 1703 is further used to:

[0253] Four B channel positions in the second sampling image corresponding to the B channel that are adjacent to the first target interpolation position in the diagonal direction and the anti-diagonal direction are determined as the multiple second candidate positions.

[0254] In one implementation of the present application, the first interpolation module 1703 is further used to:

[0255] The method of using the full-color image for guidance to interpolate the second target interpolation position in the intermediate interpolation map corresponding to the target color channel to obtain a first interpolation map corresponding to the target color channel includes:

[0256] Determining, in the intermediate interpolation map corresponding to the target color channel, a plurality of third candidate positions for interpolating the second target interpolation position;

[0257] determining weights corresponding to the plurality of third candidate positions according to a W channel value corresponding to the second target interpolation position in the full-color image and W channel values ​​corresponding to the plurality of third candidate positions;

[0258] Based on the weights corresponding to the multiple third candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple third candidate positions to obtain the color difference value corresponding to the second target interpolation position, wherein the color difference value of each third candidate position is the difference between the target color channel value corresponding to the third candidate position and the W channel value;

[0259] Based on the W channel value corresponding to the second target interpolation position and the color difference value corresponding to the second target interpolation position, the second target interpolation position is interpolated to obtain a first interpolation map corresponding to the target color channel.

[0260] In one implementation of the present application, the target color channel is an R or B channel, and the second target interpolation position is the same as the G channel position of the RGBW image; the first interpolation module 1703 is further used to:

[0261] Two R or B channel positions adjacent to the second target interpolation position in the first direction and two B or R channel positions in the second direction in the intermediate interpolation image corresponding to the R or B channel are determined as the multiple third candidate positions.

[0262] In one implementation of the present application, the second interpolation module 1704 is further configured to:

[0263] Each color channel is taken as a target color channel respectively, and the full-color image is used as a guide to interpolate the second interpolation position in the first interpolation map to obtain a full-resolution color image corresponding to the target color channel, wherein the second interpolation position corresponds to the position of the W channel of the RGBW image.

[0264] In one implementation of the present application, the second interpolation module 1704 is further configured to:

[0265] Determining, in the first interpolation map corresponding to the target color channel, a plurality of fourth candidate positions for interpolating the second interpolation position;

[0266] Determining weights corresponding to the plurality of second candidate positions according to the W channel value corresponding to the second interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of fourth candidate positions;

[0267] Based on the weights corresponding to the multiple fourth candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple fourth candidate positions to obtain the color difference value corresponding to the second interpolation position, wherein the color difference value corresponding to the fourth candidate position is the difference between the target color channel value and the W channel value of the fourth candidate position;

[0268] The second interpolation position is interpolated according to the W channel value of the second interpolation position and the color difference value corresponding to the second interpolation position to obtain a full-resolution color image corresponding to the target color channel.

[0269] In one implementation of the present application, the second interpolation module 1704 is further configured to:

[0270] Calculating a horizontal gradient and a vertical gradient of the second interpolation position according to W channel values ​​corresponding to a plurality of W channel positions adjacent to the second interpolation position in the full-color image;

[0271] determining, according to the horizontal gradient and the vertical gradient, a region type of the first region where the second interpolation position is located;

[0272] The plurality of fourth candidate positions are selected in the first interpolation map corresponding to the target color channel according to the region type.

[0273] In one implementation of the present application, Fig.18 As shown, the generation module 1205 includes:

[0274] The noise suppression unit 1205a is used to perform denoising on the full-resolution color images corresponding to the color channels using the full-color image as a guide, so as to generate denoised color images corresponding to the color channels;

[0275] The merging unit 1205b is used to merge the denoised color images corresponding to the color channels to generate the RGB image.

[0276] In another implementation of the present application, Fig.15 As shown, the noise suppression unit 1205a includes:

[0277] A color difference generating unit 1901 is used to generate a color difference map corresponding to each color channel according to the full-color image and the full-resolution color image corresponding to each color channel;

[0278] A filter weight estimation unit 1902 is used to determine a plurality of target filter kernels for color difference images corresponding to each color channel according to the full-color image;

[0279] A color gamut noise filtering unit 1903 is used to perform filtering processing on the color difference map corresponding to each color channel using the multiple target filter cores to obtain a filtered color difference map corresponding to each color channel;

[0280] The adding back unit 1904 is used to perform fusion processing on the full-color image and the filtered color difference image corresponding to the color channel to obtain the denoised color image corresponding to each color channel.

[0281] In one implementation of the present application, the filtering weight estimation unit 1902 is used to:

[0282] For each color difference map corresponding to a color channel, determining a plurality of target areas to be filtered in the color difference map corresponding to the color channel;

[0283] For each target area in the color difference map corresponding to each color channel, determine the gradient information of the target area according to the W channel value corresponding to the target area in the full-color image;

[0284] For each target area in the color difference image corresponding to each color channel, a target filter kernel for the target area is determined from a plurality of preset filter kernels according to gradient information of the target area.

[0285] In one implementation of the present application, Fig.19 As shown, the noise suppression unit 1205a further includes a brightness noise suppression unit 1905, which is used to perform low-pass filtering on the full-color image before fusing the full-color image with the filtered third color difference images corresponding to each color channel.

[0286] The image processing device provided in this embodiment is used to execute the image processing method of the aforementioned embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0287] Fig. 20 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig. 20 As shown, the image processing device 200 includes a processor 2001 and a memory 2002 .

[0288] The memory 2002 is used to store computer programs, and the processor 2001 is used to call and run the computer program stored in the memory 2002 to execute any image processing method in the method embodiment.

[0289] The image processing device of this embodiment is used to implement the corresponding methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. The electronic device can be, for example, a portable or mobile computing device such as a smart phone, a laptop, a tablet computer, a gaming device, etc., and the embodiments of this application are not limited to this.

[0290] An embodiment of the present application provides a chip, which includes a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes an image processing method such as any of the aforementioned methods.

[0291] The chip can be applied to the image processing device or electronic device in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the image processing device or electronic device in each method of the embodiments of the present application, and achieve the corresponding effects. For the sake of brevity, it will not be repeated here.

[0292] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0293] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as CDROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the verification code generation method described herein is implemented. In addition, when a general-purpose computer accesses a code for implementing the verification code generation method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the verification code generation method shown here.

[0294] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0295] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. An image processing method for processing an RGBW image acquired by an image sensor, wherein a W channel and R, G, and B color channels are arranged alternately in the horizontal direction and the vertical direction, the method comprising: Sampling the W channel in the RGBW image to obtain a first sampling image, and sampling each color channel in the RGBW image to obtain a second sampling image corresponding to each color channel; interpolating the first sampling image into a full-resolution full-color image; Using the full-color image as a guide, performing first interpolation processing on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel; Using the full-color image as a guide, performing second interpolation processing on the first interpolation images corresponding to the respective color channels, to obtain full-resolution color images corresponding to the respective color channels; Merging the full-resolution color images corresponding to the color channels to generate an RGB image; The method of using the full-color image as a guide to perform first interpolation processing on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel includes: Taking each color channel as a target color channel, using the full-color image as a guide, interpolating a first interpolation position in the second sampling map corresponding to the target color channel to obtain a first interpolation map corresponding to the target color channel, wherein the first interpolation position is a position in the R channel, the G channel, and the B channel of the RGBW image except the target color channel; The method of using the full-color image as a guide to perform second interpolation processing on the first interpolation images corresponding to each color channel to obtain a full-resolution color image corresponding to each color channel includes: Each color channel is taken as a target color channel respectively, and the full-color image is used as a guide to interpolate the second interpolation position in the first interpolation map to obtain a full-resolution color image corresponding to the target color channel, wherein the second interpolation position corresponds to the position of the W channel of the RGBW image.

2. The method according to claim 1, characterized in that: The method of using the full-color image for guidance to interpolate the first interpolation position in the second sampling map corresponding to the target color channel to obtain the first interpolation map corresponding to the target color channel includes: Determining, in a second sampling map corresponding to the target color channel, a plurality of first candidate positions for interpolating the first interpolation position; Determining weights corresponding to the plurality of first candidate positions according to the W channel value corresponding to the first interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of first candidate positions; Based on the weights corresponding to the multiple first candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple first candidate positions to obtain the color difference value corresponding to the first interpolation position, wherein the color difference value of the first candidate position is the difference between the target color channel value and the W channel value corresponding to the first candidate position; Based on the W channel value corresponding to the first interpolation position and the color difference value corresponding to the first interpolation position, the first interpolation position is interpolated to obtain a first interpolation map corresponding to the target color channel.

3. The method according to claim 2, characterized in that The target color channel is the G channel; The determining, in the second sampling map corresponding to the target color channel, a plurality of first candidate positions for interpolating the first interpolation position comprises: Four G channel positions in the second sampling image corresponding to the G channel that are adjacent to the first interpolation position in the horizontal direction and the vertical direction are determined as the multiple first candidate positions.

4. The method according to claim 1, characterized in that The method of using the full-color image for guidance to interpolate the first interpolation position in the second sampling map corresponding to the target color channel to obtain the first interpolation map corresponding to the target color channel includes: The first interpolation position includes a first target interpolation position and a second target interpolation position, and the first target interpolation position and the second target interpolation position correspond to different color channels in the RGBW image; Using the full-color image as a guide, interpolating the first target interpolation position in the second sampling image corresponding to the target color channel to obtain an intermediate interpolation image corresponding to the target color channel; The second target interpolation position in the intermediate interpolation map corresponding to the target color channel is interpolated using the full-color image as a guide to obtain a first interpolation map corresponding to the target color channel.

5. The method according to claim 4, characterized in that The step of using the full-color image for guidance to interpolate the first target interpolation position in the second sampling map corresponding to the target color channel to obtain an intermediate interpolation map corresponding to the target color channel includes: In a second sampling map corresponding to the target color channel, determining a plurality of second candidate positions for interpolating the first target interpolation position; Determining weights corresponding to the plurality of second candidate positions according to a W channel value corresponding to the first target interpolation position in the full-color image and W channel values ​​corresponding to the plurality of second candidate positions; Based on the weights corresponding to the multiple second candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple second candidate positions to obtain the color difference value corresponding to the first target interpolation position, wherein the color difference value of the second candidate position is the difference between the target color channel value corresponding to the second candidate position and the W channel value; Based on the W channel value corresponding to the first target interpolation position and the color difference value corresponding to the first target interpolation position, the first target interpolation position is interpolated to obtain an intermediate interpolation map corresponding to the target color channel.

6. The method according to claim 5, characterized in that The target color channel is an R channel, and the first target interpolation position is the same as a position of a B channel in the RGBW image; Determining, in the second sampling map corresponding to the target color channel, a plurality of second candidate positions for interpolating the first target interpolation position includes: Four R channel positions in the second sampling image corresponding to the R channel that are adjacent to the first target interpolation position in the diagonal direction and the anti-diagonal direction are determined as the multiple second candidate positions.

7. The method according to claim 5, characterized in that The target color channel is a B channel, and the first target interpolation position is the same as a position of an R channel in the RGBW image; Determining, in the second sampling map corresponding to the target color channel, a plurality of second candidate positions for interpolating the first target interpolation position includes: Four B channel positions in the second sampling image corresponding to the B channel that are adjacent to the first target interpolation position in the diagonal direction and the anti-diagonal direction are determined as the multiple second candidate positions.

8. The method according to claim 4, characterized in that The method of using the full-color image for guidance to interpolate the second target interpolation position in the intermediate interpolation map corresponding to the target color channel to obtain a first interpolation map corresponding to the target color channel includes: Determining, in the intermediate interpolation map corresponding to the target color channel, a plurality of third candidate positions for interpolating the second target interpolation position; determining weights corresponding to the plurality of third candidate positions according to a W channel value corresponding to the second target interpolation position in the full-color image and W channel values ​​corresponding to the plurality of third candidate positions; Based on the weights corresponding to the multiple third candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple third candidate positions to obtain the color difference value corresponding to the second target interpolation position, wherein the color difference value of each third candidate position is the difference between the target color channel value corresponding to the third candidate position and the W channel value; Based on the W channel value corresponding to the second target interpolation position and the color difference value corresponding to the second target interpolation position, the second target interpolation position is interpolated to obtain a first interpolation map corresponding to the target color channel.

9. The method according to claim 8, characterized in that The target color channel is the R channel, and the second target interpolation position is the same as the G channel position of the RGBW image; The step of determining, in the intermediate interpolation map corresponding to the target color channel, a plurality of third candidate positions for interpolating the second target interpolation position comprises: determining two R channel positions adjacent to the second target interpolation position in the first direction and two B channel positions in the second direction in the intermediate interpolation map corresponding to the R channel as the plurality of third candidate positions, wherein the first direction and the second direction are perpendicular to each other; or, The target color channel is a B channel, and the second target interpolation position is the same as a G channel position of the RGBW image; The step of determining, in the intermediate interpolation map corresponding to the target color channel, a plurality of third candidate positions for interpolating the second target interpolation position comprises: determining two B channel positions in the intermediate interpolation map corresponding to the B channel that are adjacent to the second target interpolation position in the first direction and two R channel positions in the second direction as the plurality of third candidate positions, wherein the first direction and the second direction are perpendicular to each other.

10. The method according to claim 1, characterized in that The method of using the full-color image as a guide to interpolate the second interpolation position in the first interpolation map to obtain a full-resolution color image corresponding to the target color channel includes: Determining, in the first interpolation map corresponding to the target color channel, a plurality of fourth candidate positions for interpolating the second interpolation position; Determining weights corresponding to the plurality of second candidate positions according to the W channel value corresponding to the second interpolation position in the full-color image and the W channel values ​​corresponding to the plurality of fourth candidate positions; Based on the weights corresponding to the multiple fourth candidate positions, weighted fusion is performed on the color difference values ​​corresponding to the multiple fourth candidate positions to obtain the color difference value corresponding to the second interpolation position, wherein the color difference value corresponding to the fourth candidate position is the difference between the target color channel value and the W channel value of the fourth candidate position; The second interpolation position is interpolated according to the W channel value of the second interpolation position and the color difference value corresponding to the second interpolation position to obtain a full-resolution color image corresponding to the target color channel.

11. The method according to claim 10, characterized in that The determining, in the first interpolation map corresponding to the target color channel, a plurality of fourth candidate positions for interpolating the second interpolation position comprises: Calculating a horizontal gradient and a vertical gradient of the second interpolation position according to W channel values ​​corresponding to a plurality of W channel positions adjacent to the second interpolation position in the full-color image; determining, according to the horizontal gradient and the vertical gradient, a region type of the first region where the second interpolation position is located; The plurality of fourth candidate positions are selected in the first interpolation map corresponding to the target color channel according to the region type.

12. The method according to claim 1, characterized in that The merging process of the full-resolution color images corresponding to the color channels to generate an RGB image includes: Using the full-color image as a guide, denoising is performed on the full-resolution color images corresponding to the color channels to generate denoised color images corresponding to the color channels; The denoised color images corresponding to the color channels are merged to generate the RGB image.

13. The method according to claim 12, characterized in that The method of using the full-color image as a guide to perform denoising on the full-resolution color images corresponding to the color channels to generate denoised color images corresponding to the color channels includes: Generating color difference maps corresponding to each color channel respectively according to the full-color image and the full-resolution color images corresponding to each color channel; Determining, according to the full-color image, a plurality of target filter kernels for color difference images corresponding to each color channel; Using the multiple target filter cores to perform filtering processing on the color difference images corresponding to each color channel to obtain a filtered color difference image corresponding to each color channel; The full-color image and the filtered color difference image corresponding to the color channel are fused to obtain the denoised color image corresponding to each color channel.

14. The method according to claim 13, characterized in that The step of determining, based on the full-color image, a plurality of target filter kernels for the color difference map corresponding to each color channel comprises: For each color difference map corresponding to a color channel, determining a plurality of target areas to be filtered in the color difference map corresponding to the color channel; For each target area in the color difference map corresponding to each color channel, determine the gradient information of the target area according to the W channel value corresponding to the target area in the full-color image; For each target area in the color difference image corresponding to each color channel, a target filter kernel for the target area is determined from a plurality of preset filter kernels according to gradient information of the target area.

15. An image processing device for processing an RGBW image acquired by an image sensor, wherein color pixels of a W channel and color pixels of each of R, G, and B color channels in the RGBW image are arranged alternately in the horizontal direction and the vertical direction, the device comprising: A sampling module, configured to sample the W channel in the RGBW image to obtain a first sampling map, and to sample each color channel in the RGBW image to obtain a second sampling map corresponding to each color channel; W interpolation module, used for interpolating the first sampling image into a full-resolution full-color image; A first interpolation module is used to use the full-color image as a guide to perform a first interpolation process on the second sampling images corresponding to each color channel to obtain a first interpolation image corresponding to each color channel; A second interpolation module is used to use the full-color image as a guide to perform a second interpolation process on the first interpolation map corresponding to each color channel to obtain a full-resolution color image corresponding to each color channel; A generating module, used for performing a merging process based on the full-resolution color images corresponding to the color channels to generate an RGB image; The first interpolation module is specifically used to: use each color channel as a target color channel, use the full-color image as a guide, interpolate the first interpolation position in the second sampling map corresponding to the target color channel, and obtain a first interpolation map corresponding to the target color channel, wherein the first interpolation position is a position in the R channel, the G channel, and the B channel of the RGBW image except the target color channel; The second interpolation module is specifically used to: take each color channel as the target color channel, use the full-color image for guidance, interpolate the second interpolation position in the first interpolation map, and obtain a full-resolution color image corresponding to the target color channel, wherein the second interpolation position corresponds to the position of the W channel of the RGBW image.

16. A chip, comprising: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the image processing method according to any one of claims 1 to 14.

17. An image processing device, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the image processing method according to any one of claims 1 to 14.

Images