Demosaicing and sharpening combined processing method, device and equipment and readable medium

Abstract

The invention provides a demosaicing and sharpening combined processing method, device and equipment and a readable medium, and belongs to the technical field of image processing. The method comprises the following steps: processing original data of a target color channel of an image sensor to obtain a full-resolution target color channel layer; sharpening the target color channel layer with the full resolution to obtain a sharpened layer; and performing interpolation on original data of other color channels except the target color channel in each color channel of the image sensor so as to remove mosaics, and carrying out sharpening on the full-resolution target color channel layer and other color channel layers obtained by demosaicing according to a pixel coordinate superposition sharpening value of a corresponding position of the sharpening layer. The image demosaicing method and device are used for achieving demosaicing and sharpening combined processing and guaranteeing the color consistency of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, equipment and readable medium for combined processing of de-mosaicing and sharpening. Background Art

[0002] In the optical imaging system, the light signal is downsampled to the Bayer pattern through the sensor, and the digital image signal processor (ISP) receives the sensor signal and upsamples the downsampled signal to the RGB primary color channels through the demosaic system to ensure the correct color restoration of the image in subsequent processing.

[0003] The purpose of image sharpening is to increase the significance of image details. In the ISP pipeline, sharpening is usually performed in the YUV domain, and only the brightness (Y) channel is processed, while the color channels (U and V) remain unchanged to ensure that the image does not have hue shift.

[0004] Artificial intelligence (AI) is widely and deeply integrated with imaging systems, especially in latency-sensitive fields such as autonomous driving, which further increases the demand for low processing latency of ISP pipelines. Summary of the invention

[0005] The present disclosure provides a method, device, equipment and readable medium for joint processing of de-mosaicing and sharpening.

[0006] A first aspect of the present disclosure provides a joint processing method of demosaicing and sharpening, comprising:

[0007] Processing the raw data of the target color channel of the image sensor to obtain a full-resolution target color channel layer;

[0008] Sharpening the full-resolution target color channel layer to obtain a sharpened layer;

[0009] The original data of other color channels of each color channel of the image sensor except the target color channel are interpolated to demosaic, and the full-resolution target color channel layer and the other color channel layers obtained by demosaicing are sharpened by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

[0010] A second aspect of the present disclosure provides a digital image signal processor for executing the method described in the first aspect.

[0011] A third aspect of the present disclosure provides a combined processing device for demosaicing and sharpening, comprising:

[0012] An acquisition module is used to process the raw data of the target color channel of the image sensor to obtain a full-resolution target color channel layer;

[0013] A sharpening module, used for sharpening the full-resolution target color channel layer to obtain a sharpened layer;

[0014] The processing module is used to interpolate the original data of other color channels of each color channel of the image sensor except the target color channel to demosaic, and sharpen the full-resolution target color channel layer and the other color channel layers obtained by demosaicing by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

[0015] A third aspect of the present disclosure provides an electronic device, including:

[0016] at least one processor;

[0017] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect;

[0018] At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

[0019] A fourth aspect of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to the first aspect is implemented.

[0020] The present disclosure has the following advantages:

[0021] A full-resolution target color channel layer is obtained based on the original data of the target color channel of the image sensor. The full-resolution target color channel layer is sharpened to obtain a sharpened layer. The original data of other color channels except the target color channel in each color channel of the image sensor are interpolated to demosaic. The full-resolution target color channel layer and other color channel layers obtained by demosaicing are sharpened by superimposing the sharpening value of the corresponding position of the sharpening layer according to the pixel coordinates. In this way, each color channel is sharpened by the sharpening layer to realize the joint processing of demosaicing and sharpening, and the sharpening is pre-placed in the Bayer domain. By superimposing the same sharpening layer on each color channel, the color consistency before and after sharpening is guaranteed while reducing the system delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic flow chart of a combined de-mosaicing and sharpening processing method provided in an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of an example of an RGGB color filter array provided in an embodiment of the present disclosure;

[0024] Figure 3 It is a schematic diagram of the structure of a combined processing device for de-mosaicing and sharpening provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0027] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0029] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0030] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0031] The disclosed embodiment provides a joint processing method of demosaicing and sharpening, which can be applied to a digital image signal processor to perform joint processing of demosaicing and sharpening in the Bayer domain and maintain color consistency. The characteristic of the Bayer domain signal is that each pixel position contains only a single-channel color value. The purpose of demosaicing is to restore the color value of each pixel position to multiple channels according to the sensor capability. The results before and after demosaicing are respectively called the Bayer domain and the RGB domain.

[0032] Figure 1The figure shows a flowchart of a combined processing method of demosaicing and sharpening. The combined processing method of demosaicing and sharpening mainly includes the following steps:

[0033] Step 101, processing the original data of the target color channel of the image sensor (Sensor) to obtain a full-resolution target color channel layer.

[0034] In some embodiments, the target color channel is a color channel with the highest confidence among the color channels of the image sensor.

[0035] In some embodiments, when the format of the raw data output by the image sensor is RGGB format, the target color channel is the G channel, the R (Red) represents red, the G (Green) represents green, and the B (Blue) represents blue;

[0036] When the format of the raw data output by the image sensor is the RCCB format, the target color channel is the C channel, the R represents red, the C (Clear) represents white, and the B represents blue;

[0037] When the format of the raw data output by the image sensor is the RYYCy format, the target color channel is the Y channel, the R represents red, the Y (Yellow) represents yellow, and the Cy (Cyan) represents cyan.

[0038] Specifically, the Bayer formats supported by the sensor generally include RGGB, RCCB, RYYCy, etc., among which RGGB can also be transformed into BGGR, GRBG, GBRG. To obtain a correct color picture, the sensor needs to support the collection of information of at least 3 colors. The method proposed in the embodiment of the present disclosure not only supports the RGGB format, but also supports the formats such as RCCB and RYYCy. For example, the C and Y channels in RCCB and RYYCy can be treated as equivalent to the G channel in the RGGB format.

[0039] In some embodiments, the raw data of the target color channel acquired by the image sensor is processed to obtain a full-resolution target color channel layer, including: based on the target color pixels in the raw data of the target color channel, pixel interpolation is performed on pixel positions where the target color does not exist in the raw data of the target color channel to obtain the full-resolution target color channel layer.

[0040] It should be noted that there is no limitation on the specific interpolation method used for pixel interpolation, and a variety of interpolation methods may be used for pixel interpolation, for example, the undirected nearest neighbor method, direction or gradient-based interpolation, or anisotropic directional interpolation, etc.

[0041] In some embodiments, based on the target color pixels in the original data of the target color channel, pixel interpolation is performed on the pixel positions where the target color does not exist in the original data of the target color channel to obtain the full-resolution target color channel layer, including:

[0042] Any pixel that does not have the target color in the original data of the target color channel is taken as the target pixel, and the following interpolation process is performed: the pixel coordinates of the target pixel in the original data of the target color channel are obtained, and the adjacent target color pixels of the target pixel are obtained from the original data of the target color channel according to the pixel coordinates; the horizontal gradient value and the vertical gradient value are determined according to the adjacent target color pixels of the target pixel; and the target pixel is directional interpolated according to the horizontal gradient value and the vertical gradient value to obtain the target color channel pixel value corresponding to the target pixel.

[0043] Wherein, the horizontal gradient value and the vertical gradient value are determined according to the adjacent target color pixels of the target pixel, including: taking the difference between the adjacent target color pixel on the left side of the target pixel and the adjacent target color pixel on the right side as the horizontal gradient value of the target pixel; taking the difference between the adjacent target color pixel above the target pixel and the adjacent target color pixel below the target pixel as the vertical gradient value of the target pixel.

[0044] Wherein, according to the horizontal gradient value and the vertical gradient value, the target pixel is directional interpolated to obtain the target color channel pixel value corresponding to the target pixel, including: when the absolute value of the difference between the horizontal gradient value and the vertical gradient value is less than or equal to a preset threshold value, it indicates that the target pixel is in a flat non-directional area, and the average value of the target color pixel adjacent to the left, the target color pixel adjacent to the right, the target color pixel adjacent above, and the target color pixel adjacent below the target pixel is used as the target color channel interpolation corresponding to the target pixel; when the absolute value of the difference between the horizontal gradient value and the vertical gradient value is greater than a preset threshold value, and the horizontal gradient value is less than the vertical gradient value, the average pixel value of the target color pixel adjacent to the left and the target color pixel adjacent to the right of the target pixel is obtained, and the average pixel value is used as the target color channel pixel value obtained by interpolation of the target pixel; when the absolute value of the difference between the horizontal gradient value and the vertical gradient value is greater than the preset threshold value, and the horizontal gradient value is greater than the vertical gradient value, the average pixel value of the target color pixel adjacent to the upper and the lower target color pixels of the target pixel is obtained, and the average pixel value is used as the target color channel pixel value obtained by interpolation of the target pixel.

[0045] In an exemplary embodiment, Figure 2The figure shows an example of RGGB color filter array (CFA), where the left side is a schematic diagram of RGGB format and the right side is the corresponding coordinate index. Taking the G channel interpolation of the 4 pixels in the bold rectangle as an example, an example of G channel interpolation using line buffer = 3 is as follows:

[0046] In order to calculate the G channel pixel value G22 at the R22 position, the horizontal gradient diff_H and the vertical gradient diff_V are first calculated, as shown below:

[0047] diff_H = Gr21-Gr23;

[0048] diff_V = Gb12 - Gb32;

[0049] Directional interpolation is performed according to the neighborhood gradient value, which is expressed as follows:

[0050] G22=(Gr21+Gr23+Gb12+Gb32) / 4, if abs(diff_H-diff_V)<=th;

[0051] G22=(Gr21+Gr23) / 2, if abs(diff_H-diff_V)>th, and diff_H <diff_V;

[0052] G22=(Gb12+Gb32) / 2, if abs(diff_H-diff_V)>th, and diff_H>diff_V;

[0053] The G channel interpolation in other bold rectangles can be expressed as:

[0054] G23 = Gr23;

[0055] G32 = Gb32;

[0056] The calculation method of G33 and G22 is the same principle;

[0057] Among them, th is a pre-configured threshold value. If diff_H and diff_V are close to, that is, less than or equal to, th, that is, the current pixel is in a flat and directionless area, multiple G channels in the neighborhood can be used for average interpolation. abs(.) represents the absolute value. The larger the value of line buffer, the more accurate the G channel interpolation result is, but the more resources are consumed. Here, only linebuffer=3 is used as an example for explanation.

[0058] Step 102, sharpening the full-resolution target color channel layer to obtain a sharpened layer.

[0059] The high-frequency detail layer, i.e., the sharpened layer, is obtained by sharpening the full-resolution target color channel layer. For example, a full-resolution G channel layer (map) is obtained, and a high-frequency detail layer Gs is obtained based on the G map.

[0060] In the embodiments of the present disclosure, there are multiple methods for sharpening the full-resolution target color channel layer, which are not limited to a specific sharpening method. For example, the sharpening methods that can be used include Laplacian filters, multi-band high-pass filters, etc., or edge-preserving filters such as bilateral filters, guided filters, etc.

[0061] In an exemplary embodiment, a full-resolution target color channel layer is sharpened as a guide layer to obtain a sharpened layer, including: performing a two-dimensional convolution operation on the full-resolution target color channel image and the convolution kernel of a Laplacian filter to obtain a high-frequency detail layer of the full-resolution target color channel image; and adjusting the high-frequency detail layer using a preset coefficient to obtain a sharpened layer.

[0062] For example, assuming that the target color channel is the G channel, an example of a method for sharpening using a 3x3 Laplacian filter is as follows: First, define the 3x3 Laplacian filter convolution kernel as:

[0063] kernel_laplacian=

[0064] [-1 -1-1],

[0065] [-1 8 -1],

[0066] [-1 -1-1]

[0067] The basic high-frequency detail layer Gs_b can be obtained by performing a two-dimensional convolution of the G map and the Laplacian filter, which can be expressed as:

[0068] Gs_b=conv2d(G,kernel_laplacian)

[0069] Where conv2d(,) means performing a two-dimensional convolution operation on two matrices.

[0070] The sharpening intensity of the basic high-frequency detail layer Gs_b is adjusted by the coefficient alpha, which is expressed as:

[0071] Gs = alpha * Gs_b;

[0072] The high-frequency detail layer Gs obtained above can be used as a guide map to guide other R and B channels for sharpening.

[0073] In addition, other filters can be used for sharpening, such as other high-pass and band-pass filters, or filters with larger line buffers, such as 5x5, 7x7, etc. In other words, the common operations for sharpening in the YUV domain can be used to optimize the detail layer Gs. For other types of target color channels, a similar processing method to the G channel is used to obtain a sharpened layer, which will not be repeated here.

[0074] Step 103, interpolating the original data of other color channels of each color channel of the image sensor except the target color channel to demosaic, and sharpening the full-resolution target color channel layer and the other color channel layers obtained by demosaicing by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

[0075] In some embodiments, interpolating raw data of other color channels of the image sensor except the target color channel to demosaic includes:

[0076] Each of the other color channels is respectively used as a designated color channel, and each pixel of the designated color channel is respectively used as a designated pixel to perform the following interpolation process: when the designated pixel has a corresponding designated color pixel value, the designated pixel is directly interpolated using the designated color pixel value; when the designated pixel does not have a corresponding designated color pixel value, according to the pixel position where the designated pixel is located, the first pixel value of the designated color channel corresponding to the adjacent pixel is obtained, and the target color pixel value of the pixel position where the designated pixel is located and the second pixel value of the target color channel corresponding to the adjacent pixel are obtained to interpolate the designated pixel.

[0077] In some embodiments, the full-resolution target color channel layer and other color channel layers obtained by de-mosaicing are sharpened by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates, including: when the sharpening enable function is turned on, on the basis of the pixel values ​​of the same pixel coordinates of the full-resolution target color channel layer and other color channel layers obtained by de-mosaicing, the sharpening values ​​corresponding to the pixel coordinate positions in the sharpening layer are respectively superimposed for sharpening.

[0078] In the exemplary embodiment, Figure 2 Taking the interpolation of the R and B channels of the four pixels in the bold rectangle as an example, a feasible solution to maintain color consistency before and after sharpening is enabled is given as follows:

[0079] First, define enable_sharpen as the sharpening enable switch. enable_sharpen=0 means disabled, and enable_sharpen=1 means enabled. Take line buffer=3 as an example:

[0080] For R channel interpolation, if enable_sharpen = 0, there is

[0081] R22=R22;

[0082] R23=(R22-G22) / 2+(R24-G24) / 2+G23;

[0083] R32=(R22-G22) / 2+(R42-G42) / 2+G32;

[0084] R33=(R22-G22) / 4+(R24-G24) / 4+(R42-G42) / 4+(R44-G44) / 4+G33;

[0085] For R channel interpolation, if enable_sharpen = 1, there is

[0086] R22_s=R22+Gs22;

[0087] R23_s=(R22-G22) / 2+(R24-G24) / 2+G23+Gs23;

[0088] R32_s=(R22-G22) / 2+(R42-G42) / 2+G32+Gs32;

[0089] R33_s=(R22-G22) / 4+(R24-G24) / 4+(R42-G42) / 4+(R44-G44) / 4+G33+Gs33;

[0090] Among them, [R22_s, R23_s, R32_s, R33_s] represents the demosaic result after sharpening is enabled at the corresponding position.

[0091] For B channel interpolation, if enable_sharpen = 0, there is

[0092] B22=(B11-G11) / 4+(B13-G13) / 4+(B31-G31) / 4+(B33-G33) / 4+G22;

[0093] B23=(B13-G13) / 2+(B33-G33) / 2+G32;

[0094] B32=(B31-G31) / 2+(B33-G33) / 2+G32;

[0095] B33=B33;

[0096] For B channel interpolation, if enable_sharpen = 1, there is

[0097] B22_s=(B11-G11) / 4+(B13-G13) / 4+(B31-G31) / 4+(B33-G33) / 4+G22+Gs22;

[0098] B23_s=(B13-G13) / 2+(B33-G33) / 2+G32+Gs23;

[0099] B32_s=(B31-G31) / 2+(B33-G33) / 2+G32+Gs32;

[0100] B33_s=B33+Gs33;

[0101] Among them, [B22_s, B23_s, B32_s, B33_s] represents the demosaic result after sharpening is enabled at the corresponding position.

[0102] For the G channel, if enable_sharpen=0, each pixel value in the full-resolution G channel layer obtained in step 101 is the G channel interpolation value;

[0103] For the G channel, if enable_sharpen = 1, there is

[0104] G22_s=G22+Gs22;

[0105] G23_s=G23+Gs23;

[0106] G32_s=G32+Gs32;

[0107] G33_s=G33+Gs33;

[0108] If a pixel is [R, G, B] before sharpening and [Rs, Gs, Bs] after sharpening, it satisfies delta_s=Rs-R=Gs-G=Bs-B, that is, the sharpening increment value delta_s of the three channels R, G and B remains consistent, satisfying the color difference constancy theory, which means that the consistency of the color space can be guaranteed.

[0109] According to the joint processing method of demosaicing and sharpening provided in the embodiment of the present disclosure, after demosaicing and sharpening in the Bayer domain, the color space is converted from the RGB domain to the YUV domain, and the consistency of the color space can be ensured. The specific reasoning process is as follows:

[0110] According to the ITU Radiocommunication Assembly (ITU-R) standard, the commonly used RGB to YUV color conversion standards and the corresponding color conversion matrix are as follows. The following representation retains only 3 significant digits:

[0111] BT601 Standard:

[0112] [0.299,0.587,0.114],

[0113] [0.5,-0.418,-0.081],

[0114] [-0.168,-0.331,0.5],

[0115] BT709 Standard:

[0116] [0.212,0.715,0.072],

[0117] [0.5,-0.454,-0.045],

[0118] [-0.114,-0.385,0.5],

[0119] BT2020 Standard:

[0120] [0.262,0.6780,0.0593],

[0121] [0.5,-0.459,-0.04],

[0122] [-0.139,-0.360,0.5],

[0123] YCrCbJPEG standard:

[0124] [0.299,0.587,0.114],

[0125] [0.499,-0.4181,-0.081],

[0126] [-0.168,-0.331,0.499],

[0127] The above color conversion matrix can convert the image signal from the RGB color space to the corresponding YUV color space, also known as the YCrCb color space. For example, [Y,Cr,Cb] = M*[R,G,B], where [Y,Cr,Cb] and [R,G,B] represent 3×1 vectors, representing the image signal before and after the color space conversion, respectively, and the matrix M represents the 3x3 color space conversion matrix.

[0128] The common feature of the above color space conversion matrices is that the sum of the second and third rows of the matrix is ​​approximately zero. Taking the BT601 standard as an example, the Cr and Cb values ​​of the color after Sharpen can be expressed as:

[0129] Cr=0.5*Rs-0.418*Gs-0.081*Bs

[0130] =0.5*(R+delta_s)-0.418*(G+delta_s)-0.081*(B+delta_s)

[0131] =0.5*R-0.418*G-0.081*B;

[0132] Cb=-0.168*Rs-0.331*Gs+0.5*Bs

[0133] =-0.168*(R+delta_s)-0.331*(G+delta_s)+0.5*(B+delta_s)

[0134] =-0.168*R-0.331*G+0.5*B;

[0135] From the above derivation process, it can be seen that satisfying the condition that the sum of the second and third rows of the color space conversion matrix is ​​approximately zero can ensure that the colors Cr and Cb remain consistent before and after Sharpen is enabled.

[0136] Taking BT601 and pixel position R22 as an example, the above interpolation method can maintain color constancy as follows:

[0137] For enable_sharpen = 0, there are

[0138] Cr=0.5*R22-0.418*G22-0.081*B22;

[0139] For enable_sharpen = 1, there are

[0140] Cr_s=0.5*R22_s-0.418*G22_s-0.081*B22_s

[0141] =0.5*(R22+Gs22)-0.418*(G22+Gs22)-0.081*(B22+Gs22)

[0142] =0.5*R22-0.418*G22-0.081*B22;

[0143] Where Cr and Cr_s represent the colors with Sharpen disabled and enabled, and it is obvious that Cr = Cr_s. The same can be said for Cb color and other pixel positions.

[0144] The above proves that the method proposed in the embodiment of the present disclosure can ensure that the color component CbCr in the YCbCr domain (ie, the YUV domain) remains consistent before and after sharpening. For the sharpening effect analysis, take the BT601 standard as an example:

[0145] When enable_sharpen=0, there is

[0146] Y=0.299*R+0.587*G+0.114*B;

[0147] When enable_sharpen=1, there is

[0148] Y_s=0.299*Rs+0.587*Gs+0.114*Bs+Gs;

[0149] Among them, [R, G, B] represents the result after demodulation without sharpening enabled, [Rs, Gs, Bs] represents the result after sharpening and demodulation enabled, Y represents the brightness channel value without sharpening enabled, and Y_s represents the brightness channel value after sharpening enabled. It can be seen that the intensity of the brightness channel Y_s is adjusted by G_s, which has a sharpening enhancement effect.

[0150] It should be noted that the demosaic interpolation method based on color difference consistency is only an optional implementation method. Other implementation methods that use a sharpening layer as a guide layer and add the same sharpening value as a sharpening increment to the color channels other than the target color channel during the demosaic interpolation process for different color channel values ​​of the same pixel coordinate are also within the protection scope of the embodiments of the present disclosure. For example, other methods that use a Gs map as a guide map and then add the same value Gs as a Sharpen increment to the R and B channels during the demosaic color difference interpolation process include using a larger line buffer or a more accurate directional calculation method during the interpolation process of the R channel and the B channel.

[0151] In the above embodiments, for other image sensor Bayer formats such as RCCB, RYYCy, etc., the G channel in the described RGGB format processing method is replaced by the C channel or the Y channel, and the B channel is replaced by the Cy channel. Similarly, the processing process of the RGGB format is also applicable to the variant formats BGGR, GRBG, GBRG, and the processing processes of RCCB and RYYCy are also applicable to the variant formats.

[0152] The method provided in the embodiments of the present disclosure can be used in camera products or ISP IP or ISP products of chip manufacturers, in scenarios where de-mosaicing and sharpening are jointly processed by analyzing the Bayer domain. For example, it can be used in application fields such as autonomous driving that are highly sensitive to image processing pipeline latency and complexity, or in other machine vision fields such as security, mobile phones, and extended reality (XR) industries.

[0153] In the disclosed embodiment, a full-resolution target color channel layer is obtained based on the original data of the target color channel of the image sensor, and a sharpened layer is obtained by sharpening the full-resolution target color channel layer. The original data of other color channels except the target color channel in each color channel of the image sensor are interpolated to demosaic, and the full-resolution target color channel layer and other color channel layers obtained by demosaicing are sharpened by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates, so that each color channel is sharpened by the sharpening layer to achieve the purpose of adaptive sharpening, realize the joint processing of demosaicing and sharpening, and pre-position the sharpening to the Bayer domain. By superimposing the same sharpening layer on each color channel, the consistency of color before and after sharpening is guaranteed while reducing the system delay.

[0154] In the disclosed embodiment, for time-delay-sensitive machine vision applications such as autonomous driving, the traditional sharpening operation processed in the YUV domain is effectively embedded in the Bayer domain, and efficiently combined with the Bayer domain demosaicing module, while supporting adaptive adjustment of image detail significance, greatly reducing system complexity and delay overhead. It can also ensure the consistency of image color before and after sharpening is enabled, and by efficiently using the high-frequency detail layer as a guide map, the upsampling and sharpening of the image are both performed in the Bayer domain, without the need for complex domain conversion operations.

[0155] It is proved through analysis that the method provided by the embodiment of the present disclosure not only effectively improves the contrast of high-frequency details of the image, but also theoretically proves that the consistency of Cr and Cb colors is ensured when the sharpening function is turned on or off.

[0156] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this disclosure. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this disclosure.

[0157] Based on the same concept, a digital image signal processor is also provided in an embodiment of the present disclosure, for executing the joint processing method of de-mosaicing and sharpening described above.

[0158] An embodiment of the present disclosure provides a combined processing device for de-mosaicing and sharpening. The specific implementation of the device can be found in the relevant description of the method embodiment, which will not be repeated here. Figure 3 The schematic diagram of the structure of the device is shown, which mainly includes:

[0159] The acquisition module 301 is used to process the raw data of the target color channel of the image sensor to obtain a full-resolution target color channel layer;

[0160] A sharpening module 302, used for sharpening the full-resolution target color channel layer to obtain a sharpened layer;

[0161] The processing module 303 is used to interpolate the original data of other color channels of each color channel of the image sensor except the target color channel to demosaic, and sharpen the full-resolution target color channel layer and the other color channel layers obtained by demosaicing by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

[0162] The functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the method embodiments above, and will not be described again here for the sake of brevity.

[0163] It should be noted that all modules involved in this embodiment are logic modules. In practical applications, a logic unit may be a physical unit, or a part of a physical unit, or may be implemented as a combination of multiple physical units. In addition, in order to highlight the innovative part of the present disclosure, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by the present disclosure, but this does not mean that there are no other units in this embodiment.

[0164] Reference Figure 4 , an embodiment of the present disclosure provides an electronic device, comprising:

[0165] at least one processor 401;

[0166] A memory 402 having at least one program stored thereon, and when the at least one program is executed by the at least one processor, the at least one processor implements the above method;

[0167] At least one I / O interface 403 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.

[0168] Among them, the processor 401 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 402 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 403 is connected between the processor 401 and the memory 402, and can realize information interaction between the processor 401 and the memory 402, including but not limited to a data bus (Bus), etc.

[0169] In some embodiments, the processor 401 , the memory 402 , and the I / O interface 403 are connected to each other through a bus, and further connected to other components of the computing device.

[0170] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.

[0171] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods applied for above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0172] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0173] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0174] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A combined processing method of demosaicing and sharpening, characterized in that: include: Processing the raw data of the target color channel of the image sensor to obtain a full-resolution target color channel layer; Sharpening the full-resolution target color channel layer to obtain a sharpened layer; The original data of other color channels of each color channel of the image sensor except the target color channel are interpolated to demosaic, and the full-resolution target color channel layer and the other color channel layers obtained by demosaicing are sharpened by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

2. The method according to claim 1, characterized in that The target color channel is the color channel with the highest confidence among the color channels of the image sensor.

3. The method according to claim 1, characterized in that The processing of the original data of the target color channel acquired by the image sensor to obtain a full-resolution target color channel layer includes: Based on the target color pixels in the original data of the target color channel, pixel interpolation is performed on pixel positions where the target color does not exist in the original data of the target color channel to obtain the full-resolution target color channel layer.

4. The method according to claim 1, characterized in that: The interpolating original data of other color channels of the image sensor except the target color channel to perform demosaicing includes: Each of the other color channels is used as a designated color channel, and each pixel of the designated color channel is used as a designated pixel to perform the following interpolation process: When the designated pixel has a corresponding designated color pixel value, directly interpolating the designated pixel using the designated color pixel value; When there is no corresponding designated color pixel value for the designated pixel, the first pixel value of the designated color channel corresponding to the adjacent pixel is obtained according to the pixel position of the designated pixel, and the target color pixel value of the pixel position of the designated pixel and the second pixel value of the target color channel corresponding to the adjacent pixel are obtained to interpolate the designated pixel.

5. The method according to claim 1, characterized in that The sharpening of the full-resolution target color channel layer and other color channel layers obtained by demosaicing by superimposing the sharpening values ​​of corresponding positions of the sharpening layer according to pixel coordinates includes: When the sharpening enable function is turned on, based on the pixel values ​​of the same pixel coordinates of the full-resolution target color channel layer and other color channel layers obtained by de-mosaicing, the sharpening values ​​corresponding to the pixel coordinate positions in the sharpening layer are respectively superimposed to perform sharpening.

6. The method according to claim 1, characterized in that When the format of the raw data output by the image sensor is RGGB format, the target color channel is the G channel, R represents red, G represents green, and B represents blue; When the format of the raw data output by the image sensor is the RCCB format, the target color channel is the C channel, the R represents red, the C represents white, and the B represents blue; In the case that the format of the raw data output by the image sensor is the RYYCy format, the target color channel is the Y channel, the R represents red, the Y represents yellow, and the Cy represents cyan.

7. A digital image signal processor, characterized in that: Used to perform the method according to any one of claims 1 to 6.

8. A combined processing device for demosaicing and sharpening, characterized in that: include: An acquisition module is used to process the raw data of the target color channel of the image sensor to obtain a full-resolution target color channel layer; A sharpening module, used for sharpening the full-resolution target color channel layer to obtain a sharpened layer; The processing module is used to interpolate the original data of other color channels of each color channel of the image sensor except the target color channel to demosaic, and sharpen the full-resolution target color channel layer and the other color channel layers obtained by demosaicing by superimposing the sharpening values ​​of the corresponding positions of the sharpening layer according to the pixel coordinates.

9. An electronic device, characterized in that: include: at least one processor; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6; At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

10. A computer readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images