Image processing methods, image processing devices and chips

Abstract

This application provides an image processing method, an image processing apparatus, and a chip. The method includes: obtaining Hexa-deca data based on a Hexa-deca configuration color filter array, wherein the Hexa-deca data includes G pixels and non-G pixels; determining a plurality of first interpolation positions and a plurality of second interpolation positions among the non-G pixel positions; performing a first interpolation process on the first interpolation positions according to a pre-trained interpolation filter to obtain G pixel values ​​for the plurality of first interpolation positions; and performing a second interpolation process on the second interpolation positions according to the G pixel values ​​for the plurality of first interpolation positions to generate a full-resolution G image. This application embodiment can obtain a full-resolution G-channel image of a Hexa-deca configuration CFA through interpolation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to image processing methods, image processing apparatus and chips. Background Technology

[0002] Image sensors are used to capture images and typically employ CCD or CMOS image sensors covered with a Color Filter Array (CFA). Common configurations of CFAs include Bayer, Quad Bayer, RGBW, and Hexa-deca. Hexa-deca configurations can improve imaging capabilities and signal-to-noise ratio for specific scenes by using pixel binning (4-to-1 or 16-to-1) of adjacent pixels. However, the Hexa-deca configuration suffers from difficulty in acquiring full-resolution G-channel images. Therefore, an effective method for acquiring full-resolution G-channel images with the Hexa-deca configuration is urgently needed. Summary of the Invention

[0003] This application provides an image processing method, an image processing device, and a chip that can obtain a full-resolution G-channel image of a Hexa-deca configuration CFA through an effective interpolation method, and then perform de-mosaic processing on the CFA of this configuration.

[0004] A first aspect of this application provides an image processing method, the method comprising: obtaining Hexa-deca data based on a Hexa-deca configuration color filter array, the Hexa-deca data including G pixels and non-G pixels; determining a plurality of first interpolation positions and a plurality of second interpolation positions among the non-G pixel positions; performing a first interpolation process on the first interpolation positions according to a pre-trained interpolation filter to obtain G pixel values ​​of the plurality of first interpolation positions; and performing a second interpolation process on the second interpolation positions according to the G pixel values ​​of the plurality of first interpolation positions to generate a full-resolution G image.

[0005] A second aspect of this application provides an image processing method, further comprising: downsampling the full-resolution G image and the Hexa-deca data to generate a half-resolution G image and a half-resolution RGB image, respectively; performing guided interpolation on the half-resolution RGB image based on the half-resolution G image to generate a half-resolution R image and a B image; and performing guided upsampling on the half-resolution R image and B image based on the full-resolution G image to generate a full-resolution RGB image.

[0006] A third aspect of this application provides an image processing apparatus, characterized in that it includes: an acquisition module, configured to acquire Hexa-deca data according to a Hexa-deca configuration color filter array, wherein the Hexa-deca data includes G pixels and non-G pixels; a determination module, configured to determine a plurality of first interpolation positions and a plurality of second interpolation positions among the non-G pixel positions; a first interpolation module, configured to perform a first interpolation process on the first interpolation positions according to a pre-trained interpolation filter to obtain G pixel values ​​of the plurality of first interpolation positions; and a second interpolation module, configured to perform a second interpolation process on the second interpolation positions according to the G pixel values ​​of the plurality of first interpolation positions to generate a full-resolution G image.

[0007] A fourth aspect of this application provides a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is mounted to perform the image processing method.

[0008] A fourth aspect of this application provides an image processing apparatus, characterized in that it includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the image processing method.

[0009] This application addresses the issue that the R, G, and B pixel distributions obtained from a Hexa-deca configuration color filter array are relatively concentrated and uneven. It proposes determining multiple first interpolation positions and multiple second interpolation positions within the non-G pixel locations. First, the first interpolation positions undergo a first interpolation process based on a pre-trained interpolation filter. Then, the second interpolation positions undergo a second interpolation process with reference to the interpolated pixel values ​​of the first interpolation positions. This staged interpolation method effectively obtains a full-resolution G-channel image without complex detection modules, reducing algorithm complexity, image processor consumption, and improving computational speed. Furthermore, the full-resolution G-channel image allows for de-mosaic processing of the CFA of this configuration to obtain a full-resolution RGB color image. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments conforming to this application and, together with the specification, serve to explain the technical solutions of this application. Some specific embodiments of this application will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings denote the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale.

[0011] Figure 1A schematic diagram of a Bayer mode filter array provided for an embodiment of this application;

[0012] Figure 2 This is a schematic diagram of the pixel array corresponding to the Hexa-deca configuration color filter array in the embodiments of this application;

[0013] Figure 3 A flowchart illustrating the steps of an image processing method provided in this application embodiment;

[0014] Figure 4 A flowchart of step S2 in an image processing method provided in an embodiment of this application;

[0015] Figure 5 This is a schematic diagram of the interpolation process of pixels R2 and R3 in the Hexa-deca Bayer CFA of this application embodiment;

[0016] Figure 6 A flowchart of step S3 in an image processing method provided in this application embodiment;

[0017] Figure 7 This is a schematic diagram illustrating the quantization direction in an embodiment of this application;

[0018] Figure 8 This is a schematic diagram of a 7×7 filter for R3 and R2 pixels trained under the conditions of direction level 6 (i.e., 135-degree direction), intensity level 2, and consistency level 2 in an embodiment of this application.

[0019] Figure 9 This is a schematic diagram of the interpolation process of pixel R1 in the Hexa-deca Bayer CFA of this application embodiment;

[0020] Figure 10 A flowchart illustrating step S4 in an image processing method provided in this application embodiment;

[0021] Figure 11 A flowchart of the demosaic algorithm for Hex-decaBayerCFA in the image processing method provided in the embodiments of this application;

[0022] Figure 12 This is a schematic diagram illustrating the complete demosaic algorithm flow for Hexa-deca BayerCFA in an embodiment of this application;

[0023] Figure 13 This is a schematic diagram of the process of using the G channel to guide the interpolation of the R / B channel after downsampling by two times according to the embodiments of this application;

[0024] Figure 14 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0025] Figure 15 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Various details are set forth in this application because these details relate to certain embodiments. However, this application may also be implemented in a manner different from that described herein. Modifications to the discussed embodiments can be made by those skilled in the art without departing from this application. Therefore, this application is not limited to the specific embodiments disclosed herein.

[0027] The image processing method provided in this application is applicable to processing RGB images acquired by an image sensor. To facilitate understanding of the technical solution provided in this application, the image sensor (or pixel array) will be briefly described below.

[0028] Each pixel in a pixel array has a similar structure. Typically, each pixel's structure includes a microlens, a filter, and a photosensitive element. The microlens is located above the filter, which in turn is located above the photosensitive element. Light returning from the image object is focused by the microlens, filtered by the filter, and then enters the photosensitive element, where it converts the light signal into an electrical signal. Depending on the type of light that different filters can transmit, the pixel array can include red pixels (hereinafter referred to as R pixels), blue pixels (hereinafter referred to as B pixels), and green pixels (hereinafter referred to as G pixels). For example, an R pixel refers to a pixel that, after being filtered by the filter, only allows red light to enter the photosensitive element, where it is converted into an electrical signal. The principles of B pixels and G pixels are similar to those of R pixels and will not be elaborated upon here.

[0029] Since each pixel in a pixel array can only convert one type of light signal into an electrical signal, interpolation is required by combining the light signals collected by surrounding pixels of other types to restore the image color of the area captured by that pixel. This process is called demosaicing, and it is usually performed in the image processor at the back end of the image sensor or in the processor of the electronic device containing the image sensor (hereinafter collectively referred to as the processor).

[0030] To acquire color images, filters with specific color arrangements need to be set in the pixel array; this is also called a color filter array (CFA). There are various types of CFAs, but electronic devices typically use... Figure 1The Bayer pattern filter array shown consists of three types of filters that transmit specific wavelengths of red (R), green (G), and blue (B) at a sampling rate interval of 1:2:1. However, with the miniaturization of image sensors, the size of sensor pixels is getting smaller and smaller. In order to improve the signal-to-noise ratio and sensitivity, as well as the sensor's imaging capability in low-light environments at night, more and more CFAs are adopting a multi-pixel binning approach.

[0031] Figure 2 This is a schematic diagram of the pixel array corresponding to the color filter array of the Hexa-deca configuration. The Hexa-deca CFA includes an 8×8 minimum repeating unit. Each minimum repeating unit comprises: a 4×4 square R-cluster formed by 16 red filter units, a first 4×4 square G-cluster formed by 16 green (G) filter units, a second 4×4 square G-cluster formed by 16 G filter units, and a 4×4 square B-cluster formed by 16 blue (B) filter units. The 4×4 square R-clusters and 4×4 square B-clusters are diagonally adjacent, the first 4×4 square G-clusters and 4×4 square B-clusters are horizontally adjacent, and the second 4×4 square G-clusters and the first 4×4 square G-clusters are diagonally adjacent. In this minimum repeating unit, green pixels occupy 1 / 2, and red and blue pixels each occupy 1 / 4. This CFA can use adjacent pixel binning (4-to-1 or 16-to-1) to improve the imaging capability and signal-to-noise ratio for specific scenes, while also possessing the ability to generate full-resolution images.

[0032] Since the Hexa-deca Bayer CFA is a novel filter array, there is currently no demosaic algorithm specifically for it. However, numerous demosaic algorithms exist for Bayer CFAs. These algorithms typically use interpolation to obtain a full-resolution G-channel image, which is then used to guide the generation of R and B-channel images. The demosaic algorithm proposed in this application for the Hexa-deca Bayer CFA faces challenges in obtaining a full-resolution G-channel image under the Hexa-deca configuration due to the relatively concentrated and uneven pixel distribution.

[0033] Based on this, embodiments of this application provide an image processing method that can effectively obtain a full-resolution G-channel image through a staged interpolation method, and then use the G-channel to guide the generation of full-resolution R and B-channel images to achieve de-mosaic processing of the Hexa-deca configuration CFA.

[0034] See Figure 3 The image processing method includes:

[0035] Step S1: Obtain Hexa-deca data based on the Hexa-deca configuration color filter array, wherein the Hexa-deca data includes G pixels and non-G pixels;

[0036] Hexa-deca data is a full-resolution image obtained from a pixel array. For example, the pixel array consists of m rows and n columns of pixels, and Hexa-deca data is a collection of information acquired for each pixel. In a Hexa-deca configuration color filter array, 50% of the pixels are green (G) and 50% are non-green (non-G) pixels. For each non-G pixel, its G pixel value is calculated using interpolation based on the pixel values ​​of its surrounding G pixels. This yields the G value for each pixel in the pixel array, thus generating a full-resolution G image.

[0037] Step S2: Determine multiple first interpolation positions and multiple second interpolation positions in the non-G pixel positions;

[0038] The interpolation position is the pixel position where the green (G) pixel value to be interpolated is to be interpolated, i.e., the non-G pixel position is the interpolation position, and the specific color to be interpolated is green (G). In one implementation, multiple first interpolation positions and multiple second interpolation positions are determined among the non-G pixel positions, and interpolation calculations are performed at the first interpolation positions and the second interpolation positions using different interpolation calculation directions.

[0039] Step S3: Based on the pre-trained interpolation filter, perform a first interpolation process on the first position to be interpolated to obtain multiple G pixel values ​​of the first position to be interpolated.

[0040] Specifically, the interpolation filter is trained using the least squares method, and the interpolation filter model contains the correspondence between the feature information of different image patches and different filter kernels. Each filter in the interpolation filter model contains interpolation weights applied to each pixel of the corresponding image patch when interpolating the pixel value of G at the first interpolation position.

[0041] Step S4: Based on the G pixel values ​​of multiple first interpolation positions, perform a second interpolation process on the second interpolation position to generate a full-resolution G image;

[0042] Specifically, after interpolating a pixel value of G at the first interpolation position, using this interpolated pixel value to perform interpolation processing at the second interpolation position provides more reference information, making interpolation easier and more accurate. Furthermore, the pixel value of G at the second interpolation position can be obtained through color difference interpolation or gradient-weighted interpolation, eliminating the need for various complex detection modules and reducing algorithm complexity.

[0043] In a specific implementation of an embodiment of this application, see [link to relevant documentation]. Figure 4 Step S2 further includes:

[0044] Step S21: The non-G pixel positions include a first type of position, a second type of position, and a third type of position, wherein the first type of position is adjacent to two G pixels on all four sides, the second type of position is adjacent to one G pixel on all four sides, and the third type of position is not adjacent to any G pixels on all four sides.

[0045] Since G-channel interpolation mainly requires calculating the G values ​​on R and B, and since R and B are positionally equivalent, this embodiment will use R-pixel interpolation for G value interpretation. Similarly, B-pixel interpolation can also be used as an example. This embodiment is not intended to limit the scope of this application. Specifically, as... Figure 5 The diagram illustrates the interpolation process for pixels R2 and R3 in a Hexa-deca Bayer CFA. The first pixel array on the left is a schematic diagram of an 8×8 region pixel array centered on a 4×4 R pixel block. As indicated by the markings in the diagram, R pixels can be divided into three pixel types: the first type, indicated by position (R1, r1), is denoted as the first type position, and is adjacent to two G pixels on all four sides; the second type, indicated by position (R2, r2), is denoted as the second type position, and is adjacent to one G pixel on all four sides; the third type, indicated by position (R3, r3), is denoted as the third type position, and is not adjacent to any G pixels on all four sides, but is surrounded by R pixels. Similarly, B pixels can also be divided into the above three pixel types.

[0046] Furthermore, since pixel r1 can be transformed to position R1 by flipping the image block horizontally or vertically, pixel r2 can be transformed to position R2 by flipping horizontally or vertically and rotating clockwise or counterclockwise, and pixel r3 can be transformed to position R3 by flipping the image block horizontally or vertically, this embodiment will only describe the interpolation of pixels at positions R1, R2, and R3. Pixels at other positions can be flipped and rotated first, and then interpolated.

[0047] Step S22: Determine the second type of position and the third type of position as the plurality of first interpolation positions;

[0048] Specifically, the pixel positions (R2,r2) and (R3,r3) in the 4×4 R-pixel image block are determined as the first interpolation positions and interpolation calculations are performed to obtain the interpolated pixel G value of the first interpolation position.

[0049] Step S23: Determine the first type of position as the plurality of second interpolation positions.

[0050] Specifically, the pixel position (R1, r1) in a 4×4 R-pixel image block is determined as the second interpolation position. Referring to the interpolated pixel G value of the first interpolation position and the original pixel value, interpolation is performed again on the second interpolation position to obtain its interpolated pixel G value. This yields the G pixel values ​​for all non-G pixel points. Furthermore, after interpolating the first interpolation position with its G pixel value, the interpolation process for the second interpolation position has more reference information, making interpolation easier and more accurate.

[0051] In another specific implementation of the embodiments of this application, see [link to relevant documentation]. Figure 6 Step S3 further includes:

[0052] Step S31: Based on each of the first interpolation locations, determine the gradient image block corresponding to the first interpolation location;

[0053] As described above, the pixel positions (R2,r2) and (R3,r3) in the 4×4 R pixel image block are determined as the first interpolation positions. Interpolation calculation is performed first to obtain the interpolated pixel G value of the first interpolation position.

[0054] Firstly, for the second type of location (R2, r2), determining the gradient image patch corresponding to each of the first interpolation locations further includes:

[0055] In the second type of position (R2, r2), a first candidate position R2 is determined for interpolation of the second type of position (R2, r2); a 5×3 target pixel block is obtained as a gradient image block with the first candidate position R2 as the center; for non-first candidate positions r2 in the second type of position (R2, r2), the Hexa-deca data is rotated or flipped to the first candidate position R2 and the corresponding gradient image block is determined.

[0056] Specifically, for the second type of position (R2, r2), this embodiment uses R2 as the candidate position for interpolation. Other pixels at position r2 can be transformed to the R2 position by rotating or flipping the Hexa-deca data before interpolation. For example... Figure 5 As shown, the interpolation process for the pixel at position R2 in a 4×4 R pixel image block first includes obtaining a 5×3 target pixel block centered on the candidate position R2 to be interpolated as the gradient image block. The gradient image block is selected here because there is no G value under the 3×3 target pixel block centered on R2, and its vertical gradient calculation has defects. Therefore, the 5×3 target pixel block centered on R2 is selected as the gradient image block.

[0057] For the third type of location (R3, r3), determining the gradient image patch corresponding to each of the first interpolation locations further includes:

[0058] In the third type of position (R3, r3), a second candidate position R3 is determined for interpolation of the third type of position (R3, r3); a 3×3 target pixel block is obtained as a gradient image block with the second candidate position R3 as the center; for non-second candidate positions r3 in the third type of position (R3, r3), the Hexa-deca data is transformed to the second candidate position R3 by performing a horizontal or vertical flip operation to determine the corresponding gradient image block.

[0059] Specifically, for the third type of position (R3, r3), this embodiment uses R3 as the candidate position for interpolation. Other pixels at position r3 can be transformed to the R3 position by performing a horizontal or vertical flip operation on the Hexa-deca data before interpolation. For example... Figure 5 As shown, the interpolation process for pixel R3 in a 4×4 R-pixel image block first includes obtaining a 3×3 target pixel block centered on the candidate position R3 to be interpolated as a gradient image block.

[0060] Step S32: Based on the gradient image patch, calculate the gradient information for each of the first interpolation locations;

[0061] Calculate the orientation, intensity, and consistency information of the first candidate position and the second candidate position based on the gradient image patch;

[0062] Specifically, gradient information at positions R2 and R3 is calculated based on the acquired gradient image patches. This gradient information mainly includes three features: direction, intensity, and consistency. The following details how to calculate the required gradient information. First, the average gradient g of the gradient image patch is calculated. x and the average vertical gradient g y That is, to calculate the average gradient g of R2 based on a 5×3 gradient image patch centered at R2. x and the average vertical gradient g y Calculate the average gradient g of R3 based on a 3×3 gradient image patch centered at R3. x and the average vertical gradient g y ;

[0063] According to the average gradient g x and the average vertical gradient g y The gradient image patch structure tensor can be obtained, and the calculation formula is shown below:

[0064]

[0065] Simultaneously, the required matrix trace T and determinant value D can be obtained from the structure tensor.

[0066]

[0067] As mentioned above, the structure tensor is a 2×2 matrix S. The two eigenvalues ​​λ1 and λ2 of the structure tensor matrix S can be calculated mathematically, as shown below (λ1 ≥ λ2):

[0068]

[0069] Finally, using the two eigenvalues ​​λ1 and λ2 of the structure tensor matrix S, the orientation, intensity, and consistency information of the gradient image patch can be calculated as follows:

[0070] (1) Direction angle = atan2(I xy +ε,λ1-I yy ) represents the angular direction of the neighborhood of this pixel, where ε is a very small value used to ensure computational stability. The angular direction is first normalized to 0 to π, and then quantized to, for example, Figure 7 The eight quantization directions shown are as follows: Figure 7 The numbers 0 to 7 are used in the text.

[0071] (2) Strength This represents the magnitude of the gradient within the neighborhood, with the intensity quantized to three levels: 0, 1, and 2, with the magnitude gradually increasing.

[0072] (3) Consistency It represents the degree of consistency of the image structure within the neighborhood. The consistency is quantified to three levels: 0, 1, and 2, with the value gradually increasing.

[0073] The pre-trained filters are selected based on the direction, intensity, and consistency information; interpolation is performed based on the selected filters to obtain the G-pixel value for each of the first interpolation positions.

[0074] Specifically, during the training process of the filters, the filters are classified by referring to the three feature information of orientation, intensity, and consistency of the gradient image patch and their quantization levels. The orientation feature has 8 quantization levels, the intensity feature has 3 quantization levels, and the consistency feature has 3 quantization levels. Therefore, 8×3×3=72 different types of filters can be trained for the reference R2 pixel and the reference R3 pixel, respectively. Each type of filter is suitable for different image regions. In the interpolation process, the gradient information of the gradient image patch in the neighborhood of the target R2 or target R3 pixel is calculated first. Then, the filters corresponding to the quantization levels of the three features are selected according to the gradient information to complete the interpolation calculation.

[0075] Step S33: Select a pre-trained filter based on the gradient information to perform interpolation calculation, and obtain the G pixel value of each first interpolation position.

[0076] As described above, during the interpolation process, the gradient information of the gradient image block in the neighborhood of pixel R2 or R3 is first calculated, and then the filters corresponding to the quantization levels of the three features are selected according to the gradient information to complete the interpolation calculation.

[0077] Specifically, such as Figure 8 A schematic diagram of a 7×7 filter trained under the conditions of orientation level 6 (i.e., 135 degrees), intensity level 2, and consistency level 2 is given, targeting pixels R3 and R2. Pixel R3 is the left filter, and pixel R2 is the right filter. The filter range is a 7×7 image patch centered at R3. Interpolation is calculated by multiplying the weight coefficients in the image patch by the corresponding G-pixel value. Based on this, the pre-trained filter can be selected for interpolation calculation based on the previously calculated gradient information to obtain the G-pixel values ​​at positions R2 and R3. As mentioned above, pixel r2 can be transformed to the R2 position through horizontal or vertical flipping and clockwise or counterclockwise rotation operations for unified processing. Similarly, pixel r3 can be transformed to the R3 position through horizontal or vertical flipping of the image patch for unified processing. Likewise, the G-pixel values ​​at positions r2 and r3 can be calculated. Figure 5 The lowercase 'g' is used to represent the green pixels obtained by interpolating multiple positions of the first interpolation point.

[0078] In another specific implementation of this application, based on the G pixel values ​​of multiple first interpolation positions, a second interpolation process is performed on the second interpolation position to generate a full-resolution G image, which further includes:

[0079] In the second interpolation position, a third candidate position R1 is determined for interpolating the first type of position (R1, r1); the third candidate position R1 is subjected to a second interpolation process to obtain the G-pixel value of the third candidate position R1; for the non-third candidate position r1 in the first type of position (R1, r1), the Hexa-deca data is transformed to the third candidate position R1 by a horizontal or vertical flip operation, and then subjected to a second interpolation process to obtain the G-pixel value of the non-third candidate position r1; based on the G-pixel values ​​of the first interpolation position and the second interpolation position, a full-resolution G image is generated.

[0080] Specifically, for the first type of position (R1, r1), interpolation is performed using R1 as the candidate position. For other pixels at position r1, the Hexa-deca data can be transformed to the R1 position through rotation or flipping before interpolation. For example... Figure 9The diagram shows the interpolation process for pixel R1 in Hexa-deca Bayer CFA. After obtaining the G pixel values ​​at positions R2 and R3, the interpolation of the G pixel at R1 becomes easier due to the increased number of neighboring G pixel reference points. The following section details... Figure 9 The interpolation process of pixel R1 in the Hexa-deca Bayer CFA shown is explained in detail.

[0081] As mentioned above, refer to Figure 10 The step of performing a second interpolation process on the second interpolation position based on the G pixel values ​​of multiple first interpolation positions to generate a full-resolution G image further includes:

[0082] Step S41: Determine the direction information of the third candidate position based on the direction information obtained when interpolating the first position to be interpolated;

[0083] Obtain the direction information of multiple first interpolation positions adjacent to the third candidate position R1; select the mode of the direction information as the direction information of the third candidate position R1.

[0084] Specifically, such as Figure 9 As shown, the orientation information of positions (R2,r2) and (R3,r3) is obtained. The orientation of pixel R1 is determined based on the orientation information calculated during interpolation of positions (R2,r2) and (R3,r3). The determination rule is based on the consistency of orientation information in the neighboring regions. Specifically, the orientation information of R1 can be determined by voting from D1, D3, and D4; or the orientation information of R1 can be determined by voting from the mode of D1-D12.

[0085] Step S42: Based on the orientation information of the third candidate position, the G pixel value of the third candidate position is obtained by gradient weighted interpolation or color difference interpolation.

[0086] Furthermore, step S42 also includes:

[0087] Step S421: Determine whether the direction information of the third candidate position belongs to the first type of direction or the second type of direction. The pixel position in the first type of direction has both the original pixel value and the interpolated G pixel value, while the pixel position in the second type of direction only has the original pixel value.

[0088] Specifically, the original pixel value can be a B or R pixel value.

[0089] Step S422: For the first type of direction, the G pixel value of the third candidate position is obtained by using a color difference interpolation method;

[0090] Step S43: For the second type of direction, the G pixel value of the third candidate position is obtained by gradient-weighted interpolation.

[0091] As described above, after determining the direction information of the third candidate position R1, further, as... Figure 9 As shown, the direction of R1 is consistent with the quantization level of the direction information, and there are 8 possible directions. For directions 0, 4, 5, 6, and 7, since both the R pixel value (i.e., the original pixel value) and the G pixel value from the first interpolation process exist simultaneously, an interpolation method based on the color difference assumption can be used. For directions 1, 2, and 3, a gradient-weighted interpolation method can be used to obtain the G pixel value. The interpolation methods for the two different types of directions are illustrated with examples below.

[0092] Assuming the direction R1 is determined to be 6, since both R and G pixel values ​​exist in direction 6, an interpolation method based on the color difference assumption can be used. The principle of the color difference assumption is that the color difference is assumed to be equal or very small within a small range. Therefore, the color difference at the three pixel positions (R1, g4, g8) in direction 6 is equal. Thus, the G pixel value can be calculated using the following formula based on the color difference interpolation:

[0093] g1=R1+0.5×((g4-rr4)+(g8-rr8)) (4)

[0094] Assuming the direction of R1 is determined to be 2, since only the G pixel value exists in direction 2, interpolation cannot be calculated based on color difference. Instead, the G pixel value can be calculated using gradient weighting. First, calculate the gradient values ​​in the upper right (ur) direction and the lower left (dl) direction. Then, calculate the G pixel value at position R1 by gradient weighting based on the positions of G3 and G6.

[0095] grad ur =ABS(G2-G3)+ABS(G3-R1) (5)

[0096] grad dl =ABS(G7-G6)+ABS(G6-R1) (6)

[0097]

[0098] Based on the above calculations, the G pixel values ​​on all types of R pixels can be calculated, and similarly, the G values ​​on B pixels can also be calculated, thus obtaining the G channel values ​​for the full resolution.

[0099] The process of interpolating the G channel at full resolution uses the least squares method to pre-train interpolation filters suitable for different image regions. By using these pre-trained interpolation filters, the G pixels at positions R2 and R3 are recovered. Finally, the G pixels at position R1 can be obtained through color difference interpolation or gradient weighted interpolation. In this way, various complex detection modules can be eliminated, reducing the algorithm complexity, thereby reducing the consumption of the image processor and improving the computing speed.

[0100] Furthermore, the image processing scheme described in this application also includes interpolating the R and B channels after completing the G channel interpolation, thereby achieving complete demosaic processing of the Hexa-deca configuration. The guided interpolation of the R and B channels is described in detail below. In this embodiment, a bilinear upsampling method based on color difference is used. In addition to using interpolation based on the color difference assumption, guided interpolation based on color ratio can also be used, or a pre-trained filter method similar to the G channel interpolation can be used for guided interpolation. This application does not limit this.

[0101] Further, refer to Figure 11 The image processing method further includes:

[0102] Step S5: Downsample the full-resolution G image and the Hexa-deca data to generate a half-resolution G image and a half-resolution RGB image, respectively.

[0103] Specifically, such as Figure 12 The diagram shows the complete demosaic algorithm flow for Hexa-deca BayerCFA. (Refer to...) Figure 12 In this embodiment, the Hexa-deca Bayer CFA channel and the full-resolution G channel are binning to obtain a G channel with double downsampling and a Quad CFA channel image with a period of 4×4, thus generating a half-resolution G image and a half-resolution RGB image. Based on the half-resolution G image, the R / B channels in the Quad CFA channel image are subjected to guided interpolation to generate a half-resolution R image and a half-resolution B image. After the guided interpolation of the G channel on the R and B channels is completed at the double downsampling resolution, the full-resolution G channel is finally used to perform guided upsampling on the R and B channels to generate a full-resolution RGB image.

[0104] Step S6: Perform guided interpolation processing on the half-resolution RGB image based on the half-resolution G image to generate half-resolution R and B images;

[0105] Specifically, such as Figure 13The diagram illustrates the process of using the G channel to guide the interpolation of the R / B channels after doubling downsampling. This scheme uses the G channel to guide the interpolation of the R and B channels, utilizing gradient information on the G channel to determine the interpolation direction, and then performing interpolation calculations on the color difference channels. After merging the Hexa-deca Bayer CFA channel and the full-resolution G channel into a doubly downsampled G channel and a 4×4 Quad CFA channel, the color difference channel is obtained by subtracting the Quad CFA channel from the G channel, and interpolation calculations are then performed on the color difference channel; as shown... Figure 13 As shown, firstly, the B color difference information on the R pixel and the R color difference information on the B pixel are interpolated and calculated. Then, the R and B color difference information on the G pixel are calculated. Finally, the R and B channels are obtained by adding the color difference channel to the original G channel. This interpolation can also be based on other interpolation methods such as color ratio assumptions, which are not limited in this application.

[0106] Step S7: Guide upsampling of the half-resolution R and B images based on the full-resolution G image to generate a full-resolution RGB image.

[0107] Specifically, after performing guided interpolation of the G channel onto the R and B channels at a resolution that is twice the downsampled value, guided upsampling of the R and B channels is finally performed using the full-resolution G channel. This scheme uses chromatic difference-based guided upsampling, that is, first performing bilinear upsampling on the GR and GB chromatic difference channels that are twice the downsampled value, and then adding the full-resolution GR and GB chromatic difference channels to the full-resolution G channel, thus obtaining the final RGB color image. Guided upsampling can also be performed using a learning-based approach, and this application does not limit this approach.

[0108] In summary, the embodiments of this application can effectively obtain a full-resolution G-channel image of the Hexa-deca configuration CFA through the aforementioned staged interpolation method, without requiring a complex detection module, thus reducing algorithm complexity, computational and space consumption, and improving computational speed. Furthermore, the full-resolution G-channel image can be used to perform de-mosaic processing on the Hexa-deca configuration CFA to obtain a full-resolution RGB color image. This configuration CFA can use a 4-in-1 or 16-in-1 pixel binning method to improve the imaging capability and signal-to-noise ratio of specific scenes.

[0109] Reference Figure 14 This diagram illustrates a structural schematic of an image processing apparatus according to an embodiment of this application. The apparatus 140 is used to process RGB images of Hexa-deca configuration acquired by an image sensor, such as... Figure 14 As shown, the device 140 includes:

[0110] The acquisition module 1401 is used to obtain Hexa-deca data according to the Hexa-deca configuration color filter array, wherein the Hexa-deca data includes G pixels and non-G pixels;

[0111] The determining module 1402 is used to determine a plurality of first interpolation positions and a plurality of second interpolation positions in the non-G pixel positions;

[0112] The first interpolation module 1403 is used to perform a first interpolation process on the first position to be interpolated according to a pre-trained interpolation filter to obtain multiple G pixel values ​​of the first position to be interpolated.

[0113] The second interpolation module 1404 is used to perform a second interpolation process on the second interpolation position based on the G pixel values ​​of multiple first interpolation positions to generate a full-resolution G image.

[0114] In another implementation of this application, the first interpolation module 1403 is further used for:

[0115] Based on each of the first interpolation locations, determine the gradient image patch corresponding to the first interpolation location;

[0116] Based on the gradient image patch, the gradient information of each of the first interpolation positions is calculated;

[0117] Based on the gradient information, a pre-trained filter is selected for interpolation calculation to obtain the G pixel value at each of the first interpolation positions.

[0118] In another implementation of this application, the first interpolation module 1403 is further used for:

[0119] A first candidate position is determined from the second type of positions for interpolation of the second type of positions;

[0120] A 5×3 target pixel block is obtained as a gradient image block, centered on the first candidate position;

[0121] For non-first candidate positions in the second type of position, the Hexa-deca data is rotated or flipped to the first candidate position and the corresponding gradient image block is determined.

[0122] In another implementation of this application, the first interpolation module 1403 is further used for:

[0123] A second candidate position is determined from the third type of positions for interpolation of the third type of positions;

[0124] A 3×3 target pixel block is obtained as a gradient image block, centered on the second candidate position;

[0125] For non-second candidate positions in the third category, the Hexa-deca data is transformed to the second candidate position by performing a horizontal or vertical flipping operation to determine the corresponding gradient image block.

[0126] In another implementation of this application, the first interpolation module 1403 is further used for:

[0127] Calculate the orientation, intensity, and consistency information of the first candidate position and the second candidate position based on the gradient image patch;

[0128] The pre-trained filters are selected based on the direction, intensity, and consistency information.

[0129] Interpolation calculations are performed based on the selected filter to obtain the G pixel value for each of the first interpolation positions.

[0130] In another implementation of this application, the second interpolation module 1404 is further used for:

[0131] A third candidate position is determined from the second position to be interpolated for interpolating the first type of position;

[0132] A second interpolation process is performed on the third candidate position to obtain the G pixel value of the third candidate position;

[0133] For non-third candidate positions in the first type of position, the Hexa-deca data is transformed to the third candidate position by performing a horizontal or vertical flip operation, and then a second interpolation process is performed to obtain the G pixel value of the non-third candidate position.

[0134] A full-resolution G image is generated based on the G pixel value of the first interpolation position and the G pixel values ​​of the third candidate position and the non-third candidate position in the second interpolation position.

[0135] In another implementation of this application, the second interpolation module 1404 is further used for:

[0136] The direction information of the third candidate position is determined based on the direction information obtained when interpolating the first position to be interpolated;

[0137] The G pixel value of the third candidate position is obtained by gradient weighted interpolation or color difference interpolation based on the orientation information of the third candidate position.

[0138] In another implementation of this application, the second interpolation module 1404 is further used for:

[0139] Obtain the orientation information of multiple first interpolation positions adjacent to the third candidate position;

[0140] The mode of the direction information is selected as the direction information for the third candidate position.

[0141] In another implementation of this application, the second interpolation module 1404 is further used for:

[0142] The direction information of the third candidate position is determined to belong to either the first type of direction or the second type of direction. The pixel position in the first type of direction has both the original pixel value and the interpolated G pixel value, while the pixel position in the second type of direction only has the original pixel value.

[0143] For the first type of direction, the G pixel value of the third candidate position is obtained by using a color difference interpolation method;

[0144] For the second type of direction, the G pixel value of the third candidate position is obtained by gradient-weighted interpolation.

[0145] In another possible implementation of this application, the image processing apparatus further includes:

[0146] The downsampling module is used to downsample the full-resolution G image and the Hexa-deca data to generate a half-resolution G image and a half-resolution RGB image, respectively.

[0147] The guided interpolation module is used to perform guided interpolation processing on the half-resolution RGB image based on the half-resolution G image to generate half-resolution R and B images.

[0148] The upsampling module is used to guide the upsampling of the half-resolution R and B images based on the full-resolution G image to generate a full-resolution RGB image.

[0149] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 15 As shown, the image processing device 150 includes a processor 1501 and a memory 1502.

[0150] The memory 1502 is used to store computer programs, and the processor 1501 is used to call and run the computer programs stored in the memory 1502 to perform the image processing method as described in any of the method embodiments.

[0151] The image processing apparatus of this embodiment is used to implement the corresponding methods in the foregoing multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. The electronic device may be, for example, a smartphone, laptop, tablet, gaming device, or other portable or mobile computing device, etc., and this application embodiment is not limited to this.

[0152] This application provides a chip including a processor for calling and running a computer program from a memory, causing a device with the chip installed to perform an image processing method as described in any of the foregoing methods.

[0153] This chip can be applied to the image processing apparatus or electronic device in the embodiments of this application, and can implement the corresponding processes implemented by the image processing apparatus or electronic device in the various methods of the embodiments of this application, and achieve the corresponding effects. For the sake of brevity, further details are omitted here.

[0154] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0155] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the checksum generation method described herein. Furthermore, when a general-purpose computer accesses code used to implement the checksum generation method shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the checksum generation method shown herein.

[0156] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0157] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: Hexa-deca data is obtained according to a Hexa-deca configuration color filter array, wherein the Hexa-deca data comprises G pixels and non-G pixels; a plurality of first interpolation positions and a plurality of second interpolation positions are determined in the non-G pixel positions; first interpolation processing is performed on the first interpolation positions according to a pre-trained interpolation filter to obtain G pixel values of the plurality of first interpolation positions; second interpolation processing is performed on the second interpolation positions according to the G pixel values of the plurality of first interpolation positions to generate a full-resolution G image; the step of determining the plurality of first interpolation positions and the plurality of second interpolation positions in the non-G pixel positions comprises: the non-G pixel positions comprise first-type positions, second-type positions and third-type positions, wherein the first-type positions are adjacent to two G pixels around, the second-type positions are adjacent to one G pixel around, and the third-type positions are not adjacent to G pixels around; the second-type positions and the third-type positions are determined as the plurality of first interpolation positions; the first-type positions are determined as the plurality of second interpolation positions.

2. The image processing method of claim 1, wherein, the step of performing first interpolation processing on the first interpolation positions according to a pre-trained interpolation filter to obtain G pixel values of the plurality of first interpolation positions comprises: a gradient image block corresponding to each first interpolation position is determined based on the first interpolation position; gradient information of each first interpolation position is calculated based on the gradient image block; G pixel values of each first interpolation position are obtained by performing interpolation calculation according to the gradient information and a pre-trained filter.

3. The image processing method of claim 2, wherein, the step of determining a gradient image block corresponding to each first interpolation position based on the first interpolation position comprises: a first candidate position for interpolation of the second-type positions is determined in the second-type positions; a 5*3 target pixel block centered on the first candidate position is obtained as a gradient image block; for non-first candidate positions in the second-type positions, a gradient image block is determined after a rotation or flipping operation is performed on Hexa-deca data and the Hexa-deca data is transformed to the first candidate position.

4. The image processing method of claim 3, wherein, the step of determining a gradient image block corresponding to each first interpolation position based on the first interpolation position comprises: a second candidate position for interpolation of the third-type positions is determined in the third-type positions; a 3*3 target pixel block centered on the second candidate position is obtained as a gradient image block; for non-second candidate positions in the third-type positions, a gradient image block is determined after a horizontal or vertical flipping operation is performed on Hexa-deca data and the Hexa-deca data is transformed to the second candidate position.

5. The image processing method of claim 4, wherein, the step of calculating gradient information of each first interpolation position based on the gradient image block and obtaining G pixel values of each first interpolation position by performing interpolation calculation according to the gradient information and a pre-trained filter further comprises: direction, intensity and consistency information of the first candidate position and the second candidate position are calculated respectively based on the gradient image block. select a pre-trained filter based on the direction, intensity and consistency information; perform interpolation calculation according to the selected filter to obtain G pixel values of each first interpolation position.

6. The image processing method of claim 5, wherein, The second interpolation processing of the second interpolation position according to the G pixel values of the plurality of first interpolation positions includes: determining a third candidate position for interpolation of the first type position in the second interpolation position; and performing second interpolation processing on the third candidate position to obtain the G pixel value of the third candidate position. For non-third candidate positions in the first type position, the Hexa-deca data is transformed by horizontal or vertical flipping operation to the third candidate position and then subjected to second interpolation processing to obtain the G pixel value of the non-third candidate position. Based on the G pixel values of the first interpolation position and the third candidate position and the non-third candidate position in the second interpolation position, a full-resolution G image is generated.

7. The image processing method of claim 6, wherein, The second interpolation processing includes: determining the direction information of the third candidate position according to the direction information obtained when interpolating the first interpolation position; obtaining the G pixel value of the third candidate position by gradient weighted interpolation or chromatic aberration interpolation based on the direction information of the third candidate position.

8. The image processing method of claim 7, wherein, The determination of the direction information of the third candidate position according to the direction information obtained when interpolating the first interpolation position includes: obtaining the direction information of a plurality of first interpolation positions adjacent to the third candidate position; and selecting the mode of the direction information as the direction information of the third candidate position.

9. The image processing method of claim 8, wherein, The obtaining of the G pixel value of the third candidate position based on the direction information of the third candidate position includes: determining whether the direction information of the third candidate position belongs to a first type direction or a second type direction, wherein a pixel position in the first type direction has both an original pixel value and an interpolated G pixel value, and a pixel position in the second type direction has only an original pixel value; for the first type direction, the G pixel value of the third candidate position is obtained by a chromatic aberration interpolation based method; and for the second type direction, the G pixel value of the third candidate position is obtained by a gradient weighted interpolation based method.

10. The image processing method of any one of claims 1-5, wherein, Further comprising: down-sampling the full-resolution G image and the Hexa-deca data to generate a half-resolution G image and a half-resolution RGB image, respectively; performing guided interpolation processing on the half-resolution RGB image according to the half-resolution G image to generate a half-resolution R image and a half-resolution B image; and performing guided up-sampling on the half-resolution R image and the half-resolution B image according to the full-resolution G image to generate a full-resolution RGB image.

11. An image processing apparatus characterized by comprising: The method comprises: an acquisition module configured to obtain Hexa-deca data from a Hexa-deca configuration color filter array, the Hexa-deca data including G pixels and non-G pixels; a determination module configured to determine a plurality of first interpolation positions and a plurality of second interpolation positions in the non-G pixel positions; The first interpolation module is configured to perform first interpolation processing on the first to-be-interpolated positions according to a pre-trained interpolation filter to obtain G pixel values of the first to-be-interpolated positions. The second interpolation module is configured to perform second interpolation processing on the second to-be-interpolated positions according to the G pixel values of the first to-be-interpolated positions to generate a full-resolution G image. The determining of the first to-be-interpolated positions and the second to-be-interpolated positions in the non-G pixel positions comprises: The non-G pixel positions comprise first-type positions, second-type positions and third-type positions, wherein the first-type positions are adjacent to two G pixels around, the second-type positions are adjacent to one G pixel around, and the third-type positions are not adjacent to G pixels around. The second-type positions and the third-type positions are determined as the first to-be-interpolated positions. The first-type positions are determined as the second to-be-interpolated positions.

12. A chip, characterized by The image processing method comprises: A processor is configured to call and run a computer program from a memory, so that a device installed with the chip performs the image processing method according to any one of claims 1 to 10.

13. An image processing apparatus characterized by comprising: The image processing method comprises a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory to perform the image processing method according to any one of claims 1 to 10.

Images