A Bayer sensor super-resolution imaging method and device

Abstract

The present invention provides a Bayer sensor super-resolution imaging method and device, which processes the data acquired by the Bayer sensor based on weighted interpolation and chroma and saturation to realize Bayer sensor super-resolution imaging, including directly cloning the nearest color value of the pixels at the four corners of the image; weighted interpolation of the color value of each other pixel of the image, taking 2×2 four grids as the basic imaging unit, taking the red, green and blue color values ​​of the pixel to be restored as unknowns, establishing a relationship equation according to the red, green and blue color values ​​and color composition relationship recorded by the Bayer sensor, obtaining the equation solution expression, and realizing weighted interpolation; re-optimization processing of the image pixel color value, including corresponding RGB and IHS conversion according to the chroma and saturation invariant characteristics in the imaging unit; enhancement processing of the image pixel color value, including enhancement through a low-pass filter and an edge enhancer, to obtain a super-resolution imaging result. The present invention can solve the problem of reduced imaging quality and resolution of the Bayer sensor.

Description

Technical Field

[0001] The invention belongs to the field of optical imaging and image processing, and relates to a method for processing data acquired by a Bayer sensor through weighted interpolation and hue and saturation related technical solutions to achieve Bayer sensor super-resolution imaging. Background Art

[0002] With the continuous development of imaging sensor technology and image processing technology, imaging technology has become more and more mature, and various cameras have also developed overwhelmingly, entering society and becoming an indispensable item in people's daily life. In order to reduce the manufacturing cost of cameras, manufacturers can be said to have racked their brains. The Bayer sensor used in low-cost cameras is a very representative advanced technology. The Bayer sensor uses CMOS imaging and is mainly composed of four parts: microlens, color filter, photodiode, and pixel processing. After passing through the microlens, the light is focused on the color filter. The filter splits the visible light into three components of red, green, and blue and passes it to the photosensitive element. The photosensitive element is arranged in a Bayer array to receive the split visible light. The pixel processing part integrates the Bayer array to form an image. The classic Bayer array is a 2×2 four-grid arrangement of four groups of green, red, blue, and green photosensitive elements. The arrangement order in the 2×2 four-grid is usually green, red, blue, and green, that is, the 2×2 four-grid is actually just one pixel, and the red, green, and blue photosensitive elements are arranged in alternate rows and columns. The problem caused by this is that the resolution of the sensor is reduced. In addition, this method of arranging photosensitive elements in alternate rows and columns will also lead to reduced imaging quality. There is no doubt that relevant technical personnel have carried out various studies to solve the above problems in an attempt to restore the image quality and resolution of the Bayer sensor. Among them, the most classic method is to perform linear interpolation on the 2×2 four-grid Bayer array, and restore the red, green, and blue color values ​​of each grid (that is, each pixel) of the 2×2 four-grid image through the front and back, upper and lower photosensitive elements. Linear interpolation can achieve visual smoothness of the image, but it will lose some features. In order to find other methods to better restore the image quality and resolution of the Bayer sensor, the present invention conducts research on this issue.

[0003] Related terms:

[0004] Bayer encoding method

[0005] CMOS Metal Oxide Semiconductor Imaging Technology Summary of the invention

[0006] In response to the problem of reduced imaging quality and resolution of Bayer sensors, the present invention proposes a method for processing data acquired by the Bayer sensor through weighted interpolation and hue and saturation related technical solutions to achieve Bayer sensor super-resolution imaging, that is, a super-resolution imaging method for Bayer sensors is proposed.

[0007] The technical solution of the present invention is a Bayer sensor super-resolution imaging method, which processes the data obtained by the Bayer sensor based on weighted interpolation and hue and saturation to achieve Bayer sensor super-resolution imaging. The implementation process includes the following steps:

[0008] Step 1: directly clone the nearest color values ​​of the pixels at the four corners of the image;

[0009] Step 2, weighted interpolation of the color value of each other pixel of the image, including taking 2×2 four grids as the basic imaging unit, taking the red, green, and blue color values ​​of the pixel to be restored as unknowns, establishing a relationship equation based on the red, green, and blue color values ​​and color composition relationship recorded by the Bayer sensor, obtaining the equation solution expression, and realizing weighted interpolation;

[0010] Step 3, re-optimization of the color value of the image pixel, including performing corresponding RGB and IHS conversion to achieve re-optimization according to the invariant characteristics of hue and saturation in the imaging unit;

[0011] Step 4, enhancement processing of the color value of the image pixel, including enhancement through a low-pass filter and an edge enhancer, to obtain a super-resolution imaging result of the corresponding pixel.

[0012] Moreover, in step 2, the following steps are used to implement weighted interpolation for each pixel in the image:

[0013] Step 2.1, taking the 2×2 four grids to which the pixel belongs as the processing unit, marking the red, green, and blue color values ​​recorded in the 2×2 four grids as G, R, B, and GG, and marking the 2×2 four grids as grid 1, grid 2, grid 3, and grid 4;

[0014] Step 2.2, for the 2×2 four-grid grid 1, its recorded color is G. Suppose the color components to be restored are the red component R1 and the blue component B1. For R1, the color values ​​recorded by the two pixels closest to grid 1 are R and R1', respectively, and their average value is recorded as avR1. For B1, the color values ​​recorded by the two pixels closest to grid 1 are B and B1', respectively, and their average value is recorded as avB1. Suppose the errors of the red component R1 and the blue component B1 in the color components to be restored are dR1 and dB1, then the weighted interpolation formula is:

[0015] R1+dR1=(R+avR1) / 2=(3R+R1') / 4

[0016] B1+dB1=(B+avB1) / 2=(3B+B1') / 4

[0017] Among them, R1' and B1' are the color values ​​of the pixels adjacent to the pixel of grid 1;

[0018] Step 2.3, for the 2×2 four-grid grid 2, grid 3, and grid 4, assume that the color components to be restored are green component G2, blue component B2, red component R3, green component G3, red component R4, and blue component B4, and use similar processing as grid 1. The corresponding weighted interpolation formulas are:

[0019] G2+dG2=(G+avG2) / 2=(3G+G2') / 4

[0020] B2+dB2=(B+avB2) / 2=(3B+B2') / 4

[0021] R3+dR3=(R+avR3) / 2=(3R+R3') / 4

[0022] G3+dG3=(G+avG3) / 2=(3G+G3') / 4

[0023] R4+dR4=(R+avR4) / 2=(3R+R4') / 4

[0024] B4+dB4=(B+avB4) / 2=(3B+B4') / 4

[0025] Among them, G2' and B2' are the color values ​​of pixels adjacent to grid 2 pixel, R3' and G3' are the color values ​​of pixels adjacent to grid 3 pixel, R4' and B4' are the color values ​​of pixels adjacent to grid 4 pixel; avG2, avB2, avR3, avG3, avR4, avB4 are the average values ​​of the two nearest pixels of G2, B2, R3, G3, R4, B4 respectively, and dG2, dB2, dR3, dG3, dR4, dB4 are the corresponding errors of G2, B2, R3, G3, R4, B4.

[0026] Moreover, in step 3, the pixel color value is further optimized as follows:

[0027] Step 3.1, using the 2×2 grid to which the pixel belongs as the processing unit, convert the original recorded G, R, and B into the IHS space, with the components recorded as brightness I, hue H, and saturation S; use the right-multiplied triangular approximate conversion matrix formula for processing, and record the RGB to IHS matrix as A,

[0028] The calculation formula is:

[0029] (I, H, S) = A × (R, G, B)

[0030] Expanded:

[0031] I=A[0][0]×R+A[0][1]×G+A[0][2]×B

[0032] H=A[1][0]×R+A[1][1]×G+A[1][2]×B

[0033] S=A[2][0]×R+A[2][1]×G+A[2][2]×B

[0034] The matrix converted from IHS to RGB is the inverse matrix of A, denoted by A -1 , since A is an orthogonal matrix, the inverse matrix is ​​its transposed matrix, denoted by A T , the calculation formula is:

[0035] (R, G, B) = A T ×(I, H, S)

[0036] Expanded:

[0037] R=A[0][0]×I+A[1][0]×H+A[2][0]×S

[0038] G=A[0][1]×I+A[1][1]×H+A[2][1]×S

[0039] B=A[0][2]×I+A[1][2]×H+A[2][2]×S

[0040] Step 3.2, mark the four 2×2 grids as grid 1, grid 2, grid 3, grid 4, and do the following processing on grid 1: take the color components R1+dR1, G, B1+dB1 of grid 1 as row vectors, multiply them with matrix A and transform them into IHS space. The transformation results are recorded as I1, H1, and S1 respectively; to ensure the consistency of hue and saturation within the processing unit, directly replace H1 and S1 with H and S of the four 2×2 grids, and then use I1, H, S and A to convert them into IHS space. T Multiply them to get new R1", G1", and B1", as the three color components of the optimized grid 1. The calculation process is as follows:

[0041] (I1, H1, S1) = A × (R1 + dR1, G, B1 + dB1)

[0042] (R1", G1", B1") = A T ×(I1, H, S)

[0043] Step 3.3, similarly, for grid 2, grid 3, grid 4, similar processing is applied to grid 1 to obtain the optimized color components of grid 2, grid 3, grid 4, recorded as R2", G2", B2", R3", G3", B3", R4", G4", B4".

[0044] The calculation formula for cell 2 is:

[0045] (I2, H2, S2)=A×(R, G2+dG2, B2+dB2)

[0046] (R2", G2", B2") = A T ×(I2,H,S)

[0047] The calculation formula for cell 3 is:

[0048] (I3, H3, S3) = A × (R3 + dR3, G3 + dG3, B)

[0049] (R3", G3", B3") = A T ×(I3,H,S)

[0050] The calculation formula for cell 4 is:

[0051] (I4, H4, S4) = A × (R1 + dR1, GG, B1 + dB1)

[0052] (R4", G4", B4") = A T ×(I4,H,S)

[0053] Among them, (I2, H2, S2), (I3, H3, S3), and (I4, H4, S4) are the conversion results of color components of grids 2, 3, and 4 multiplied by matrix A and converted to IHS space.

[0054] Moreover, in step 4, the method for enhancing the color value of the image pixel is:

[0055] Step 4.1, design a low-pass filter h to smooth the three color components separately.

[0056] The calculation formula is:

[0057] p (i，j) =g (i-1，j-1) ×h (0，0) +g (i-1，j) ×h (0，1) +g (i-1，j+1) ×h (0，2) +

[0058] g (i，j-1) ×h (1，0) +g (i，j) ×h (1，1) +g (i，j+1) ×h (1，2) +

[0059] g (i+1，j-1) ×h (2，0) +g (i+1，j) ×h (2，1) +g (i+1，j+1) ×h (2，2)

[0060] In the formula, h(i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, p (i，j) is the processing result of the pixel with row number i and column number j;

[0061] Step 4.2, set an edge enhancer to perform edge enhancement processing on the three color components respectively.

[0062] The specific calculation formula is:

[0063] q (i，j) =g (i-1，j-1) ×f (0，0) +g (i-1，j) ×f (0，1) +g (i-1，j+1) ×f (0，2) +

[0064] g (i，j-1) ×f (1，0) +g (i，j) ×f (1，1) +g (i，j+1) ×f (1，2) +

[0065] g (i+1，j-1) ×f (2，0) +g (i+1，j) ×f (2，1) +g (i+1，j+1) ×f (2，2)

[0066] In the formula, f (i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, q (i，j) It is the processing result of the pixel with row number i and column number j.

[0067] Moreover, the DSP chip is used to complete the super-resolution imaging process.

[0068] On the other hand, the present invention provides a computer device / apparatus / system, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the super-resolution imaging method of the Bayer sensor as described above.

[0069] On the other hand, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the super-resolution imaging method of the Bayer sensor as described above.

[0070] On the other hand, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the super-resolution imaging method of the Bayer sensor as described above.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] (1) The traditional Bayer sensor uses linear interpolation for interlaced and interlaced pixel restoration. The present invention first proposes a weighted interpolation formula and uses weighted interpolation to enhance the camera imaging quality.

[0073] (2) Effectively utilize the advantages of Bayer sensors and ensure that the hue and saturation of the imaging unit remain unchanged through RGB and IHS conversion, thereby improving the camera imaging quality.

[0074] (3) Easy to implement in hardware. All processing is expressed in matrices and can be completed directly using the relevant hardware functions of the DSP chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Schematic diagram of the red, green and blue photosensitive elements of the Bayer sensor imaging in the prior art.

[0076] Figure 2 This is a schematic diagram of the meaning of variables related to the processing units in the four 2×2 grids according to the embodiment of the present invention.

[0077] Figure 3 The following is a schematic diagram of the relationship between R1' and B1' adjacent to the pixel of grid 1, taking grid 1 as an example in the embodiment of the present invention.

[0078] Figure 4 The figure is a flow chart of the processing process of an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0080] The principle of the technical solution of the present invention is to use 2×2 four grids as the basic imaging unit, take the red, green and blue color values ​​of the pixel to be restored as unknowns, establish a relationship equation based on the red, green and blue color values ​​and color composition relationship recorded by the Bayer sensor, and then obtain the equation solution expression to realize weighted interpolation, and then design the corresponding RGB and IHS conversion methods according to the invariant characteristics of hue and saturation in the imaging unit, and finally introduce two filters to enhance the image, and finally realize the super-resolution imaging of the Bayer sensor.

[0081] Figure 1This is a schematic diagram of the placement of the red, green, and blue photosensitive elements for Bayer sensor imaging. It can be seen that the red, green, and blue photosensitive elements are placed in alternate rows and columns respectively, and the cardinal rows of the resulting image are repeated arrangements of green and red, while the even rows are repeated arrangements of blue and green, that is, the four values ​​of green, red, blue, and green are sampled respectively in small blocks of 2 rows and 2 columns with a total of 4 pixels.

[0082] The technical process of the embodiment of the present invention is as follows Figure 4 As shown, the super-resolution imaging method of the Bayer sensor provided in the embodiment includes weighted interpolation processing, IHS conversion with unchanged hue and saturation, and filtering enhancement, which can be implemented by the following process steps:

[0083] Step 1: directly clone the nearest color values ​​for the pixels at the four corners of the image. Since the four pixels at the corners of the entire image cannot be interpolated and have little effect on the entire image, the processing can be simplified and the red, green, and blue color values ​​recorded in the 2×2 grid can be directly assigned to the four corner pixels of the image.

[0084] Step 2: weighted interpolation of the color value of each pixel in the image (all pixels except the four corners of the image).

[0085] In the embodiment, it is further proposed that any pixel except the four pixels at the most corners of the image is subjected to the following steps to implement weighted interpolation processing:

[0086] Step 2.1, such as Figure 2 As shown, the 2×2 four grids to which the pixel belongs are used as processing units, the red, green, and blue color values ​​recorded in the 2×2 four grids are marked as G, R, B, and GG, and the 2×2 four grids are marked as grid 1, grid 2, grid 3, and grid 4;

[0087] Because in the Bayer sensor, a small block of 4 pixels in 2 rows and 2 columns contains 4 values: green, red, blue, and green. In order to distinguish the two greens, the present invention marks G as the first green channel, R as the red channel, B as the blue channel, and GG as the second green channel.

[0088] Step 2.2, for the 2×2 four-grid grid 1, the recorded color is G. Assume that the color components to be restored are the red component R1 and the blue component B1. The color values ​​recorded for the two pixels closest to R1 and grid 1 are R and R1' respectively. For B1, the color values ​​recorded for the two pixels closest to grid 1 are B and B1' respectively. Figure 3 As shown, the average values ​​are recorded as avR1 (avR1 = (R + R1') / 2), avB1 (avB1 = (B + B1') / 2), assuming that the errors of the red component R1 and the blue component B1 in the color component to be restored are dR1 and dB1, then the following formula is obtained:

[0089] R1+dR1=R

[0090] B1+dB1=B

[0091] R1+dR1=avR1

[0092] B1+dB1=avB1

[0093] The joint solution can obtain the expression of the color component that needs to be restored:

[0094] R1+dR1=(R+avR1) / 2=(3R+R1') / 4

[0095] B1+dB1=(B+avB1) / 2=(3B+B1') / 4

[0096] Where R1' and B1' are the color values ​​of the pixels adjacent to the pixel in grid 1.

[0097] Step 2.3, for the 2×2 four-grid grid 2, grid 3, and grid 4, assume that the color components to be restored are green component G2, blue component B2, red component R3, green component G3, red component R4, and blue component B4, respectively. Use similar processing as grid 1, and the formulas used are:

[0098] G2+dG2=(G+avG2) / 2=(3G+G2') / 4

[0099] B2+dB2=(B+avB2) / 2=(3B+B2') / 4

[0100] R3+dR3=(R+avR3) / 2=(3R+R3') / 4

[0101] G3+dG3=(G+avG3) / 2=(3G+G3') / 4

[0102] R4+dR4=(R+avR4) / 2=(3R+R4') / 4

[0103] B4+dB4=(B+avB4) / 2=(3B+B4') / 4

[0104] Among them, G2' and B2' are the color values ​​of pixels adjacent to grid 2 pixel, R3' and G3' are the color values ​​of pixels adjacent to grid 3 pixel, R4' and B4' are the color values ​​of pixels adjacent to grid 4 pixel; avG2, avB2, avR3, avG3, avR4, avB4 are the average values ​​of the two nearest pixels of G2, B2, R3, G3, R4, B4 respectively, and dG2, dB2, dR3, dG3, dR4, dB4 are the corresponding errors of G2, B2, R3, G3, R4, B4.

[0105] Specifically, dG2 and dB2 are the errors of G2 and B2 respectively, dR3 and dG3 are the errors of R3 and G3 respectively, dR4 and dB4 are the errors of R4 and B4 respectively, avG2 is the average of G and G2', avB2 is the average of B and B2', avR3 is the average of R and R3', avG3 is the average of G and G3', avR4 is the average of R and R4', and avB4 is the average of B and B4'.

[0106] Step 3, re-optimization of the color value of the image pixel. Considering the 2×2 four-grid processing unit, the color components it actually receives are G, R, B, and GG. In theory, no matter how the interpolation is performed, the hue and saturation of the 2×2 four-grid processing unit should be kept consistent, otherwise the color will be cast. For this reason, the present invention proposes the following processing.

[0107] Step 3.1, using the 2×2 grid to which the pixel belongs as the processing unit, convert the original recorded G, R, and B into the IHS space, and record the components as brightness I, hue H, and saturation S. For fast calculation, the right-multiplied triangular approximate conversion matrix formula is used for processing, and the matrix of RGB to IHS is recorded as A, and its value is:

[0108]

[0109] The specific calculation formula is:

[0110] (I, H, S) = A × (R, G, B)

[0111] Expanded:

[0112] I=A[0][0]×R+A[0][1]×G+A[0][2]×B

[0113] H=A[1][0]×R+A[1][1]×G+A[1][2]×B

[0114] S=A[2][0]×R+A[2][1]×G+A[2][2]×B

[0115] The matrix converted from IHS to RGB is the inverse matrix of A, denoted by A -1 , since A is an orthogonal matrix, the inverse matrix is ​​its transposed matrix, denoted by A T , which is the matrix after swapping the rows and columns of A.

[0116] Among them, A[][] is the mathematical expression of the matrix, and [][] are the row and column numbers respectively.

[0117] The specific calculation formula is:

[0118] (R, G, B) = A T ×(I, H, S)

[0119] Expanded:

[0120] R=A[0][0]×I+A[1][0]×H+A[2][0]×S

[0121] G=A[0][1]×I+A[1][1]×H+A[2][1]×S

[0122] B=A[0][2]×I+A[1][2]×H+A[2][2]×S

[0123] Step 3.2, consistent with step 2.2, mark the 2×2 four grids as grid 1, grid 2, grid 3, grid 4, and do the following processing on grid 1: take the color components R1+dR1, G, B1+dB1 of grid 1 as row vectors, multiply them with matrix A and transform them into IHS space. The transformation results are recorded as I1, H1, and S1 respectively. To ensure the consistency of hue and saturation within the processing unit, H1 and S1 need to be directly replaced with H and S of the 2×2 four grids, and then I1, H, S and A are used to calculate the color space. T Multiply them to get new R1", G1", and B1", and the results are the three color components of the optimized grid 1. The specific calculation process is:

[0124] (I1, H1, S1) = A × (R1 + dR1, G, B1 + dB1)

[0125] (R1", G1", B1") = A T ×(I1, H, S)

[0126] Step 3.3, similarly, for grid 2, grid 3, grid 4, similar processing is applied to grid 1, and the optimized color components of grid 2, grid 3, grid 4 are recorded as R2", G2", B2", R3", G3", B3", R4", G4", B4". The specific calculation formula for grid 2 is:

[0127] (I2, H2, S2)=A×(R, G2+dG2, B2+dB2)

[0128] (R2", G2", B2") = A T ×(I2,H,S)

[0129] The specific calculation formula for cell 3 is:

[0130] (I3, H3, S3) = A × (R3 + dR3, G3 + dG3, B)

[0131] (R3", G3", B3") = A T ×(I3,H,S)

[0132] The specific calculation formula for cell 4 is:

[0133] (I4, H4, S4) = A × (R1 + dR1, GG, B1 + dB1)

[0134] (R4", G4", B4") = A T ×(I4,H,S)

[0135] Among them, (I2, H2, S2), (I3, H3, S3), and (I4, H4, S4) are the conversion results of color components of grids 2, 3, and 4 multiplied by matrix A and converted to IHS space.

[0136] Step 4: Enhancement of the color value of the image pixel. The previous steps 2 and 3 are essentially weighted downsampling of the Bayer sensor data, which is essentially high-pass filtering. In order to eliminate the impact of high-pass filtering on the image information, the image can be processed again as follows.

[0137] Step 4.1, set a low-pass filter to perform smoothing on the three color components respectively. The filter h preferably set in the embodiment of the present invention is:

[0138]

[0139] The specific calculation formula is:

[0140] p (i，j) =g (i-1，j-1) ×h (0，0) +g (i-1，j) ×h (0，1) +g (i-1，j+1) ×h (0，2) +

[0141] g (i，j-1) ×h (1，0) +g (i，j) ×h (1，1) +g (i，j+1) ×h (1，2) +

[0142] g (i+1，j-1) ×h (2，0) +g (i+1，j) ×h (2，1) +g (i+1，j+1) ×h (2，2)

[0143] Where h (i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, p (i，j) It is the processing result of the pixel with row number i and column number j.

[0144] Step 4.2, setting an edge enhancer to perform edge enhancement processing on the three color components respectively. The edge enhancer preferably set in the embodiment of the present invention is:

[0145]

[0146] The specific calculation formula is:

[0147] q (i，j) =g (i-1，j-1) ×f (0，0) +g (i-1，j) ×f (0，1) +g (i-1，j+1) ×f (0，2) +

[0148] g (i，j-1) ×f (1，0) +g (i，j) ×f (1，1) +g (i，j+1) ×f (1，2) +

[0149] g (i+1，j-1) ×f (2，0) +g (i+1，j) ×f (2，1) +g (i+1，j+1) ×f (2，2)

[0150] Where f (i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, q (i，j) It is the processing result of the pixel with row number i and column number j.

[0151] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0152] In some possible embodiments, a computer device / equipment / system is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned Bayer sensor super-resolution imaging method.

[0153] In some possible embodiments, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the super-resolution imaging method of the above-mentioned Bayer sensor are implemented.

[0154] In some possible embodiments, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned Bayer sensor super-resolution imaging method when executed by a processor.

[0155] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A Bayer sensor super-resolution imaging method, characterized in that: The data obtained by the Bayer sensor is processed based on weighted interpolation and hue and saturation to achieve Bayer sensor super-resolution imaging. The implementation process includes the following steps: Step 1: directly clone the nearest color values ​​of the pixels at the four corners of the image; Step 2, weighted interpolation of the color value of each other pixel of the image, including taking 2×2 four grids as the basic imaging unit, taking the red, green, and blue color values ​​of the pixel to be restored as unknowns, establishing a relationship equation based on the red, green, and blue color values ​​and color composition relationship recorded by the Bayer sensor, obtaining the equation solution expression, and realizing weighted interpolation; Step 3, re-optimization of the color value of the image pixel, including performing corresponding RGB and IHS conversion to achieve re-optimization according to the invariant characteristics of hue and saturation in the imaging unit; Step 4, enhancing the color value of the image pixel, including enhancing through a low-pass filter and an edge enhancer, to obtain a super-resolution imaging result of the corresponding pixel; In step 2, the following steps are used to implement weighted interpolation for each pixel in the image: Step 2.1, taking the 2×2 four grids to which the pixel belongs as the processing unit, marking the red, green, and blue color values ​​recorded in the 2×2 four grids as G, R, B, and GG, and marking the 2×2 four grids as grid 1, grid 2, grid 3, and grid 4; Step 2.2, for the 2×2 four-grid grid 1, its recorded color is G. Suppose the color components to be restored are the red component R1 and the blue component B1. For R1, the color values ​​recorded by the two pixels closest to grid 1 are R and R1', respectively, and their average value is recorded as avR1. For B1, the color values ​​recorded by the two pixels closest to grid 1 are B and B1', respectively, and their average value is recorded as avB1. Suppose the errors of the red component R1 and the blue component B1 in the color components to be restored are dR1 and dB1, then the weighted interpolation formula is: R1+dR1=(R+avR1) / 2=(3R+R1') / 4 B1+dB1=(B+avB1) / 2=(3B+B1') / 4 Among them, R1' and B1' are the color values ​​of the pixels adjacent to the pixel of grid 1; Step 2.3, for the 2×2 four-grid grid 2, grid 3, and grid 4, assume that the color components to be restored are green component G2, blue component B2, red component R3, green component G3, red component R4, and blue component B4, and the corresponding weighted interpolation formulas are: G2+dG2=(G+avG2) / 2=(3G+G2') / 4 B2+dB2=(B+avB2) / 2=(3B+B2') / 4 R3+dR3=(R+avR3) / 2=(3R+R3') / 4 G3+dG3=(G+avG3) / 2=(3G+G3') / 4 R4+dR4=(R+avR4) / 2=(3R+R4') / 4 B4+dB4=(B+avB4) / 2=(3B+B4') / 4 Among them, G2' and B2' are the color values ​​of pixels adjacent to grid 2 pixel, R3' and G3' are the color values ​​of pixels adjacent to grid 3 pixel, R4' and B4' are the color values ​​of pixels adjacent to grid 4 pixel; avG2, avB2, avR3, avG3, avR4, avB4 are the average values ​​of the two nearest pixels of G2, B2, R3, G3, R4, B4 respectively, and dG2, dB2, dR3, dG3, dR4, dB4 are the corresponding errors of G2, B2, R3, G3, R4, B4.

2. The super-resolution imaging method of a Bayer sensor according to claim 1, characterized in that: In step 3, the pixel color value is further optimized as follows: Step 3.1, using the 2×2 grid to which the pixel belongs as the processing unit, convert the original recorded G, R, and B into the IHS space, with the components recorded as brightness I, hue H, and saturation S; use the right-multiplied triangular approximate conversion matrix formula for processing, and record the RGB to IHS matrix as A, The calculation formula is: (I, H, S) = A × (R, G, B) Expanded: I=A[0][0]×R+A[0][1]×G+A[0][2]×B H=A[1][0]×R+A[1][1]×G+A[1][2]×B S=A[2][0]×R+A[2][1]×G+A[2][2]×B The matrix converted from IHS to RGB is the inverse matrix of A, denoted by A -1 , since A is an orthogonal matrix, the inverse matrix is ​​its transposed matrix, denoted by A T , the calculation formula is: (R,G,B)=A T ×(I,H,S) Expanded: R=A[0][0]×I+A[1][0]×H+A[2][0]×S G=A[0][1]×I+A[1][1]×H+A[2][1]×S B=A[0][2]×I+A[1][2]×H+A[2][2]×S Step 3.2, mark the four 2×2 grids as grid 1, grid 2, grid 3, grid 4, and do the following processing on grid 1: take the color components R1+dR1, G, B1+dB1 of grid 1 as row vectors, multiply them with matrix A and transform them into IHS space. The transformation results are recorded as I1, H1, and S1 respectively; to ensure the consistency of hue and saturation within the processing unit, directly replace H1 and S1 with H and S of the four 2×2 grids, and then use I1, H, S and A to convert them into IHS space. T Multiply them to get new R1", G1", and B1", as the three color components of the optimized grid 1. The calculation process is as follows: (I1, H1, S1) = A × (R1 + dR1, G, B1 + dB1) (R1”,G1”,B1”)=A T ×(I1,H、S) Step 3.3, similarly process grid 2, grid 3, grid 4, and obtain the optimized color components of grid 2, grid 3, grid 4, recorded as R2", G2", B2", R3", G3", B3", R4", G4", B4"; The calculation formula for cell 2 is: (I2, H2, S2)=A×(R, G2+dG2, B2+dB2) (R2”,G2”,B2”)=A T ×(I2,H、S) The calculation formula for cell 3 is: (I3, H3, S3) = A × (R3 + dR3, G3 + dG3, B) (R3”, G3”, B3”)=A T ×(I3, H, S) The calculation formula for cell 4 is: (I4, H4, S4) = A × (R1 + dR1, GG, B1 + dB1) (R4”, G4”, B4”)=A T ×(I4, H, S) Among them, (I2, H2, S2), (I3, H3, S3), and (I4, H4, S4) are the conversion results of color components of grids 2, 3, and 4 multiplied by matrix A and converted to IHS space.

3. The super-resolution imaging method of a Bayer sensor according to claim 1, characterized in that: In step 4, the method for enhancing the color value of image pixels is: Step 4.1, design a low-pass filter h to smooth the three color components respectively, the calculation formula is: p (i，j) =g (i-1，j-1) ×h (0，0) +g (i-1，j) ×h (0，1) +g (i-1，j+1) ×h (0，2) + g (i，j-1) ×h (1，0) +g (i，j) ×h (1，1) +g (i，j+1) ×h (1，2) + g (i+1，j-1) ×h (2，0) +g (i+1，j) ×h (2，1) +g (i+1，j+1) ×h (2，2) In the formula, h (i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, p (i，j) is the processing result of the pixel with row number i and column number j; Step 4.2, set an edge enhancer to perform edge enhancement processing on the three color components respectively. The specific calculation formula is: q (i，j) =g (i-1，j-1) ×f (0，0) +g (i-1，j) ×f (0，1) +g (i-1，j+1) ×f (0，2) + g (i，j-1) ×f (1，0) +g (i，j) ×f (1，1) +g (i，j+1) ×f (1，2) + g (i+1，j-1) ×f (2，0) +g (i+1，j) ×f (2，1) +g (i+1，j+1) ×f (2，2) In the formula, f (i，j) is the element of the filter, i and j are the element numbers, and g (i，j) is the value of the R, G, and B components of the image pixel with row number i and column number j, q (i，j) It is the processing result of the pixel with row number i and column number j.

4. A Bayer sensor super-resolution imaging method as claimed in claim 1, 2 or 3, characterized in that: The super-resolution imaging process is completed using DSP chips.

5. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the Bayer sensor super-resolution imaging method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the super-resolution imaging method of the Bayer sensor according to any one of claims 1 to 4 are implemented.

7. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by a processor, the steps of the super-resolution imaging method of the Bayer sensor according to any one of claims 1 to 4 are implemented.

Images