Image sensors and electronics

Abstract

An image sensor and an electronic device are provided, which can improve the performance of the image sensor. The image sensor includes: a filter unit array, including a plurality of filter unit groups, each of the plurality of filter unit groups includes 4×4 filter units; wherein, in the 4×4 filter units, each row, each column and each diagonal line includes 2 white filter units and 2 color filter units; a pixel unit array, including a plurality of pixel units, the pixel unit array is located below the filter unit array, and a plurality of pixel units in the pixel unit array correspond one to one with a plurality of filter units in the filter unit array. Through this solution, the white filter units and the color filter units are evenly distributed in the filter unit array, which is not only convenient for subsequent image algorithms to perform image processing, but also conducive to image color restoration and avoidance of color moiré fringes, and can further improve image quality parameters such as SNR and resolution of the image.

Description

[0001] This application claims priority to a Chinese invention application filed with the China Patent Office on May 15, 2020, with application number 202010410639.2 and invention name “Image sensor and electronic device”, all contents of which are incorporated into this application by application. Technical Field

[0002] The present application relates to the field of sensors, and more particularly, to an image sensor and an electronic device. Background Art

[0003] An image sensor is an electronic device that converts an optical image into a digital signal. It usually includes a pixel array composed of multiple pixel units, each of which is used to form a pixel value in the image. In order to enable the image sensor to capture color images, a color filter (CF) can be set above the pixel unit so that the pixel unit can receive a light signal of a specific color and form a pixel value corresponding to the light signal of the specific color.

[0004] However, when the color filter is set, the amount of light received by each pixel unit decreases, resulting in a smaller signal-to-noise ratio (SNR), which affects the image quality. If the image sensor is used in a mobile device, the size of the image sensor is limited, and the corresponding photosensitive area of ​​the pixel array is also limited, so in a low-light environment, the image quality will be further limited. Summary of the invention

[0005] The embodiments of the present application provide an image sensor and an electronic device, which are intended to solve the problem of poor image quality acquired by the image sensor in a low-light environment.

[0006] In a first aspect, an image sensor is provided, comprising: a filter unit array, comprising a plurality of filter unit groups, each of the plurality of filter unit groups comprising 4×4 filter units; wherein, in the 4×4 filter units, each row, each column and each diagonal line comprises 2 white filter units and 2 color filter units;

[0007] The pixel unit array comprises a plurality of pixel units. The pixel unit array is located below the filter unit array, and the plurality of pixel units in the pixel unit array correspond one to one with the plurality of filter units in the filter unit array.

[0008] Through the scheme of the embodiment of the present application, since each filter unit group includes 2 white filter units and 2 color filter units in each row, each column and each diagonal line, the filter unit array formed by the multiple filter unit groups also includes two types of filter units, namely white filter units and color filter units, in each row, each column and diagonal direction. The white filter units and the color filter units are evenly distributed in the filter unit array, which not only facilitates the subsequent image algorithm to perform image processing, is beneficial to image color restoration, reduces image color loss, avoids the generation of color moiré fringes, but also can obtain more accurate full-resolution grayscale images, further improving image quality parameters such as SNR and resolution.

[0009] In some possible implementations, the filter unit group includes first color filter units, second color filter units, and third color filter units of different colors, and the number of the first color filter units is equal to the sum of the number of the second color filter units and the third color filter units.

[0010] In some possible implementations, in the filter unit group, the number of the second color filter units is equal to the number of the third color filter units.

[0011] In some possible implementations, in the filter unit group, the two color filter units in each row have different colors, and the two color filter units in each column have different colors.

[0012] In some possible implementations, in each row of the filter unit group, two white filter units and two color filter units are arranged alternately; in each column of the filter unit group, two white filter units are arranged continuously, and two color filter units are arranged continuously.

[0013] In some possible implementations, color filter units of the same color in the filter unit group are arranged in a centrally symmetrical manner.

[0014] In some possible implementations, two target filter unit sets are arranged in the two middle rows of the filter unit group, and the target filter unit sets include two color filter units of the same color arranged at the same vertex angle.

[0015] In some possible embodiments, in the filter unit group, the white filter unit is respectively located in the first row and the second column, the first row and the fourth column, the second row and the second column, the third row and the first column, the third row and the third column, the fourth row and the first column, and the fourth row and the third column; the second color filter unit is respectively located in the second row and the first column and the third row and the second column; the first color filter unit is respectively located in the first row and the first column, the second row and the third column, the third row and the fourth column, and the third color filter unit is respectively located in the first row and the third column and the fourth row and the fourth column.

[0016] In some possible embodiments, in each row of the filter unit group, two white filter units and two color filter units are arranged alternately; in two columns of the filter unit group, two white filter units are arranged continuously and two color filter units are arranged separately; in the other two columns of the filter unit group, two white filter units are arranged separately and two color filter units are arranged continuously.

[0017] In some possible implementations, two target filter unit sets are arranged in the middle two columns of the filter unit group, and the target filter unit sets include two color filter units of the same color arranged at the same vertex angle.

[0018] In some possible implementations, four target filter unit sets are arranged in the filter unit group, and the target filter unit sets include two color filter units of the same color arranged at a common vertex angle.

[0019] In some possible embodiments, in the filter unit group, the white filter unit is respectively located in the first row and second column, the first row and fourth column, the second row and first column, the second row and third column, the third row and third column, the third row and third column, the fourth row and fourth column; the second color filter unit is respectively located in the first row and third column and the second row and fourth column; the first color filter unit is respectively located in the first row and first column, the second row and second column, the third row and fourth column, and the fourth row and third column; the third color filter unit is respectively located in the third row and first column and the fourth row and first column.

[0020] In some possible implementations, the first color filter unit, the second color filter unit, and the third color filter unit are used to pass three color light signals respectively, and the wavelength bands of the three color light signals cover the visible light band.

[0021] In some possible implementations, colors of the first color filter unit, the second color filter unit, and the third color filter unit are three colors of red, green, blue, cyan, magenta, and yellow, respectively.

[0022] In some possible implementations, the first color filter unit is a green filter unit, and the second color filter unit and the third color filter unit are a red filter unit and a blue filter unit, respectively.

[0023] In some possible embodiments, the image sensor further includes: a microlens array, including a plurality of microlenses, the microlens array being located above the filter unit array and being used to converge the light signal returned by the photographed object to the filter unit array, wherein the plurality of microlenses in the microlens array correspond one-to-one to the plurality of filter units in the filter unit array.

[0024] In some possible embodiments, the pixel values ​​of the white pixel units in the pixel array are used to generate first image data of the object, and the pixel values ​​of the color pixel units in the pixel array are used to generate second image data of the object, and the first image data and the second image data are used to synthesize the target image of the object; wherein the white pixel unit is a pixel unit corresponding to the white filter unit, and the color pixel unit is a pixel unit corresponding to the color filter unit.

[0025] In some possible implementations, the pixel values ​​of the color pixel units in the pixel array are used to generate an intermediate image through an interpolation process, and the intermediate image is used to generate the second image data in a Bayer format through a Remosaic process.

[0026] In some possible implementations, among the 2×2 pixel values ​​of the intermediate image, 2 pixel values ​​are original pixel values ​​of the color pixel unit, and the other 2 pixel values ​​are pixel values ​​obtained through interpolation processing.

[0027] In some possible implementations, the first image data and the second image data have the same resolution.

[0028] In some possible implementations, the image sensor is a complementary metal oxide semiconductor (CMOS) image sensor, or a charge coupled device (CCD) image sensor.

[0029] In a second aspect, an electronic device is provided, comprising: an image sensor according to the above-mentioned first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the structure of an image sensor provided in an embodiment of the present application.

[0031] Figure 2 is a schematic top view of another image sensor provided in an embodiment of the present application.

[0032] Figure 3 yes Figure 2 A schematic cross-sectional view of the image sensor along the AA' direction.

[0033] Figure 4 yes Figure 2 Another schematic cross-sectional view of the image sensor along the AA' direction.

[0034] Figure 5 It is a schematic arrangement diagram of filter units in a filter unit array provided in an embodiment of the present application.

[0035] Figures 6 to 14It is a schematic arrangement diagram of the filter units in several filter unit groups provided in the embodiments of the present application.

[0036] Fig.15 It is a schematic flowchart of an image processing method provided in an embodiment of the present application.

[0037] Fig.16 yes Fig.15 Image schematic diagram of the image processing method in . DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0039] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application.

[0040] It should also be understood that the various implementation methods described in this specification can be implemented individually or in combination, and the embodiments of the present application are not limited to this.

[0041] The technical solution of the embodiments of the present application can be applied to various image sensors, such as complementary metal oxide semiconductor (CMOS) image sensor (CIS) or charge coupled device (CCD) image sensor, but the embodiments of the present application are not limited to this.

[0042] As a common application scenario, the image sensor provided in the embodiments of the present application can be applied to smart phones, cameras, webcams, tablet computers and other mobile terminals or other terminal devices with imaging functions.

[0043] Figure 1 FIG. 1 shows a schematic diagram of the structure of an image sensor. Figure 1 As shown, the image sensor 100 includes: a pixel unit array 110, a row selection circuit 120, a column selection circuit 130, a control circuit 140, and an analog to digital converter (ADC) circuit 150, a front-end signal processing circuit 160 and a back-end signal processing circuit 170.

[0044] Specifically, Figure 1As shown, a plurality of square pixel units in the pixel unit array 110 are arranged into M rows×N columns, where M and N are positive integers. Generally speaking, on the plane where the pixel unit array 110 is located, the row direction of the M rows and the column direction of the N columns are perpendicular to each other. In some cases, for the convenience of description, in a plane, two perpendicular directions, such as the row direction and the column direction in this application, may be referred to as the horizontal direction and the vertical direction.

[0045] exist Figure 1 In the pixel unit array 110 shown, any side of each square pixel unit is parallel or perpendicular to the row direction or the column direction.

[0046] Optionally, the pixel unit may include devices such as a photodiode and a field effect switch tube, which are used to receive an optical signal and convert the optical signal into a corresponding electrical signal.

[0047] Optionally, if the image sensor needs to capture a color image, a color filter array (CFA) may be provided above the pixel unit array 110, wherein a color filter unit may be provided above each pixel unit. For the sake of description, the pixel unit provided with a color filter unit above is also referred to as a color pixel unit hereinafter. For example, the pixel unit provided with a red filter unit above is referred to as a red pixel unit ( Figure 1 The pixel unit with a green filter unit on top is called a green pixel unit ( Figure 1 The pixel unit with a blue filter unit on top is called a blue pixel unit ( Figure 1 Indicated by B).

[0048] Currently, most image sensors use a Bayer format based on the three primary colors RGB. Figure 1 As shown, a Bayer format CFA is arranged above the pixel unit array 110, and the pixel unit array 110 uses 2×2 pixel units as basic units, each basic unit includes 1 red pixel unit, 1 blue pixel unit, and 2 green pixel units, wherein the 2 green pixel units are arranged adjacent to each other at the same vertex angle.

[0049] The row selection circuit 120 is connected to each row of pixel units in the pixel unit array 110 through M row control lines, and can be used to turn on and off each pixel unit in each row of pixel units. For example, the row selection circuit 120 is connected to the gate of the field effect switch tube of each pixel unit in the first row of the pixel unit array 110 through a row control line, and controls the working state of the photodiode by turning on or off the field effect switch tube. Among them, the M row control lines are parallel to the horizontal direction.

[0050] The column selection circuit 130 is connected to each column of pixel units in the pixel unit array 110 through N column control lines, and can be used to select the signal value output of each pixel unit in each column. For example, the column selection circuit 130 is connected to the source of the field effect switch tube of each pixel unit in the first column of the pixel unit array 110 through a column control line to control the output of the electrical signal converted by the photodiode. Among them, the N column control lines are parallel to the vertical direction.

[0051] The control circuit 140 is connected to the row selection circuit 120 and the column selection circuit 130, and is used to provide timing to the row selection circuit 120 and the column selection circuit 130, control the row selection circuit 120 and the column selection circuit 130 to select a pixel unit in the pixel unit array 110, and output the pixel value of the pixel unit.

[0052] Optionally, after the row selection circuit 120, column selection circuit 130 and control circuit 140 cooperate to output the pixel values ​​generated by the pixel unit array 110, the pixel values ​​of the pixel unit array 110 are transmitted to the ADC circuit 150 for analog-to-digital conversion, and the analog pixel values ​​are converted into digital pixel values ​​to form a digital image, so as to facilitate the subsequent signal processing circuit 160 to perform image processing to output an optimized color image.

[0053] Optionally, the signal processing circuit 160 may include, but is not limited to, an image signal processor (ISP), which is used to perform linearization processing, bad pixel removal, noise removal, color correction, demosaicing, automatic exposure control (AEC), automatic gain control (AGC), automatic white balance (AWB), etc. on digital images.

[0054] For the above-mentioned image sensor 100 using the Bayer format CFA, the red pixel unit can only receive red light signals, the green pixel unit can only receive green light signals, and the blue pixel unit can only receive blue light signals. The intensity of the light signal received by each pixel unit is relatively small, resulting in a large SNR of the image, thereby affecting the image quality.

[0055] Moreover, for the image sensor of the Bayer format CFA, the high-frequency information of brightness and the chromaticity information in the image are likely to overlap, resulting in color aliasing and color moire.

[0056] Based on the above problems, the present application proposes an image sensor, which adds a white filter unit in the CFA, and some pixel units in the pixel unit array receive color light signals, and some pixel units receive white light signals, thereby increasing the intensity of light signals received by some pixel units. On this basis, the pixel values ​​of multiple pixel units in the pixel unit array are processed. On the basis of ensuring the image color information, the image quality parameters such as SNR and resolution are improved to obtain an optimized color image.

[0057] Figure 2 is a schematic top view of an image sensor 200 provided in an embodiment of the present application, Figure 3 is a schematic cross-sectional view of the image sensor 200 along the AA′ direction.

[0058] like Figure 2 and Figure 3 As shown, the image sensor 200 includes:

[0059] The filter unit array 210 includes a plurality of filter unit groups 211, each of the plurality of filter unit groups 211 includes 4×4 filter units;

[0060] In the 4×4 filter units, each row, each column and each diagonal line includes 2 white filter units and 2 color filter units.

[0061] The pixel unit array 220 is located below the filter unit array 210 and includes a plurality of pixel units. The plurality of pixel units in the pixel unit array 220 correspond one-to-one to the plurality of filter units in the filter unit array 210 .

[0062] In a possible implementation, Figure 3 As shown, multiple filter units in the filter unit array 210 can be set on the upper surface of multiple pixel units in the pixel unit array 220; in another possible implementation, multiple filter units in the filter unit array 210 can also be suspended above multiple pixel units in the pixel unit array 220.

[0063] In addition, if Figure 3 As shown, as an example, each filter unit in the filter unit array 210 is correspondingly arranged directly above each pixel unit in the pixel unit array 220. In other words, the center of each filter unit coincides with the center of its corresponding pixel unit in the vertical direction. In addition to this method, each filter unit in the filter unit array 210 is correspondingly arranged obliquely above each pixel unit in the pixel unit array 220. At this time, each pixel unit in the pixel unit array 220 can receive an optical signal in an oblique direction. The specific position of the filter unit array 210 is not limited in the embodiment of the present application.

[0064] The pixel unit corresponding to the color filter unit in the pixel unit array 220 is used to receive the color light signal passing through the color filter unit, and outputs the color pixel value accordingly; the pixel unit corresponding to the white filter unit in the pixel unit array 220 is used to receive the white light signal passing through the white filter unit, and outputs the white pixel value accordingly; the color light signal and the white light signal are used together to generate the target image of the photographed object. For example, the pixel unit corresponding to the red filter unit receives the red light signal, and the corresponding output pixel value can be called the red pixel value; the pixel unit corresponding to the white filter unit receives the white light signal, and the corresponding output pixel can be called the white pixel value.

[0065] Furthermore, if Figure 4 As shown, Figure 4 FIG. 2 shows a schematic cross-sectional view of another image sensor 200 along the AA′ direction.

[0066] like Figure 4 As shown, in addition to the above-mentioned filter unit array 210 and pixel unit array 220, the image sensor 200 also includes:

[0067] The microlens array 230 is arranged above the filter unit array 210, and is used to converge the light signal returned by the photographed object to the filter unit array 210 and reduce the crosstalk of the light signal between adjacent pixel units, wherein the microlens array 230 includes a plurality of microlenses, and the plurality of microlenses correspond one-to-one to the plurality of filter units in the filter unit array 210 and the plurality of pixel units in the pixel unit array 220.

[0068] In some embodiments, the pixel structure in the image sensor may be referred to as an on-chip microlens (OCL) pixel structure.

[0069] Alternatively, if Figure 4 As shown, in the image sensor 200 , a dielectric layer 240 may be further included between the filter unit array 210 and the pixel unit array 220 , for connecting the pixel unit array 220 and the filter unit array 210 .

[0070] In addition, the filter unit array 210 may further include a dielectric 215 and a reflective grid 216 located around it; the reflective grid is used to reflect optical signals incident at a large angle to prevent optical signal loss.

[0071] The pixel unit array 220 may include a semiconductor substrate 221 and a photosensitive element 222, wherein the photosensitive element 222 is located in the semiconductor substrate 221, and the photosensitive element 222 includes but is not limited to a photodiode (PD). Optionally, the pixel unit array 220 may also include an isolation region 223 between two photosensitive elements 222 to prevent electrical signal interference between two adjacent photosensitive elements.

[0072] It is understandable that, in addition to Figure 4 In addition to the basic structure shown in , the image sensor 200 may also include other stacked structures, such as at least one metal interconnection layer to electrically connect multiple pixel units in the pixel unit array, etc. The structure of the image sensor in the embodiment of the present application is not limited to this.

[0073] Alternatively, if Figure 2 The top view shown is actually a schematic diagram of the arrangement of a filter unit array 210 provided in an embodiment of the present application.

[0074] like Figure 2 As shown, in the embodiment of the present application, each filter unit in a filter unit group 211 is a quadrilateral filter unit, for example, they can all be square filter units, and 16 square filter units form a square filter unit group. In the filter unit group 211, the number of color filter units (as shown by the shaded blocks in the figure) and white filter units (as shown by the blank blocks in the figure) is equal, that is, it includes 8 color filter units and 8 white filter units. In the plane where the image sensor is located, a filter unit group includes two types of filter units, namely white filter units and color filter units, in the horizontal direction (row direction), vertical direction (column direction) and 45° direction (diagonal direction).

[0075] It should be noted that in the present application, the white filter unit refers to a filter or filter material for transmitting white light. In some embodiments, the white filter unit may also be a transparent material or an air gap, which is used to transmit all light signals in the environment including white light. Specifically, the white light may be a mixture of multiple colored lights. For example, the three primary colors of light in the spectrum: blue, red and green, can be mixed in a certain proportion to obtain white light, or a mixture of all visible light in the spectrum is also white light.

[0076] Correspondingly, the color filter unit refers to a filter or filter material for transmitting colored light. Specifically, the colored light can be a light signal of any wavelength range in the visible light spectrum. For example, the red filter unit can be used to transmit red light, and the red light can be a light signal with a wavelength range of 620nm to 750nm in the visible light spectrum. Similarly, color filter units of other colors are also used to transmit light signals of corresponding colors.

[0077] Through the scheme of the embodiment of the present application, since each filter unit group includes 2 white filter units and 2 color filter units in each row, each column and each diagonal line, the filter unit array formed by the multiple filter unit groups also includes two types of filter units, namely white filter units and color filter units, in each row, each column and diagonal direction. The white filter units and the color filter units are evenly distributed in the filter unit array, which not only facilitates the subsequent image algorithm to perform image processing, is beneficial to image color restoration, reduces image color loss, avoids the generation of color moiré fringes, but also can obtain more accurate full-resolution grayscale images, further improving image quality parameters such as SNR and resolution.

[0078] Optionally, in some embodiments, the filter unit group may include three color filter units (a first color filter unit, a second color filter unit, and a third color filter unit), for example, it may be a three-primary color filter unit, i.e., a filter unit of three colors of red, green, and blue (red green blue, RGB), or it may be a three-complementary color filter unit, i.e., a filter unit of three colors of cyan, magenta, and yellow (cyan magenta yellow, CMY), or it may be a filter unit of one primary color and two complementary colors, or one complementary color and two primary colors.

[0079] When the filter unit group includes color filter units of three colors, the three color light signals passing through the color filter units of three colors can cover the visible light band. The embodiment of the present application does not limit the specific colors of the color filter units.

[0080] For the convenience of description, the following description is made by taking the color filter units including red, green and blue colors in the filter unit group 211 as an example. When the filter unit group includes color filter units of other three colors, please refer to the technical solution below.

[0081] Figure 5 A schematic arrangement diagram of filter units in a filter unit group 211 is shown.

[0082] like Figure 5 As shown, the filter unit group 211 includes a white filter unit 201, a red filter unit 202, a green filter unit 203 and a blue filter unit 204. Optionally, in one filter unit group 211, the ratio of the number of white filter units 201, the number of green filter units 203, the number of red filter units 202 and the number of blue filter units 204 is 4:2:1:1.

[0083] Optionally, in the filter unit group 211, the white filter units and the color filter units in each row are arranged alternately, and the two white filter units and the two color filter units in each column are arranged continuously.

[0084] As an example, Figure 5 As shown, the two white filter units 201 in the 1st and 2nd rows are alternately arranged in the 2nd and 4th columns, and the two white filter units 201 in the 3rd and 4th rows are alternately arranged in the 1st and 3rd columns. Correspondingly, the two white filter units 201 in the 1st and 3rd columns are continuously arranged in the 3rd and 4th rows, and the two white filter units 201 in the 2nd and 4th columns are continuously arranged in the 1st and 2nd rows.

[0085] Optionally, the colors of the two color filter units in each row and the colors of the two color filter units in each column are different.

[0086] For example, in Figure 5 In the figure, the two color filter units in the first and fourth rows are the green filter unit 203 and the blue filter unit 204, and the two color filter units in the second and third rows are the red filter unit 202 and the green filter unit 203. Correspondingly, the two color filter units in the first and second columns are the green filter unit 203 and the red filter unit 202, and the two color filter units in the third and third columns are the blue filter unit 204 and the green filter unit 203.

[0087] Specifically, in Figure 5 In the filter unit group shown in FIG. 1 , two red filter units 202 may be located at the second row and the first column and the third row and the second column, respectively. The two red filter units 202 may be represented as R 21 and R 32 ;

[0088] The four green filter units 203 may be located at the first row and the first column, the second row and the third column, the third row and the fourth column, and the fourth row and the second column, respectively. The four green filter units 203 may be represented as G 11 , G 23 , G 34 and G 42 ;

[0089] The two blue filter units 204 may be located at the 3rd column of the 1st row and the 4th column of the 4th row respectively. The two blue filter units 204 may be represented as B 13 and B 44 .

[0090] Optionally, based on Figure 5The position of the white filter unit 201 remains unchanged. According to the requirement that the two color filter units in each row and the two color filter units in each column have different colors, Figure 5 The positions of the color filter units of different colors are changed.

[0091] For example, based on Figure 5 In the filter unit group, the positions of the white filter unit and the green filter unit are kept unchanged, and the positions of the red filter unit and the blue filter unit are changed.

[0092] Optionally, in some embodiments, the filter unit group includes at least two target filter unit sets, and two color filter units of the same color in each target filter unit set are arranged at a common vertex angle.

[0093] If the above conditions are met, Figure 6 Schematic arrangement diagrams of filter units in several transformed filter unit groups are shown.

[0094] exist Figure 5 and Figure 6 In the various filter unit groups shown in the figure, two target filter unit sets are located in the two middle rows, one of which is a set of two green filter units arranged at a common vertex angle, and the other is a set of two blue filter units or red filter units arranged at a common vertex angle.

[0095] For example, based on Figure 5 In the filter unit group, only the position of the white filter unit is kept unchanged, and the positions of the green filter unit, the red filter unit and the blue filter unit are changed.

[0096] Optionally, in some other embodiments, color filter units of the same color in the filter unit group are arranged in a centrally symmetrical manner.

[0097] If the above conditions are met, Figure 7 Schematic arrangement diagrams of filter units in several transformed filter unit groups are shown.

[0098] In the above Figures 5 to 7 Among the various filter unit groups shown, the filter unit group includes at least two target filter unit sets, or the color filter units of the same color are arranged in a centrally symmetrical manner, which facilitates the subsequent image pixel interpolation processing and can retain more original image color information, thereby increasing the accuracy of color restoration. The specific pixel interpolation process will be described in detail below.

[0099] It can be understood that, for each filter unit group in the above application embodiments, the filter unit group after geometric transformation, such as rotation, is also within the protection scope of the present application.

[0100] For example, for Figures 5 to 7 The filter unit group shown in FIG. 1 can be formed by rotating it 90° clockwise. Figure 8 Various filter element groups are shown.

[0101] Understandably, Figures 5 to 7 The filter unit group and Figure 8 The filter unit group in can be equivalent to the same filter unit group, the only difference is Figures 5 to 7 The arrangement of the filter units in the middle row direction is Figure 8 The arrangement rules of the filter units in the middle column direction are the same, correspondingly, Figures 5 to 7 The arrangement of the filter units in the middle column direction is Figure 8 The arrangement rules of the filter units in the middle row direction are the same.

[0102] like Figure 8 As shown, in each column of the filter unit group, two white filter units and two color filter units are arranged alternately; in each row of the filter unit group, two white filter units are arranged continuously, and two color filter units are arranged continuously.

[0103] It is also understandable that, in addition to Figure 8 In addition to the filter element group shown above Figures 5 to 7 Any filter unit group formed by rotating at other angles or undergoing other geometric transformations also falls within the protection scope of the present application and will not be listed one by one here.

[0104] The filter unit groups in the above embodiments are based on Figure 5 In addition to the structures listed above, the filter unit group of the present application can also be Figures 9 to 11 The filter unit group 211 is shown in several dotted boxes.

[0105] Understandably, Figures 9 to 11 The filter unit group 211 and Figure 2 The filter unit group 211 in the filter unit array 210 can form a filter unit array with the same central area, and the difference is only that the arrangement of the filter units in one or two outermost circles of the filter unit array 210 is different. Figure 2 , Figures 9 to 11 The filter unit array formed by the filter unit group 211 in the embodiment is equivalent to the same filter unit array.

[0106] Therefore, in this application, Figure 2 Any 4×4 filter units in the filter unit array 210 are divided into a filter unit group, and the filter unit group in any division situation is within the protection scope of the present application.

[0107] Similarly, a filter unit array formed by any filter unit group in the present application is also within the protection scope of the present application.

[0108] See also Fig. 9 The filter unit group shown in Fig. 9 and Figure 5 By comparison, it can be seen that Figure 5 The positions of the white filter unit and the color filter unit are swapped to form Fig. 9 The form of the filter unit group shown in .

[0109] Similar to the above implementation, based on Fig. 9 The position of the white filter unit in the middle remains unchanged, so Fig. 9 By changing the positions of the color filter units of different colors in Fig.12 The various filter element groups shown in .

[0110] For example, Fig.12 As shown in Figures (a) to (c) of FIG. 1 , the filter unit groups include two target filter unit sets. Fig.12 As shown in Figures (d) to (g), in the several filter unit groups, the color filter units of the same color are arranged in a centrally symmetrical manner.

[0111] Further, Fig. 9 and Fig.12 The filter unit group shown in (a) to (g) is rotated 90° clockwise to form Fig.12 The filter unit group shown in Figures (a') to (h').

[0112] Similarly, Fig. 9 as well as Fig.12 The filter unit group structures obtained by performing other rotation transformations or geometric transformations on the various filter unit groups are also within the protection scope of the present application.

[0113] See also Fig.10 and Fig.11 In the filter unit group shown in the two embodiments, in each row of the filter unit group, two white filter units and two color filter units are arranged alternately; in two columns of the filter unit group, two white filter units are arranged continuously and two color filter units are arranged separately; in the other two columns of the filter unit group, two white filter units are arranged separately and two color filter units are arranged continuously.

[0114] contrast Fig.10 and Fig.11 ,Will Fig.10 By exchanging the white filter unit and the color filter unit, we can get Fig.11 The filter unit group in.

[0115] based on Fig.10 The position of the white filter unit in the middle remains unchanged, so Fig.10 By changing the positions of the color filter units of different colors in Fig.13 The various filter element groups shown in .

[0116] Similarly, based on Fig.11 The position of the white filter unit in the middle remains unchanged, so Fig.11 By changing the positions of the color filter units of different colors in Fig.14 The various filter element groups shown in .

[0117] like Fig.10 , Fig.13 Figures (a) to (c) and Fig.14 As shown in Figures (d) to (g), in the filter unit groups, the color filter units are arranged into four target filter unit sets.

[0118] like Fig.11 , Fig.13 Figures (d) to (g) and Fig.14 As shown in Figures (a) to (c), the several filter unit groups include two target filter unit sets, and the two target unit sets include a green filter unit set.

[0119] Further, Fig.10 and Fig.13 The filter unit group shown in (a) to (g) is rotated 90° clockwise to form Fig.13 The filter unit group shown in (a') to (h') in FIG. Fig.11 and Fig.14 The filter unit group shown in (a) to (g) is rotated 90° clockwise to form Fig.14 The filter unit group shown in Figures (a') to (h').

[0120] Similarly, Fig.10 , Fig.11 , Fig.13 and Fig.14 The filter unit group structures obtained by performing other rotation transformations or geometric transformations on various filter unit groups are also within the protection scope of the present application.

[0121] Specifically, Fig.13 Figures (a') to (h') and Fig.14As shown in (a') to (h') in FIG. 1 , in each column of the filter unit group, two white filter units and two color filter units are arranged alternately; correspondingly, in two rows of the filter unit group, two white filter units are arranged continuously and two color filter units are arranged separately; in the other two rows of the filter unit group, two white filter units are arranged separately and two color filter units are arranged continuously.

[0122] Combination of the above Figures 2 to 14 The basic structure of the image sensor 200 in the present application and the arrangement diagram of the various filter unit groups 211 therein are introduced below. Fig.15 and Fig.16 An image processing method for the image sensor 200 of the present application is introduced.

[0123] Fig.15 A schematic flow chart of an image processing method is shown. Fig.16 Shows the Fig.15 Image schematic diagram of the image processing method in .

[0124] like Fig.15 As shown, the image processing method 10 includes:

[0125] S110: Sub-sampling the image formed by the pixel unit array to obtain a first sampling map including color pixel values ​​and a second sampling map including white pixel values.

[0126] As an example, Fig.16 The 1# image in the figure is an image generated by a 4×4 pixel unit in the pixel unit array. The 4×4 pixel unit corresponds to the above Fig.10 The pixel unit of the filter unit group is shown.

[0127] exist Fig.16 In Figure 1#, the white pixel value 101 is the pixel value generated after the white pixel unit receives the white light signal, and the white filter unit 201 is correspondingly arranged above it, the red pixel value 102 is the pixel value generated after the red pixel unit receives the red light signal, and the red filter unit 202 is correspondingly arranged above it; similarly, the green pixel value 103 is the pixel value generated after the green pixel unit receives the green light signal, and the green filter unit 203 is correspondingly arranged above it, and the blue pixel value 104 is the pixel value generated after the blue pixel unit receives the blue light signal, and the blue filter unit 204 is correspondingly arranged above it.

[0128] Fig.16 The 2# image in the figure is the first sampling image after subsampling the red pixel value, green pixel value and blue pixel value in the 1# image. Fig.16The 3# image in the figure is a second sampling image after subsampling the white pixel values ​​in the 1# image. In the first sampling image and the second sampling image, the relative position relationship of the pixel values ​​is consistent with the relative position relationship of the pixel values ​​in the 1# image of the original image.

[0129] S120: Perform interpolation processing on the first sampling graph and the second sampling graph to obtain a first image and a second image.

[0130] As an example, Fig.16 The 4# image in FIG. 4 is the first image after the first sampling image (2# image) is interpolated. Specifically, the pixel value of the i-th row and j-th column in the 4# image can be expressed as X ij , where X represents the color information of the pixel value. For example, the green pixel value in the first row and first column in the upper left corner can be represented as G 11 , according to G 11 and G 22 , interpolation is G 12 and G 21 , similarly, according to R 13 and R 24 , interpolation is R 14 and R 23 According to B 32 and B 41 , interpolation is B 31 and B 42 According to G 34 and G 43 , interpolation is G 33 and G 44 , thereby obtaining a first image by interpolating the pixel values ​​in the first sampling image.

[0131] It can be seen that in the first image, half of the pixel values ​​(as shown in the dotted box in the figure) retain the color information in the original image, which is more conducive to color restoration in the subsequent image processing process and improves the image quality. In addition, the interpolation process is relatively simple, which can simplify the algorithm in the image processing process and improve the efficiency of image processing.

[0132] Fig.16 Figure 5# is the second image after the second sampling image (Figure 3#) is interpolated. The interpolation process can adopt any interpolation algorithm in the prior art, and the embodiment of the present application does not limit this. It can be understood that since the second sampling image and the second image are not used to represent the color information in the image, but are used to improve the brightness of the image. In the embodiment of the present application, the white pixel values ​​are evenly distributed in the second sampling image, and the white pixel values ​​are retained in the horizontal direction, vertical direction and 45° direction of the second sampling image, so that the second image with full resolution can be obtained more accurately, which is beneficial to improve the resolution and image quality of the final image.

[0133] S130: Perform mosaic rearrangement processing on the first image to obtain a third image.

[0134] As an example, Fig.16 After the 4# image in the figure is re-arranged and mosaicked, the third image shown in the 6# image is obtained. It can be seen that the third image is a data image in the Bayer format. In the embodiment of the present application, the process of re-arranging the mosaic can adopt any re-arrangement method in the prior art, and the embodiment of the present application does not specifically limit this.

[0135] Since the third image after rearranged mosaic processing is in the Bayer format which is currently more commonly used in the field of image processing, the third image can be suitable for more types of image signal processors (ISP), so that the image sensor in this application can be adapted to more ISPs and suitable for more application scenarios.

[0136] S140: Fusing the third image with the second image to obtain an optimized color image.

[0137] Optionally, the resolution of the third image is the same as the resolution of the second image.

[0138] As an example, Fig.16 The optimized color image 7# is obtained by fusing the image 5# and the image 6#. The color image after image fusion can better maintain the brightness information while ensuring the color information of the image, thereby effectively improving the image quality under low light conditions. At the same time, in the frequency domain, the brightness and chromaticity information of the color image are not easy to overlap, so it is not easy to produce moire fringes.

[0139] It can be understood that the above-mentioned image processing method 10 can be implemented by a processor or a processing circuit. In other words, optionally, the above-mentioned image sensor can also include a processing unit, and the processing unit is used to implement the above-mentioned image processing method 10.

[0140] In addition to the image sensor 200 provided in the above application embodiments, the present application also provides an electronic device, which may include the image sensor 200 in any of the above embodiments.

[0141] The electronic device can be any electronic device with an image acquisition function, for example, it can be a mobile terminal such as a mobile phone or a computer, or a shooting device such as a camera or a camcorder, or an automated teller machine (ATM), etc. This application does not limit the application scenario of the electronic device.

[0142] Optionally, the processing unit for executing the image processing method 10 may not be located in the image sensor 200, but in a processing unit in the electronic device where the image sensor 200 is located. For example, if the electronic device is a mobile phone, the processing unit may be an image signal processing unit in the processor in the mobile phone, or the processing unit may also be a separate image signal processing chip in the mobile phone. The embodiments of the present application do not limit the specific hardware form of the processing unit.

[0143] It should be understood that the processor or processing unit of the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0144] It is understood that the image sensor of the embodiment of the present application may also include a memory, which may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. It should be noted that the memory of the system and method described herein is intended to include but is not limited to these and any other suitable types of memory.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image sensor, characterized in that: include: A filter unit array, comprising a plurality of filter unit groups, each of the plurality of filter unit groups comprising 4×4 filter units; wherein, in the 4×4 filter units, each row, each column and each diagonal line comprises 2 white filter units and 2 color filter units; A pixel unit array comprises a plurality of pixel units, wherein the pixel unit array is located below the filter unit array, and the plurality of pixel units in the pixel unit array correspond one-to-one to the plurality of filter units in the filter unit array, and each filter unit in the filter unit array is correspondingly arranged directly above each pixel unit in the pixel unit array.

2. The image sensor according to claim 1, characterized in that The filter unit group includes first color filter units, second color filter units and third color filter units of different colors, and the number of the first color filter units is equal to the sum of the number of the second color filter units and the third color filter units.

3. The image sensor according to claim 2, characterized in that In the filter unit group, the number of the second color filter units is equal to the number of the third color filter units.

4. The image sensor according to claim 3, characterized in that: In the filter unit group, the two color filter units in each row have different colors, and the two color filter units in each column have different colors.

5. The image sensor according to claim 4, characterized in that: In each row of the filter unit group, two white filter units and two color filter units are arranged alternately; In each column of the filter unit group, two white filter units are arranged continuously, and two color filter units are arranged continuously.

6. The image sensor according to claim 5, characterized in that The color filter units of the same color in the filter unit group are arranged in a centrally symmetrical manner.

7. The image sensor according to claim 5, characterized in that: Two target filter unit sets are arranged in the middle two rows of the filter unit group, and the target filter unit sets include two color filter units of the same color arranged at the same vertex angle.

8. The image sensor according to claim 7, characterized in that: In the filter unit group, the white filter units are respectively located in the first row and the second column, the first row and the fourth column, the second row and the second column, the second row and the fourth column, the third row and the first column, the third row and the third column, the fourth row and the first column, and the fourth row and the third column; The second color filter units are respectively located in the second row and the first column and the third row and the second column; The first color filter units are respectively located in the first row and the first column, the second row and the third column, the third row and the fourth column, and the fourth row and the second column; The third color filter units are located in the first row and the third column and in the fourth row and the fourth column respectively.

9. The image sensor according to claim 4, characterized in that: In each row of the filter unit group, two white filter units and two color filter units are arranged alternately; In the two columns separated by the filter unit groups, two white filter units are arranged continuously and two color filter units are arranged separately; in the other two columns separated by the filter unit groups, two white filter units are arranged separately and two color filter units are arranged continuously.

10. The image sensor according to claim 9, characterized in that: Two target filter unit sets are arranged in the middle two columns of the filter unit group, and the target filter unit sets include two color filter units of the same color arranged at the same vertex angle.

11. The image sensor according to claim 9, characterized in that: The filter unit group is provided with four target filter unit sets, and the target filter unit sets include two color filter units of the same color arranged at the same vertex angle.

12. The image sensor according to claim 11, characterized in that: In the filter unit group, the white filter units are respectively located in the first row and the second column, the first row and the fourth column, the second row and the first column, the second row and the third column, the third row and the first column, the third row and the third column, the fourth row and the second column, and the fourth row and the fourth column; The second color filter units are respectively located in the third column of the first row and the fourth column of the second row; The first color filter units are respectively located in the first row and the first column, the second row and the second column, the third row and the fourth column, and the fourth row and the third column; The third color filter units are located in the third row and the first column and the fourth row and the first column respectively.

13. The image sensor according to any one of claims 2 to 12, characterized in that: The first color filter unit, the second color filter unit and the third color filter unit are used to pass three colors of optical signals respectively, and the wavelength bands of the three colors of optical signals cover the visible light band.

14. The image sensor according to claim 13, characterized in that: The colors of the first color filter unit, the second color filter unit and the third color filter unit are respectively three colors of red, green, blue, cyan, magenta and yellow.

15. The image sensor according to claim 14, characterized in that: The first color filter unit is a green filter unit, and the second color filter unit and the third color filter unit are a red filter unit and a blue filter unit respectively.

16. The image sensor according to any one of claims 1 to 12, characterized in that: The image sensor further comprises: A microlens array includes a plurality of microlenses, and the microlens array is located above the filter unit array and is used to converge the light signal returned by the photographed object to the filter unit array, wherein the plurality of microlenses in the microlens array correspond one-to-one to the plurality of filter units in the filter unit array.

17. The image sensor according to any one of claims 1 to 12, characterized in that: The pixel values ​​of the color pixel units in the pixel unit array are used to generate first image data of the photographed object, and the pixel values ​​of the white pixel units in the pixel unit array are used to generate second image data of the photographed object, and the first image data and the second image data are used to synthesize a target image of the photographed object; The white pixel unit is a pixel unit corresponding to the white filter unit, and the color pixel unit is a pixel unit corresponding to the color filter unit.

18. The image sensor according to claim 17, characterized in that: The pixel values ​​of the color pixel units in the pixel unit array are used to generate an intermediate image through an interpolation process, and the intermediate image is used to generate the first image data in the Bayer format through a re-mosaic process.

19. The image sensor according to claim 18, characterized in that Among the 2×2 pixel values ​​of the intermediate image, 2 pixel values ​​are original pixel values ​​of the color pixel units, and the other 2 pixel values ​​are pixel values ​​obtained through interpolation processing.

20. The image sensor according to claim 17, wherein: The first image data and the second image data have the same resolution.

21. The image sensor according to any one of claims 1 to 12, characterized in that: The image sensor is a complementary metal oxide semiconductor (CMOS) image sensor, or a charge coupled device (CCD) image sensor.

22. An electronic device, characterized in that: include: An image sensor as claimed in any one of claims 1 to 21.

Images