A filter array and an image sensor

Abstract

The embodiments of the present application provide a filter array and an image sensor, wherein the filter array includes a plurality of tiled minimum repeating units, each of the minimum repeating units includes nine sub-regions of three rows by three columns, each of the nine sub-regions includes cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions are respectively provided with a green filter unit, a red filter unit, a blue filter unit, and a cyan filter unit, wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are centrally symmetrically distributed on the minimum repeating unit. The color arrangement scheme of the filter array provided in some embodiments of the present application can increase the amount of light entering without almost losing the color restoration ability, and can effectively reduce the brightness noise and color noise of the color image obtained after analysis.

Description

Technical Field

[0001] The present application relates to the field of image acquisition, and specifically, embodiments of the present application relate to a filter array and an image sensor. Background Art

[0002] In order to enable the image sensor to capture color patterns, the image sensor includes a photosensitive array (or pixel array) and a filter array arranged above the photosensitive array, that is, a color array (CFA, Color Filter Array) that can filter a characteristic wavelength range is placed on the photosensitive array (including multiple photodiodes). The most classic color array is the 2x2 Bayer array, which filters the incident light by placing two green filter units, one red filter unit and one blue filter unit in a periodic 2x2 area, and then the photosensitive unit positioned under the corresponding filter unit collects the light intensity information of the corresponding color light signal to generate a pixel signal, and finally obtains a full-size and full-color red, green and blue image based on the pixel signal and the algorithm of the image processing module (for example, the demosaic algorithm).

[0003] At present, with the increase in resolution of image sensors required in various terminals (for example, mobile phones) or systems, the reduction in pixels, and the popularization of applications such as night photography, how to better design the color array arrangement of filter units to increase light transmittance without compromising color reproduction ability has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a filter array and an image sensor. The color arrangement scheme of the filter array provided in the embodiments of the present application can effectively reduce the brightness noise and color noise after analysis compared with other schemes.

[0005] In a first aspect, some embodiments of the present application provide a filter array, comprising a plurality of tiled minimum repeating units, each of the minimum repeating units comprising nine sub-regions of three rows by three columns, each of the nine sub-regions comprising cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions are respectively provided with a green filter unit, a red filter unit, a blue filter unit and a cyan filter unit, wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are distributed centrally symmetrically on the minimum repeating unit.

[0006] The color arrangement scheme of the filter array provided in some embodiments of the present application can increase the amount of light entering without almost losing the color restoration ability, and can effectively reduce the brightness noise and color noise of the color image obtained after analysis.

[0007] In some embodiments, at least some cells in the remaining five sub-areas of the nine sub-areas are used to set up full-color filter units.

[0008] Some embodiments of the present application may arrange full-color filter units in cells where no colored filter elements are arranged. Compared with filter units that are not arranged with any colors, some embodiments of the present application may improve color fidelity by arranging full-color filter units. Compared with filter units arranged with other colors (i.e., non-full-color filter units), full-color filter units may increase the amount of light entering and thereby reduce the brightness noise of the resolved image.

[0009] In some embodiments, the four non-adjacent sub-regions are located on a short main diagonal line of the minimal repeating unit.

[0010] Some embodiments of the present application can improve color fidelity by disposing colored filter units in four non-adjacent N-row by N-column regions among nine regions.

[0011] In some embodiments, the four non-adjacent sub-regions are located on a major diagonal and a minor diagonal of the minimal repeating unit.

[0012] Some embodiments of the present application can further improve color fidelity by providing N rows by N columns of color filter units in five N rows by N columns of the nine regions.

[0013] In some embodiments, the value of N includes 1, 2 or 3.

[0014] In a second aspect, some embodiments of the present application provide an image sensor, comprising: a pixel array, the pixel array comprising a plurality of pixel units; and a filter array, the filter array comprising a plurality of filter units, the filter array being positioned above the pixel array and the filter units being optically coupled to the pixel units, the minimum repeating unit of the pattern of the filter array comprising nine sub-regions of three rows by three columns, each of the nine sub-regions comprising cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions are respectively provided with green filter units, red filter units, blue filter units and cyan filter units of N rows by N columns, wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are centrally symmetrically distributed on the minimum repeating unit.

[0015] In some embodiments, at least some cells in the remaining five sub-areas of the nine sub-areas are used to set up full-color filter units.

[0016] In some embodiments, the four non-adjacent sub-regions are located on a short main diagonal line of the minimal repeating unit.

[0017] In some embodiments, the four non-adjacent sub-regions are located on a long major diagonal and a long minor diagonal of the minimal repeating unit.

[0018] In some embodiments, the value of N includes 1, 2 or 3.

[0019] In some embodiments, the image sensor further includes a reading circuit, wherein the reading circuit is configured to combine and read the photosensitive currents of the photosensitive elements corresponding to the filter units in the same sub-region.

[0020] In some embodiments, the image sensor further includes a conversion device configured to convert electrical signals sensed by a plurality of photosensitive elements corresponding to the minimum repeating unit into output signals in a Bayer format for the image processing circuit to analyze the image.

[0021] In some embodiments, the conversion device is configured to: determine the target weight of the neural network model according to the resolution type, wherein the resolution includes a half-size mode and a full-size mode; and convert the input electrical signal to be converted into an output signal in the Bayer format according to the neural network model.

[0022] In a third aspect, some embodiments of the present application provide a digital camera, comprising: a lens; and the filter array described in the first aspect or the image sensor described in the second aspect.

[0023] In a fourth aspect, some embodiments of the present application provide a mobile phone terminal, comprising a memory, a processor, and the filter array described in the first aspect or the image sensor described in the second aspect.

[0024] In a fifth aspect, some embodiments of the present application provide a video surveillance system, comprising a memory, a display, and the filter array described in the first aspect or the image sensor described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of a portion of the structure of an image sensor provided for related technology;

[0027] Figure 2 A schematic diagram of the arrangement of the minimum repeating unit provided for the related art;

[0028] Figure 3 Schematic diagram of nine sub-regions and main diagonal lines and short main diagonal lines in the nine sub-regions included in the minimum repeating unit in the embodiment of the present application;

[0029] Figure 4 A schematic diagram of nine sub-regions and secondary diagonals and short secondary diagonals of the nine sub-regions included in a minimum repeating unit provided in some embodiments of the present application;

[0030] Figure 5-Figure 17 A schematic diagram of the arrangement of the filter units provided in the embodiment of the present application in the minimum repeating unit;

[0031] Fig.18 A schematic diagram of another part of the structure of the image sensor provided in an embodiment of the present application;

[0032] Fig.19 A block diagram of the camera structure provided in the embodiment of the present application;

[0033] Fig. 20 A block diagram of the mobile phone terminal provided in the embodiment of the present application;

[0034] Fig.21 A block diagram of the video surveillance system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0036] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0037] The color arrangement of the red filter unit, the green filter unit, the blue filter unit, and the cyan filter unit proposed in some embodiments of the present application can increase the amount of light entering without losing the color restoration ability. Therefore, the color arrangement scheme of the filter unit in some embodiments of the present application can effectively reduce the brightness noise and color noise of the analyzed image compared with other schemes.

[0038] Please see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a partial structure of an image sensor 10 of the related art, and is also a schematic diagram of a portion of the structure of the image sensor 10 on which some embodiments of the present application are based. Figure 1As shown, the image sensor 10 includes a filter array 11 and a pixel array 12. The filter array 11 includes a plurality of filter units 110. Some embodiments of the present application determine the arrangement rules of these filter units (for details, please refer to Figure 5-Figure 17 and the related description below). Figure 1 The pixel array 12 includes a plurality of pixel units 120, each pixel unit 120 corresponds to a filter unit 110 of the filter array 11. The pixel unit 120 is used to receive light passing through the corresponding filter unit 110 to generate an electrical signal. For example, the pixel unit 120 senses the light intensity information of the light signal of the corresponding color after being filtered by the filter unit of the corresponding color thereon through a set photosensitive element (such as a photodiode). In addition, Figure 1 Also shown is a microlens array 13 located above the filter array 11, wherein the microlens array 13 includes a plurality of microlenses 130, and the microlenses 130 are used to receive incident light. Figure 1 As shown, each microlens 130 corresponds to a filter unit 110 and a pixel unit 120 .

[0039] It should be noted that Figure 1 The correspondence between the filter unit 110 in the filter array 11 of the image sensor 10 and the pixel unit 120 in the pixel array 12 is used to illustrate the embodiment of the present application, and does not constitute a limitation on the application scenario of the embodiment of the present application. For example, the lens of the embodiment of the present application can also receive backlight.

[0040] Understandably, Figure 1 Each filter unit 110 (for example, a filter) can filter the incident light after being processed by the microlens 130, and only retain light of a certain frequency (corresponding to a certain color). In order to achieve the acquisition of high-quality color images, the image sensor 10 of some embodiments of the present application provides the following Figure 5-Figure 17 The filter array 11 is arranged in a pattern (different from the existing Bayer pattern). The multiple pixel units in the image sensor 10 can receive the light passing through the corresponding filter unit 110, thereby generating pixel signals with different color channels. Among them, each pixel unit 120 can only output a pixel signal of the value of one color channel, and the pixel signals of the values ​​of the other color channels need to be obtained by interpolation (that is, there is an image processing circuit).

[0041] Combine the following Figure 2-Figure 17 The arrangement rule of the filter units 110 of the filter array 11 in the minimum repeating unit is further described.

[0042] like Figure 2As shown, some embodiments of the present application provide a filter array 11, wherein the filter array 11 includes a plurality of tiled minimal repeating units 170, each of which includes nine sub-regions of three rows and three columns (for example, Figure 3 or Figure 4 ), each of the nine sub-regions including N rows by N columns of cells (e.g., Figure 5 Each sub-region of consists of 1*1 cells. Fig.12 Each sub-region includes 2*2 cells), and four non-adjacent sub-regions of the nine sub-regions are respectively arranged with N rows and N columns (for example, Figure 5 1*1 or Fig.12 The invention further comprises a green filter unit having N rows by N columns, a red filter unit having N rows by N columns, a blue filter unit having N rows by N columns, and a cyan filter unit having N rows by N columns, wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are centrally symmetrically distributed on the minimum repeating unit (for example, the four non-adjacent sub-regions are about Figure 3 or Figure 4 The fifth subregion located at the center of the minimum repeating unit is symmetrical).

[0043] In some embodiments, four non-adjacent sub-regions of the nine sub-regions are respectively arranged in N rows by N columns (for example, Figure 5 1*1 or Fig.12 The green filter unit, the red filter unit, the blue filter unit and the cyan filter unit are provided in the nine sub-areas, and at least part of the cells in the remaining five sub-areas of the nine sub-areas are used to set the full-color filter unit.

[0044] To illustrate exemplary positions of four non-adjacent sub-regions on the minimum repeating unit 170, please refer to Figure 3 and Figure 4 . Figure 3 A main diagonal line 180 and two short main diagonal lines (181, 182) located on both sides of the main diagonal line 180 are shown. Figure 4 A minor diagonal line 190 and two short minor diagonal lines (191, 192) on either side of the minor diagonal line 190 are shown. Figure 3 It can be seen that the main diagonal line 180 is a diagonal line that passes through the fifth sub-region at the center of the minimum repeating unit and connects the first sub-region at the upper left position and the ninth sub-region at the lower right position of the minimum repeating unit. The first short main diagonal line 182 is a diagonal line parallel to the main diagonal line 180 and connects the fourth sub-region and the eighth sub-region. The second short main diagonal line 181 is a diagonal line parallel to the main diagonal line 180 and connects the second sub-region and the sixth sub-region. Figure 4It can be seen that the secondary diagonal 190 is a diagonal line passing through the fifth sub-region of the center of the minimum repeating unit and connecting the third sub-region at the upper right position and the seventh sub-region at the lower left position of the minimum repeating unit. The first short secondary diagonal 192 is a diagonal line parallel to the secondary diagonal 190 and connecting the sixth sub-region and the eighth sub-region. The second short secondary diagonal 191 is a diagonal line parallel to the secondary diagonal 190 and connecting the second sub-region and the fourth sub-region.

[0045] In some embodiments of the present application (for example, Figure 6-Figure 16 ), the four non-adjacent sub-regions are located on the short main diagonal of the minimum repeating unit. For example, in some embodiments, the four sub-regions (i.e., the sub-regions for arranging the colored filter units) are respectively located Figure 3 The fourth sub-area, the eighth sub-area, the second sub-area and the sixth sub-area.

[0046] In some embodiments of the present application (for example, Figure 5 as well as Fig.17 ), the four non-adjacent sub-regions are located on the main diagonal and the secondary diagonal of the minimum repeating unit and the four non-adjacent sub-regions are symmetrical about the central sub-region of the minimum repeating unit, that is, the four non-adjacent sub-regions are distributed in the upper left corner sub-region, the lower left corner sub-region, the upper right corner sub-region and the lower right corner sub-region of the nine sub-regions of three rows and three columns (reference Figure 3 or Figure 4 , the four non-adjacent sub-regions correspond to the first sub-region, the third sub-region, the seventh sub-region and the ninth sub-region respectively. For example, in some embodiments, two of the four sub-regions (i.e., the sub-regions for arranging the colored filter units) are respectively located at Figure 3 The first sub-area and the ninth sub-area on the main diagonal 180, and the remaining two sub-areas are located at Figure 4 The third sub-area and the seventh sub-area on the minor diagonal line of .

[0047] In some embodiments, the value of N includes 1, 2 or 3. For example, in some embodiments of the present application, N is 1 (see FIG. Figure 5 as well as Figure 6 ), that is, one sub-region corresponds to one cell. In these embodiments, the corresponding minimum repeating unit includes three rows by three columns of cells, and four non-adjacent cells in the nine cells of three rows by three columns and symmetrical about the center are respectively provided with a red filter unit, a color filter unit, a blue filter unit and a cyan filter unit; in some other embodiments of the present application, N is 2 (please refer to Figure 12-Figure 17), that is, each of the nine sub-regions includes two rows and two columns of cells, and the four non-adjacent and center-symmetrical sub-region grids in the nine sub-regions corresponding to the minimum repeating unit are respectively provided with four red filter units, four color filter units, four blue filter units and four cyan filter units; in some other embodiments of the present application, N takes a value of 3 (not shown in the figure), that is, each of the nine sub-regions includes three rows and three columns of cells, and the four non-adjacent and center-symmetrical sub-region grids in the nine sub-regions corresponding to the minimum repeating unit are respectively provided with nine red filter units, nine color filter units, nine blue filter units and nine cyan filter units.

[0048] Combine the following Figure 5-Figure 17 The arrangement rules of the filter units on the filter array 11 of some embodiments of the present application are exemplarily described.

[0049] It should be noted that Figure 5-Figure 17 In the figure, R is used to indicate a red filter unit that transmits red light (e.g., with a wavelength range of approximately 605nm-700nm) and filters out light of other colors, G is used to indicate a green filter unit that transmits green light (e.g., with a wavelength range of approximately 500nm-560nm) and filters out light of other colors, B is used to indicate a blue filter unit that transmits blue light (e.g., with a wavelength range of approximately 435nm-480nm) and filters out light of other colors, C is used to indicate a cyan filter unit that transmits cyan light (e.g., with a wavelength range of approximately 480nm-490nm) and filters out light of other colors, and W is used to indicate that a full-color filter unit is set at the corresponding position (i.e., in the cell) or no filter unit is set. In some embodiments, cyan indicates a filter that can simultaneously pass blue and green light with higher efficiency (e.g., higher number of photons or higher power), that is, controlling the amount of blue light and green light (e.g., power or number of photons) passing through the cyan filter unit to be greater than the amount of red light passing can achieve the purpose of filtering other colors of light through cyan light.

[0050] like Figure 5 As shown in FIG. 1 , the figure only shows a minimum repeating unit of some embodiments of the present application. In some embodiments of the present application, the minimum repeating unit of the filter array is nine cells of three rows and three columns (i.e., each sub-region of the minimum repeating unit corresponds to one cell), each of the nine sub-regions includes one cell, and four non-adjacent cells of the nine cells (i.e., four non-adjacent sub-regions) are respectively provided with a green filter unit, a red filter unit, a blue filter unit and a cyan filter unit, with reference to FIG. Figure 5 It can be seen that the four non-adjacent cells are distributed centrally and symmetrically on the minimum repeating unit and the four non-adjacent cells are located on the main diagonal and the secondary diagonal.

[0051] and Figure 5 Only one minimal repeating unit is shown to be different. Figure 6-Figure 17 The four minimum repeating units 170 ( Figure 6-Figure 17 The dotted line box is a minimum repeating unit 170), but this does not constitute a limitation on the embodiment of the present application. In other words, the number of the minimum repeating units 170 can be adjusted according to actual conditions.

[0052] like Figure 6 As shown, the minimum repeating unit of the filter array 11 provided in some embodiments of the present application is a unit cell of 3 rows by 3 columns. Figure 6 Each of the minimum repeating units 170 includes nine sub-regions of one row by one column (i.e., each sub-region corresponds to a cell), and four non-adjacent sub-regions of the nine sub-regions (i.e., four non-adjacent cells) are respectively provided with a green filter unit G, a red filter unit R, a blue filter unit B and a cyan filter unit C, wherein a red filter unit R and a blue filter unit B are located on the first short main diagonal line, a cyan filter unit C and a green filter unit G are located on the second short main diagonal line, and the cells used to set these colored filter units (i.e., the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit) are distributed in a centrally symmetrical manner on the minimum repeating unit. Figure 6 and Figure 5 The difference is that Figure 6 The four non-adjacent units that are symmetrical about the center of the minimal repeating unit are located on the short main diagonal.

[0053] Figure 7 The arrangement rule of the filter units 110 in the filter array shown is to Figure 6 The smallest repeating unit in the arrangement pattern of the filter array is obtained by rotating 90 degrees clockwise.

[0054] Figure 8 The arrangement rule of the filter units 110 in the filter array shown is to Figure 6 The smallest repeating unit in the arrangement pattern of the filter array is obtained by horizontally flipping.

[0055] Fig. 9 The arrangement rule of the filter units 110 in the filter array shown is to Figure 6 The minimum repeating unit in the arrangement pattern of the filter array is obtained by bilateral symmetry.

[0056] Fig.10 The arrangement rule of the filter units 110 in the filter array shown is to Figure 6 The minimum repeating unit in the arrangement pattern of the filter array is obtained by being symmetrical from top to bottom.

[0057] Fig.11 The arrangement rule of the filter units 110 in the filter array shown is to Figure 6 The arrangement pattern of the filter array is obtained by the interaction of the green filter unit and the cyan filter unit in the two sub-areas in the minimum repeating unit.

[0058] like Fig.12 As shown, the minimum repeating unit of the filter array 11 provided in some embodiments of the present application includes nine sub-regions of two rows by two columns (i.e., each sub-region corresponds to four cells), and four non-adjacent sub-regions of the nine sub-regions are respectively provided with two rows by two columns of green filter units G, two rows by two columns of four red filter units R, two rows by two columns of four blue filter units B, and two rows by two columns of four cyan filter units C, wherein the four red filter units R and the four blue filter units B are located on the first short main diagonal line, the four cyan filter units C and the four green filter units G are located on the second short main diagonal line, and the sub-regions used to set these colored filter units (i.e., the red filter units, the green filter units, the blue filter units, and the cyan filter units) are distributed in a centrally symmetrical manner on the minimum repeating unit.

[0059] Fig.13 The arrangement rule of the filter units 110 in the filter array shown is to Fig.12 The arrangement of the filter units is obtained by rotating the minimum repeating unit in the arrangement pattern of the filter array 90 degrees to the right.

[0060] Fig.14 The arrangement rule of the filter units 110 in the filter array shown is to Fig.12 The arrangement of the filter units is obtained by vertically flipping the minimum repeating unit in the arrangement pattern of the filter array.

[0061] Fig.15 The arrangement rule of the filter units 110 in the filter array shown is to Fig.12 The arrangement of the filter units is obtained by performing bilateral symmetry processing on the minimum repeating unit in the arrangement pattern of the filter array.

[0062] Fig.16 The arrangement rule of the filter units 110 in the filter array shown is to Fig.12 The arrangement of the filter units is obtained by performing vertical symmetry processing on the minimum repeating unit in the arrangement pattern of the filter array.

[0063] and Fig.12 The difference is, Fig.17 The colored filter units are distributed on the main diagonal and the secondary diagonal, rather than on the short main diagonal. Fig.17As shown, each of the minimum repeating units 170 includes nine sub-regions of two rows by two columns (i.e., each sub-region corresponds to four cells), and four non-adjacent sub-regions of the nine sub-regions are respectively provided with four green filter units G, four red filter units R, four blue filter units B, and four cyan filter units C of two rows by two columns, wherein the four red filter units R and the four cyan filter units C are located on the main diagonal, the four green filter units G and the four blue filter units B are located on the secondary diagonal, and the cells for setting these colored filter units (i.e., the red filter units, the green filter units, the blue filter units, and the cyan filter units) are distributed in a centrally symmetrical manner on the minimum repeating unit.

[0064] It should be noted that Figure 5-Figure 17 The smallest repeating unit in the color arrangement of Figure 6-Figure 17 A 90-degree rotation, flipping, left-right symmetry, top-bottom symmetry, or other transformations, or exchanging the positions of the colored filter units of any two sub-regions, or increasing the number of cells in each of the nine sub-regions of the minimum repeating unit into a square should be considered to be essentially the same color arrangement (for example, Figure 6 Each of the nine sub-areas includes a 1*1 cell. Fig.12 Each of the nine sub-regions includes 2*2 cells, and each of the nine sub-regions in some other embodiments of the present application may also include 3*3, 4*4 or 5*5 cells). The cells mentioned in the embodiments of the present application are mainly used to locate each pixel unit. Since the pixel unit corresponds to the filter unit one-to-one, some embodiments of the present application also involve cells when describing the arrangement rules or arrangement diagrams of the filter units. It can be understood that the sub-regions of some embodiments of the present application are regions including at least one cell. The cells or sub-regions mentioned in the embodiments of the present application are only used for the description of the technical solution, that is, the arrangement rules of the filter units, and do not constitute a limitation on the technical solution of the embodiments of the present application.

[0065] Combine the following Fig.18 The structure and layout of the image sensor 10 are exemplarily described for the sake of simplicity and clarity of the drawings. Fig.18 Only some units or circuits included in the image sensor 10 are provided. Fig.18 For other units included in the image sensor 10 (eg, the arrangement rules of the filter units in the filter array), reference may be made to the related drawings of other embodiments of the present application (eg, Figure 5-Figure 17 ).

[0066] like Fig.18As shown, some embodiments of the present application provide an image sensor 10, the image sensor 10 comprising: a pixel array 12, the pixel array 12 comprising a plurality of pixel units 120; and a filter array (since the filter array is tiled above the pixel array 12, Fig.17 The optical filter array 11 is not shown. The optical filter array 11 includes a plurality of optical filter units (such as Figure 1 As shown), the filter array is positioned above the pixel array and the filter unit is optically coupled to the pixel unit.

[0067] The minimum repeating unit of the pattern of the filter array 11 provided in some embodiments of the present application includes nine sub-regions of three rows by three columns, each of the nine sub-regions includes cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions are respectively provided with a green filter unit, a red filter unit, a blue filter unit and a cyan filter unit, wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are centrally symmetrically distributed on the minimum repeating unit (for details, please refer to the description of the filter array in the previous text). In some embodiments, at least part of the cells in the remaining five sub-regions of the nine sub-regions are used to set a full-color filter unit. In some embodiments, the four non-adjacent sub-regions are located on the short main diagonal of the minimum repeating unit. In some embodiments, the four non-adjacent sub-regions are located on the main diagonal and the secondary diagonal of the minimum repeating unit.

[0068] It should be noted that the arrangement rules of the filter units in the filter array 11 included in the image sensor 10 may be specifically referred to the above description of the filter array and may also be referred to Figure 5-Figure 17 In order to avoid repetition, I will not go into details here.

[0069] The following focuses on other circuits or devices included in the image sensor.

[0070] like Fig.18 As shown, in some embodiments of the present application, the image sensor 10 further includes a reading circuit 330, which is configured to combine and read the photosensitive elements ( Fig.18 For example, the read circuit 330 combines the read Fig.12 The sixth sub-region in the minimum repeating unit in the dotted box (the position of the sixth sub-region in the minimum repeating unit can be referred to Figure 3 or Figure 4 ) of the four green filter units (for example, the photodiodes included in the pixel units) or the fifth sub-area located at the center of the minimum repeating unit (the location of the fifth sub-area is specifically referred to in Figure 3 or Figure 4 ) of the four cells (for example, these cells may be provided with full-color filter units or without filter units) corresponding to the induced current of the pixel unit; the reading circuit 330 combines the reading Fig.12 The photosensitive current of the four pixel units (for example, the photodiodes included in the pixel units) located by the four red filter units in the fourth sub-area of ​​the minimum repeating unit in the dotted box, or the induced current of the pixel units corresponding to the 2*2 cells (for example, these cells can be provided with full-color filter units or no filter units) in the first sub-area located at the upper left position of the minimum repeating unit; the reading circuit 330 combines the reading Fig.12 The photosensitive current of four pixel units (eg, photodiodes included in the pixel units) located by four blue filter units in the eighth sub-region of the minimum repeating unit within the dashed box; the reading circuit 330 combines the reading Fig.12 The photosensitive current of four pixel units (eg, photodiodes included in the pixel units) located in the four cyan filter units of the second sub-region in the minimum repeating unit within the dashed box; the reading circuit 330 combines the reading Fig.12 The photosensitive current of four pixel units (e.g., photodiodes included in the pixel units) located by the four full-color filter units set in the ninth sub-area at the lower right position in the dotted frame. It should be noted that in order to read out the induced electrical signals of each pixel unit, it is necessary to address each pixel unit. As an example, Fig.18 The addressing process of a certain pixel unit is: a certain row address line 311 is selected under the control of the row driving circuit 310 and a certain column address line 321 is selected under the control of the column driving circuit 320, so as to read out the current signal induced by a pixel unit 120 at the intersection of the row address line 311 and the column address line 321 through the reading line 312 connected to the reading circuit 330. Compared with the solutions in the prior art, some embodiments of the present application need to combine and read the induced current of one or more pixel units located in a filter unit of a sub-region in a minimum repeating unit.

[0071] like Fig.18 As shown, in some embodiments, the image sensor further includes a conversion device 340, and the conversion device 340 is configured to convert the electrical signals sensed by the multiple photosensitive elements corresponding to the minimum repeating unit into an output signal in the Bayer format, so that the image processing circuit can analyze the image. As an example, the conversion device 340 is configured to: determine the target weight of the neural network model according to the resolution type, wherein the resolution type includes a half-size mode and a full-size mode; and convert the input electrical signal to be converted into the output signal in the Bayer format according to the neural network model.

[0072] That is to say, some embodiments of the present application still require the conversion device 340 to perform signal processing before sending the induced electrical signal of the pixel unit read by the reading circuit 330 to the image processing circuit. This is because, in some embodiments, the image processing circuit cannot directly analyze the electrical signals sensed by the pixel units corresponding to the arrangement of the filter units provided in some embodiments of the present application, so the read data needs to be converted in the following two ways: for the half-size mode, that is, the photosensitive currents corresponding to the pixel units corresponding to the filter units in the same sub-area are read in combination, for example, the minimum repeating unit is a matrix of six rows by six columns and includes a total of 36 filter units, and the sensed currents of nine 2*2 sub-areas are read in combination, and a total of nine read sensed currents corresponding to each sub-area are obtained. Then, according to some embodiments of the present application, the conversion device 340 can convert the 9 pixel read values ​​into 9 rearranged pixel output values ​​in the standard Bayer format; for the full-size mode, that is, the current values ​​sensed by the 36 pixel units corresponding to the minimum repeating unit are read separately, that is, each pixel unit reads a sensed current value, and then the 36 sensed current values ​​are converted into a 6x6 standard Bayer format through the conversion device of the embodiment of the present application. For example, the conversion device 340 may be a separate bridge chip that obtains the readout current value of the pixel unit through the signal line; in some embodiments, the conversion device 340 may be stacked on a chip with the image sensor.

[0073] In some embodiments, the conversion device 340 can be implemented by a programmable neural network processor (NPU). As an example, the conversion device 340 for the full-size mode can be implemented by the NPU in the following manner: S1. Read the sensed signal values ​​(for example, sensed analog current values) of the photosensitive units of the pixel units 120 in sequence and enter the buffer area of ​​the NPU in sequence; S2. When there is enough data in the buffer area, start the NPU to use the pre-trained neural network weights to calculate the processed results; S3. The results calculated by the NPU are sent to the input and output units of the image sensor through the bus. It can be understood that for the required neural network weights, the following method can be used: prepare a large number of RGB images; arrange the colors according to the color arrangement method in the embodiment of the present application (i.e. Figure 5-Figure 17 ) arranges filter units and simulates the corresponding photosensitive unit (for example, the photodiode included in the pixel unit) readout value and the real value (ground truth) in Bayer format; adds simulated noise (for example, independent and identically distributed standard Gaussian noise) to the readout value of the photosensitive unit; takes the L1 norm of the predicted value of the neural network and the correct labeled value ground truth as the objective function, and uses the stochastic gradient descent SGD algorithm to adjust the weight of the neural network until convergence.

[0074] like Fig.19As shown, some embodiments of the present application further provide a digital camera 400, which includes an image sensor 10 and a lens 410. The specific structure of the image sensor 10 can be referred to the above description, and will not be described in detail here to avoid repetition.

[0075] like Fig. 20 As shown, the embodiment of the present application provides a mobile phone terminal 500, which includes a memory 510, a processor 520 and an image sensor 10. The specific structure of the image sensor 10 can refer to the above description, and no further details are given here to avoid repetition.

[0076] like Fig.21 As shown, the embodiment of the present application provides a video monitoring system 600, which includes a processor 610, a display 620, and an image sensor 10. The specific structure of the image sensor 10 can refer to the above description, and will not be described in detail here to avoid repetition.

[0077] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0078] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0079] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0080] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0081] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0082] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A filter array, characterized in that: The filter array includes a plurality of tiled minimum repeating units, each of the minimum repeating units includes nine sub-regions of three rows by three columns, each of the nine sub-regions includes cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions are respectively provided with a green filter unit, a red filter unit, a blue filter unit and a cyan filter unit, and at least part of the cells of the remaining five sub-regions of the nine sub-regions are used to set a full-color filter unit; wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions are centrally symmetrically distributed on the minimum repeating unit.

2. The filter array according to claim 1, wherein: At least part of the cells in the remaining five sub-areas of the nine sub-areas are used to set up full-color filter units.

3. The filter array according to any one of claims 1 to 2, characterized in that: The four non-adjacent sub-regions are located on the short main diagonal of the minimum repeating unit, wherein the short main diagonal includes a first short main diagonal and a second short main diagonal, the first short main diagonal is a diagonal parallel to the main diagonal and connecting the fourth sub-region and the eighth sub-region, and the second short main diagonal is a diagonal parallel to the main diagonal and connecting the second sub-region and the sixth sub-region; The main diagonal line is a diagonal line passing through the fifth sub-region of the center of the minimum repeating unit and connecting the first sub-region at the upper left position and the ninth sub-region at the lower right position of the minimum repeating unit, and the secondary diagonal line is a diagonal line passing through the fifth sub-region of the center of the minimum repeating unit and connecting the third sub-region at the upper right position and the seventh sub-region at the lower left position of the minimum repeating unit.

4. The filter array according to any one of claims 1 to 2, characterized in that: The four non-adjacent sub-regions are located on the main diagonal and the secondary diagonal of the minimum repeating unit, wherein the main diagonal is the diagonal line that passes through the fifth sub-region at the center of the minimum repeating unit and connects the first sub-region at the upper left position and the ninth sub-region at the lower right position of the minimum repeating unit, and the secondary diagonal line is the diagonal line that passes through the fifth sub-region at the center of the minimum repeating unit and connects the third sub-region at the upper right position and the seventh sub-region at the lower left position of the minimum repeating unit.

5. The filter array according to any one of claims 1 to 2, characterized in that: The value of N includes 1, 2 or 3.

6. An image sensor, characterized in that: The image sensor comprises: A pixel array, the pixel array comprising a plurality of pixel units; and A filter array, the filter array comprising a plurality of filter units, the filter array being positioned above the pixel array and the filter units being optically coupled to the pixel units, the minimum repeating unit of the pattern of the filter array comprising nine sub-regions of three rows by three columns, each of the nine sub-regions comprising cells of N rows by N columns, and four non-adjacent sub-regions of the nine sub-regions being respectively provided with a green filter unit, a red filter unit, a blue filter unit and a cyan filter unit, and at least part of the cells of the remaining five sub-regions of the nine sub-regions being used to provide a full-color filter unit; wherein N is a natural number greater than or equal to 1, and the four non-adjacent sub-regions being centrally symmetrically distributed on the minimum repeating unit.

7. The image sensor according to claim 6, wherein: At least part of the cells in the remaining five sub-areas of the nine sub-areas are used to set up full-color filter units.

8. The image sensor according to any one of claims 6 to 7, characterized in that: The four non-adjacent sub-regions are located on the short main diagonal of the minimum repeating unit, the short main diagonal is a diagonal parallel to the main diagonal and connecting the fourth sub-region and the eighth sub-region, and the second short main diagonal is a diagonal parallel to the main diagonal and connecting the second sub-region and the sixth sub-region; The main diagonal line is a diagonal line passing through the fifth sub-region of the center of the minimum repeating unit and connecting the first sub-region at the upper left position and the ninth sub-region at the lower right position of the minimum repeating unit, and the secondary diagonal line is a diagonal line passing through the fifth sub-region of the center of the minimum repeating unit and connecting the third sub-region at the upper right position and the seventh sub-region at the lower left position of the minimum repeating unit.

9. The image sensor according to any one of claims 6 to 7, characterized in that: The four non-adjacent sub-regions are located on the main diagonal and the secondary diagonal of the minimum repeating unit, wherein the main diagonal is the diagonal line that passes through the fifth sub-region at the center of the minimum repeating unit and connects the first sub-region at the upper left position and the ninth sub-region at the lower right position of the minimum repeating unit, and the secondary diagonal line is the diagonal line that passes through the fifth sub-region at the center of the minimum repeating unit and connects the third sub-region at the upper right position and the seventh sub-region at the lower left position of the minimum repeating unit.

10. The image sensor according to any one of claims 6 to 7, characterized in that: The value of N includes 1, 2 or 3.

11. The image sensor according to any one of claims 6 to 7, characterized in that: The image sensor further comprises a reading circuit, wherein the reading circuit is configured to combine and read the photosensitive currents of the photosensitive elements corresponding to the filter units in the same sub-area.

12. The image sensor according to any one of claims 6 to 7, characterized in that: The image sensor further includes a conversion device configured to convert electrical signals sensed by a plurality of photosensitive elements corresponding to the minimum repeating unit into output signals in a Bayer format for image processing circuits to analyze images.

13. The image sensor according to claim 12, wherein: The conversion device is configured to: Determining a target weight of a neural network model according to a resolution type, wherein the resolution includes a half-size mode and a full-size mode; The input electrical signal to be converted is converted into an output signal in the Bayer format according to the neural network model.

14. A digital camera, characterized in that: The camera comprises: Lenses; and The filter array according to any one of claims 1 to 5 or the image sensor according to any one of claims 6 to 13.

15. A mobile phone terminal, characterized in that: The mobile phone terminal comprises a memory, a processor, and the filter array according to any one of claims 1 to 5 or the image sensor according to any one of claims 6 to 13.

16. A video surveillance system, characterized in that: The video monitoring system comprises a memory, a display, and the filter array according to any one of claims 1 to 5 or the image sensor according to any one of claims 6 to 13.

Images