A filter array and an image sensor

Abstract

The embodiments of the present application provide a filter array and a graphic sensor, wherein the filter array includes a plurality of tiled minimum repeating units, each of the minimum repeating units includes a cell group arranged in six rows by six columns, and the cell group of six rows by six columns includes four cell groups of three rows by three columns; wherein some cells in a first cell group of the four cell groups are used to arrange green filter units; some cells in a second cell group of the four cell groups are used to arrange red filter units; some cells in a third cell group of the four cell groups are used to arrange blue filter units; and some cells in a fourth cell group of the four cell groups are used to arrange cyan filter units. The color arrangement scheme of the filter array provided in some embodiments of the present application can increase the amount of light entering without almost losing the color reproduction ability.

Description

Technical Field

[0001] The present application relates to the field of image acquisition, and specifically, embodiments of the present application relate to a filter array and an image sensor. Background Art

[0002] In order to enable the image sensor to capture color patterns, the image sensor includes a photosensitive array (or pixel array) and a filter array arranged above the photosensitive array, that is, a color array (CFA, Color Filter Array) that can filter a characteristic wavelength range is placed on the photosensitive array (including multiple photodiodes). The most classic color array is the 2x2 Bayer array, which filters the incident light by placing two green filter units, one red filter unit and one blue filter unit in a periodic 2x2 area, and then the photosensitive unit positioned under the corresponding filter unit collects the light intensity information of the corresponding color light signal to generate a pixel signal, and finally obtains a full-size and full-color red, green and blue image based on the pixel signal and the algorithm of the image processing module (for example, the demosaic algorithm).

[0003] At present, with the increase in resolution of image sensors required in various terminals (for example, mobile phones) or systems, the reduction in pixels, and the popularization of applications such as night photography, how to better design the color array arrangement of filter units to increase light transmittance without compromising color reproduction ability has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a filter array and an image sensor. The color arrangement scheme of the filter array provided in the embodiments of the present application can effectively reduce the brightness noise and color noise after analysis compared with other schemes.

[0005] In a first aspect, some embodiments of the present application provide a filter array, which includes a plurality of tiled minimum repeating units, each of which includes a cell group arranged in six rows by six columns, and the cell group of six rows by six columns includes four cell groups of three rows by three columns; wherein, some cells in a first cell group among the four cell groups are used to arrange green filter units; some cells in a second cell group among the four cell groups are used to arrange red filter units; some cells in a third cell group among the four cell groups are used to arrange blue filter units; and some cells in a fourth cell group among the four cell groups are used to arrange cyan filter units.

[0006] The color arrangement scheme of the filter array provided in some embodiments of the present application can increase the amount of light entering without almost losing the color restoration ability, and can effectively reduce the brightness noise and color noise of the color image obtained after analysis.

[0007] In some embodiments, at least some of the cells in at least some of the first cell group, the second cell group, the third cell group, and the fourth cell group that are not provided with colored filter units are used to set full-color filter units, wherein the colored filter units include red filter units, green filter units, blue filter units, and cyan filter units.

[0008] Some embodiments of the present application may arrange full-color filter units in cells where no colored filter elements are arranged. Compared with filter units that are not arranged with any colors, some embodiments of the present application may improve color fidelity by arranging full-color filter units. Compared with filter units arranged with other colors (i.e., non-full-color filter units), full-color filter units may increase the amount of light entering and thereby reduce the brightness noise of the resolved image.

[0009] In some embodiments, four non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; four non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; four non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; four non-adjacent cells out of the nine cells included in the fourth cell group are used to set the cyan filter unit; wherein the four non-adjacent cells are located on the cells other than the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0010] Some embodiments of the present application can improve color fidelity by arranging colored filter units in non-adjacent areas in four cells (excluding the most central cell) in nine cell groups.

[0011] In some embodiments, five non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; five non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; five non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; five non-adjacent cells out of the nine cells included in the fourth cell group are used to set the cyan filter unit; wherein the five non-adjacent cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0012] Some embodiments of the present application can further improve color fidelity by disposing non-connected colored filter units in five cells in the nine cell groups (cells located on the main diagonal and the secondary diagonal, including the central cell in the cell group).

[0013] In some embodiments, at least some of the multiple cells in the four three-row by three-column cell groups that are not arranged with colored filter units are used to set up full-color filter units; wherein the colored filter units include: the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit.

[0014] Some embodiments of the present application can improve color fidelity compared to not setting any color filter elements by setting full-color filter units in cells where no color filter units are set, and can increase the amount of light entering compared to setting filter elements of other colors (i.e., non-full-color filter elements), and can effectively reduce the brightness noise and color noise of the color image obtained after analysis.

[0015] In a second aspect, some embodiments of the present application provide an image sensor, comprising: a pixel array, the pixel array comprising a plurality of pixel units; and a filter array, the filter array comprising a plurality of filter units, the filter array being positioned above the pixel array and the filter units being optically coupled to the pixel units, the minimum repeating unit of the pattern of the filter array being a cell group of six rows by six columns, the cell group of six rows by six columns comprising four cell groups of three rows by three columns; wherein, some cells in a first cell group of the four cell groups are arranged with green filter units; some cells in a second cell group of the four cell groups are arranged with red filter units; some cells in a third cell group of the four cell groups are arranged with blue filter units; and some cells in a fourth cell group of the four cell groups are arranged with cyan filter units.

[0016] In some embodiments, at least some of the cells in at least some of the first cell group, the second cell group, the third cell group, and the fourth cell group that are not provided with colored filter units are used to set full-color filter units, wherein the colored filter units include red filter units, green filter units, blue filter units, and cyan filter units.

[0017] In some embodiments, four non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; four non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; four non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; four cells out of the nine cells included in the fourth non-adjacent cell group are used to set the cyan filter unit; wherein the four non-adjacent cells are located on the cells other than the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0018] In some embodiments, five non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; five non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; five non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; five non-adjacent cells out of the nine cells included in the fourth cell group are used to set the cyan filter unit; wherein the five non-adjacent cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0019] In some embodiments, at least some of the multiple cells in the four three-row by three-column cell groups that are not arranged with colored filter units are provided with full-color filter units; wherein the colored filter units include: the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit.

[0020] In some embodiments, the image sensor further includes a reading circuit configured to combine and read the photosensitive currents of the photosensitive elements corresponding to the filter units of the same color in the same cell group.

[0021] In some embodiments, the image sensor further includes a conversion device configured to convert electrical signals sensed by a plurality of photosensitive elements corresponding to the minimum repeating unit into output signals in a Bayer format for the image processing circuit to analyze the image.

[0022] In some embodiments, the conversion device is configured to: determine the target weight of the neural network model according to the resolution type, wherein the resolution includes a half-size mode and a full-size mode; and convert the input electrical signal to be converted into an output signal in the Bayer format according to the neural network model.

[0023] In a third aspect, some embodiments of the present application provide a digital camera, comprising: a lens; and the filter array described in the first aspect or the image sensor described in the second aspect.

[0024] In a fourth aspect, some embodiments of the present application provide a mobile phone terminal, comprising a memory, a processor, and the filter array provided in the first aspect or the image sensor provided in the second aspect.

[0025] In a fifth aspect, some embodiments of the present application provide a video surveillance system, comprising a memory, a display, and the filter array described in the first aspect or the image sensor described in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A schematic diagram of a portion of the structure of an image sensor provided for related technology;

[0028] Figure 2 A schematic diagram of the arrangement of the minimum repeating unit provided in the embodiment of the present application;

[0029] Figure 3-Figure 14 A schematic diagram of the arrangement of the filter units provided in the embodiment of the present application in the minimum repeating unit;

[0030] Fig.15 A schematic diagram of another part of the structure of the image sensor provided in an embodiment of the present application;

[0031] Fig.16 A block diagram of the camera structure provided in the embodiment of the present application;

[0032] Fig.17 A block diagram of the mobile phone terminal provided in the embodiment of the present application;

[0033] Fig.18 A block diagram of the video surveillance system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0035] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0036] The color arrangement of the red filter unit, the green filter unit, the blue filter unit, and the cyan filter unit proposed in some embodiments of the present application can increase the amount of light entering without losing the color restoration ability. Therefore, the color arrangement scheme of the filter unit in some embodiments of the present application can effectively reduce the brightness noise and color noise of the analyzed image compared with other schemes.

[0037] Please see Figure 1 , Figure 1 FIG. 1 is a schematic diagram of a partial structure of an image sensor 10 of the related art, and is also a schematic diagram of a portion of the structure of the image sensor 10 on which some embodiments of the present application are based. Figure 1 As shown, the image sensor 10 includes a filter array 11 and a pixel array 12. The filter array 11 includes a plurality of filter units 110. Some embodiments of the present application determine the arrangement rules of these filter units 110 (for details, please refer to Figure 3-Figure 14 and the related description below). Figure 1 The pixel array 12 includes a plurality of pixel units 120, each pixel unit 120 corresponds to a filter unit 110 of the filter array 11. The pixel unit 120 is used to receive light passing through the corresponding filter unit 110 to generate an electrical signal. For example, the pixel unit 120 senses the light intensity information of the light signal of the corresponding color after being filtered by the filter unit of the corresponding color thereon through a set photosensitive element (such as a photodiode). In addition, Figure 1 Also shown is a microlens array 13 located above the filter array 11, wherein the microlens array 13 includes a plurality of microlenses 130, and the microlenses 130 are used to receive incident light. Figure 1 As shown, each microlens 130 corresponds to a filter unit 110 and a pixel unit 120 .

[0038] It should be noted that Figure 1 The correspondence between the filter unit 110 in the filter array 11 of the image sensor 10 and the pixel unit 120 in the pixel array 12 is used to illustrate the embodiment of the present application, and does not constitute a limitation on the application scenario of the embodiment of the present application. For example, the lens of the embodiment of the present application can also receive backlight.

[0039] Understandably, Figure 1Each filter unit 110 (for example, a filter) can filter the incident light after being processed by the microlens 130, and only retain light of a certain frequency (corresponding to a certain color). In order to achieve the acquisition of high-quality color images, the image sensor 10 of some embodiments of the present application provides the following Figure 3-Figure 14 The filter array 11 is arranged in a pattern (different from the existing Bayer pattern). The multiple pixel units 120 in the image sensor 10 can receive the light passing through the corresponding filter unit 110, thereby generating pixel signals with different color channels. Among them, each pixel unit 120 can only output a pixel signal of the value of one color channel, and the pixel signals of the values ​​of the other color channels need to be obtained by interpolation (that is, there is an image processing circuit).

[0040] Combine the following Figure 2-Figure 14 The arrangement rules of the filter units 110 of the filter array 11 in the minimum repeating unit provided in some embodiments of the present application are further described.

[0041] like Figure 2 As shown, some embodiments of the present application provide a filter array 11, wherein the filter array 11 includes a plurality of tiled minimal repeating units 170, each minimal repeating unit 170 includes a unit cell group arranged in six rows and six columns (eg, Figure 3-Figure 14 The cell group within the dotted box in FIG. 1 ), the six-row by six-column cell group includes four three-row by three-column cell groups (for example, see Figure 3 Four cell groups obtained by dividing a six-row by six-column matrix by vertical dotted lines 178 and horizontal dotted lines 179, namely, an upper left cell group 171, a lower left cell group 173, an upper right cell group 172, and a lower right cell group 174); wherein, part of the cells in a first cell group among the four cell groups are used to arrange a green filter unit; part of the cells in a second cell group among the four cell groups are used to arrange a red filter unit; part of the cells in a third cell group among the four cell groups are used to arrange a blue filter unit; part of the cells in a fourth cell group among the four cell groups are used to arrange a cyan filter unit.

[0042] It should be noted that some embodiments of the present application do not limit the positional relationship between the first cell group, the second cell group, the third cell group and the fourth cell group on the minimum repeating unit, nor do they limit the specific position of each cell group on the cell. That is to say, in some embodiments, the first cell group is a 3*3 cell group located at the upper left position of the minimum repeating unit. In other embodiments, the first cell group is a 3*3 cell group located at the lower left position of the minimum repeating unit. In some embodiments, the first cell group is a 3*3 sub-cell located at the upper right position of the minimum repeating unit. In some embodiments, the first cell group is a 3*3 cell group located at the lower right position of the minimum repeating unit. Similarly, the positions of the remaining second cell group, the third cell group and the fourth cell group may also be located at the upper left, lower left, upper right or lower right cell group of the minimum repeating unit.

[0043] Combine the following Figure 3-Figure 14 The arrangement rule of the filter unit 110 of the minimum repeating unit in the filter array 11 is exemplified.

[0044] It should be noted that Figure 3-Figure 14 In the figure, R is used to indicate a red filter unit that transmits red light (e.g., with a wavelength range of approximately 605nm-700nm) and filters out light of other colors, G is used to indicate a green filter unit that transmits green light (e.g., with a wavelength range of approximately 500nm-560nm) and filters out light of other colors, B is used to indicate a blue filter unit that transmits blue light (e.g., with a wavelength range of approximately 435nm-480nm) and filters out light of other colors, C is used to indicate a cyan filter unit that transmits cyan light (e.g., with a wavelength range of approximately 480nm-490nm) and filters out light of other colors, and W is used to indicate that a full-color filter unit is set at the corresponding position (i.e., in the cell) or no filter unit is set. In some embodiments, cyan indicates a filter that can simultaneously pass blue and green light with higher efficiency (e.g., higher number of photons or higher power), that is, controlling the amount of blue light and green light (e.g., power or number of photons) passing through the cyan filter unit to be greater than the amount of red light passing can achieve the purpose of filtering other colors of light through cyan light.

[0045] Figure 3-Figure 14 Only four minimum repeating units 170 ( Figure 3-Figure 14 The dashed box in the figure shows a minimum repeating unit 170, which does not constitute a limitation on the embodiment of the present application. That is, the number of the minimum repeating units 170 of the image sensor 10 can be adjusted according to actual conditions. Figure 3-Figure 14 Only Figure 3The vertical dashed line 178 and the horizontal dashed line 179 at the center of the six-row by six-column cell group are shown as four three-row by three-column cell groups (i.e. Figure 3 The schematic diagram of the first cell group 171 at the upper left position, the second cell group 173 at the lower left position, the third cell group 174 at the lower right position, and the fourth cell group 172 at the upper right position is understood to be Figure 4-Figure 14 Each minimal repeating unit (for example, a minimal repeating unit marked by a dotted box) can also be divided into four unit cell groups by a vertical line and a horizontal line at the center of the minimal repeating unit.

[0046] In some embodiments of the present application, a colored filter unit can be set in four non-adjacent cells of the nine cells included in each of the four 3*3 cell groups obtained by dividing the six-row by six-column cell group. For example, a colored filter unit is arranged in the remaining four cells of each cell group except the main diagonal and the secondary diagonal, that is, a colored filter unit is set in the remaining four non-adjacent cells except the cells through which the main diagonal and the secondary diagonal pass. Figure 3-Figure 8 These embodiments are explained by way of example only.

[0047] like Figure 3 The arrangement regularity diagram of some of the minimum repeating units in the filter array shown in FIG. Figure 3 It can be seen that four non-adjacent cells among the nine cells of three rows by three columns included in the first cell group 171 located at the upper left position of the minimum repeating unit 170 are used to set the green filter unit G; four non-adjacent cells among the nine cells of three rows by three columns included in the second cell group 173 located at the lower left position of the minimum repeating unit 170 are used to set the red filter unit R; four non-adjacent cells among the nine cells of three rows by three columns included in the third cell group 174 located at the lower right position of the minimum repeating unit 170 are used to set the The blue filter unit B; four non-adjacent cells of the nine cells of three rows by three columns included in the fourth cell group 173 located at the upper right position of the minimum repeating unit 170 are used to set the cyan filter unit C; wherein the four cells are located on the non-main diagonal and non-sub-diagonal lines of the figure composed of the nine cells (the meaning of the non-main diagonal and non-sub-diagonal lines can be referred to the description below), that is, the four cells are located on the cells of the figure composed of the nine cells except the cells through which the main diagonal and the sub-diagonal lines pass. In some embodiments, Figure 3 Part or all of the cells in the cells that are not provided with color filter units may be provided with full-color filter units. Figure 3 No filter unit (eg, filter) may be provided in any unit cell where no color filter unit is arranged.

[0048] Figure 4 The arrangement pattern of the filter array shown is Figure 3 The arrangement pattern is rotated 90 degrees to the left. Please refer to Figure 4 , four non-adjacent cells among the nine cells of three rows by three columns included in the first cell group located at the lower left position of the minimum repeating unit 170 are used to set the green filter unit G; four non-adjacent cells among the nine cells of three rows by three columns included in the second cell group located at the lower right position of the minimum repeating unit 170 are used to set the red filter unit R; four non-adjacent cells among the nine cells of three rows by three columns included in the third cell group located at the upper right position of the minimum repeating unit 170 are used to set the blue filter unit B; four non-adjacent cells among the nine cells of three rows by three columns included in the fourth cell group located at the upper left position of the minimum repeating unit 170 are used to set the cyan filter unit C; wherein, the four cells are located on the cells of the figure composed of the nine cells except the cells through which the main diagonal and the secondary diagonal pass.

[0049] Figure 5 The arrangement rule of the filter units 110 in the filter array shown is to Figure 3 The smallest repeating unit in the arrangement pattern of the filter array is obtained by horizontally flipping it.

[0050] Figure 6 The arrangement rule of the filter units 110 in the filter array shown is to Figure 3 The minimum repeating unit in the arrangement pattern of the filter array is obtained by making it bilaterally symmetrical.

[0051] Figure 7 The arrangement rule of the filter units 110 in the filter array shown is to Figure 3 The minimum repeating unit in the arrangement pattern of the filter array is obtained by making it symmetrical from top to bottom.

[0052] Figure 8 The arrangement rule of the filter units 110 in the filter array shown is to Figure 3 The arrangement pattern of the filter array is obtained by exchanging the positions of the blue filter unit and the cyan filter unit, which are the minimum repeating units.

[0053] That is to say, Figure 3-Figure 8 The colored filter units (including: red filter units, green filter units, blue filter units and cyan filter units) are non-adjacently distributed on the non-main diagonal and non-sub-diagonal lines of each cell group. The main diagonal is the diagonal line of the cells passing through the upper left and lower right positions of each cell group (for example, Figure 3The main diagonal 180 of the first cell group, and the secondary diagonal is the cell diagonal passing through the lower right and upper right positions of each cell group (for example, Figure 3 It can be understood that in some embodiments of the present application, the non-main diagonal and non-secondary diagonal are cells that are not passed by the main diagonal and the secondary diagonal, that is, the non-main diagonal and non-secondary diagonal are cells other than the main diagonal and the secondary diagonal.

[0054] In some embodiments of the present application, colored filter units may be arranged in five non-adjacent cells of the nine cells included in each of the four 3*3 cell groups obtained by dividing the six-row by six-column matrix. For example, colored filter units may be arranged in the cells where the main diagonal and the secondary diagonal of each cell group pass. Figure 9-14 These embodiments are explained by way of example only.

[0055] like Fig. 9 In the filter array shown, five non-adjacent cells among the nine cells included in the first cell group located at the upper left position of the minimum repeating unit are used to set the green filter unit G; five non-adjacent cells among the nine cells included in the second cell group located at the lower left position of the minimum repeating unit are used to set the red filter unit R; five non-adjacent cells among the nine cells included in the third cell group located at the lower right position of the minimum repeating unit are used to set the blue filter unit B; five non-adjacent cells among the nine cells included in the fourth cell group located at the upper right position of the minimum repeating unit are used to set the cyan filter unit C; wherein the five non-adjacent cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells. In some embodiments, Fig. 9 Part or all of the cells in the cells that are not provided with color filter units may be provided with full-color filter units. Fig. 9 No filter unit (eg, filter) may be provided in any unit cell where no color filter unit is arranged.

[0056] Fig.10 The arrangement pattern of the filter array 11 is Fig. 9 Turn right 90 degrees. Fig.10As shown, five non-adjacent cells among the nine cells included in the first cell group located at the upper right position of the minimum repeating unit are used to set the green filter unit G; five non-adjacent cells among the nine cells included in the second cell group located at the upper left position of the minimum repeating unit are used to set the red filter unit R; five non-adjacent cells among the nine cells included in the third cell group located at the lower left position of the minimum repeating unit are used to set the blue filter unit B; five non-adjacent cells among the nine cells included in the fourth cell group located at the lower right position of the minimum repeating unit are used to set the cyan filter unit C; wherein, the five cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0057] Fig.11 The arrangement rule of the filter units 110 in the filter array shown corresponds to the pattern of Fig. 9 The smallest repeating unit in the arrangement pattern of the filter array is obtained by vertically flipping it.

[0058] Fig.12 The arrangement rule of the filter units 110 in the filter array shown corresponds to the pattern of Fig. 9 The minimum repeating unit in the arrangement pattern of the filter array is obtained after being bilaterally symmetric.

[0059] Fig.13 The arrangement rule of the filter units 110 in the filter array shown corresponds to the pattern of Fig. 9 The minimum repeating unit in the arrangement pattern of the filter array is obtained by being symmetrical in the upper and lower directions.

[0060] Fig.14 The arrangement rule of the filter units 110 in the filter array shown corresponds to the pattern of Fig. 9 The cyan filter unit and the green filter unit in the minimum repeating unit in the arrangement pattern of the filter array are exchanged.

[0061] That is to say, Figure 9-14 The colored filter units (including: red filter unit, green filter unit, blue filter unit and cyan filter unit) are non-adjacently distributed on the main diagonal and the sub-diagonal of each unit cell group.

[0062] In some embodiments, Figure 3-Figure 14At least some of the cells in the four three-row by three-column cell groups that are included in the minimum repeating unit and that are not provided with colored filter units are used to set up full-color filter units; wherein the colored filter units include: the red filter unit, the green filter unit, the blue filter unit, and the cyan filter unit. That is, in some embodiments, full-color filter units are arranged in some or all of the cells in the four cell groups that are not provided with colored filter units. For example, Figure 3-Figure 8 In the example, full-color filter units are arranged on all (e.g., five) or some of the cells where the main diagonal and the secondary diagonal of the minimum repeating unit pass, and no color filter may be arranged for the cells where neither the colored filter unit nor the full-color filter unit is arranged. For example, Figure 9-14 In the embodiment of the present invention, full-color filter units are arranged in all cells (four cells) or some cells where the non-main diagonal and non-sub-diagonal lines of the minimum repeating unit pass, and no color filter may be arranged in the cells where neither the colored filter unit nor the full-color filter unit is arranged. In other words, at least some of the cells in at least some of the first cell group, the second cell group, the third cell group, and the fourth cell group in some embodiments of the present application are used to set the full-color filter unit.

[0063] It should be noted that Figure 3-Figure 14 The color arrangement pattern includes the smallest repeating unit ( Figure 3-Figure 14 The dotted box in the figure shows a minimum repeating unit) can be rotated 90 degrees, flipped, symmetrical, or symmetrical, or the positions of different cell groups can be exchanged within the minimum repeating unit using a 3x3 cell group as the exchange unit (for example, Figure 8 It is an exchange Figure 3 The positions of the two cell groups in Fig.14 It is an exchange Fig. 9 The cells mentioned in the embodiments of the present application are mainly used to locate each pixel unit. Since the pixel unit corresponds to the filter unit one by one, some embodiments of the present application also involve cells when describing the arrangement rules or arrangement diagrams of the filter unit.

[0064] Combine the following Fig.15 The structure and layout of the image sensor 10 are exemplarily described for the sake of simplicity and clarity of the drawings. Fig.15 Only some units or circuits included in the image sensor 10 are provided. Fig.15For other units included in the image sensor 10 (for example, the filter units in the filter array, and the pattern of the arrangement rules of the filter units), reference may be made to the relevant descriptions and related drawings of other embodiments of the present application (for example, Figure 3-Figure 14 ).

[0065] like Fig.15 As shown, the image sensor 10 provided in some embodiments of the present application includes: a pixel array 12, wherein the pixel array 12 includes a plurality of pixel units 120; and a filter array (since the filter array 11 is a filter unit tiled on the upper layer of the pixel array 12, it is not shown in FIG. Fig.15 The filter unit located on the upper layer of each pixel unit 120 is shown).

[0066] It can be understood that the arrangement rules of the filter units in the filter array 11 provided in some embodiments of the above application are the same. Fig.15 The image sensor 10 includes a filter array including a plurality of filter units, the filter array is positioned above the pixel array and the filter units are optically coupled to the pixel units, the minimum repeating unit of the pattern of the filter array is a cell group of six rows by six columns, the cell group of six rows by six columns includes four cell groups of three rows by three columns; wherein, some cells in a first cell group of the four cell groups are arranged with green filter units; some cells in a second cell group of the four cell groups are arranged with red filter units; some cells in a third cell group of the four cell groups are arranged with blue filter units; and some cells in a fourth cell group of the four cell groups are arranged with cyan filter units.

[0067] In some embodiments, at least some of the cells in at least some of the first cell group, the second cell group, the third cell group, and the fourth cell group that are not provided with colored filter units are used to set full-color filter units, wherein the colored filter units include red filter units, green filter units, blue filter units, and cyan filter units.

[0068] In some embodiments, four non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; four non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; four non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; four non-adjacent cells out of the nine cells included in the fourth cell group are used to set the cyan filter unit; wherein the four cells are located on the cells other than the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0069] In some embodiments, five non-adjacent cells out of the nine cells included in the first cell group are used to set the green filter unit; five non-adjacent cells out of the nine cells included in the second cell group are used to set the red filter unit; five non-adjacent cells out of the nine cells included in the third cell group are used to set the blue filter unit; five non-adjacent cells out of the nine cells included in the fourth cell group are used to set the cyan filter unit; wherein the five cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

[0070] In some embodiments, at least some of the multiple cells in the four three-row by three-column cell groups that are not arranged with colored filter units are provided with full-color filter units; wherein the colored filter units include: the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit.

[0071] To avoid repetition, the specific features of the image sensor provided in the embodiment of the present application including the filter array 11 can refer to the arrangement rules of the filter units provided in the embodiment of the present application and Figure 3-Figure 14 I will not go into too much detail about the content here.

[0072] like Fig.15 As shown, in some embodiments, the image sensor further includes a reading circuit 330, which is configured to combine and read the photosensitive currents of the photosensitive elements corresponding to the filter units of the same color in the same cell group. Figure 3 The photosensitive currents of the four pixel units (e.g., the photodiodes included in the pixel units) located by the four green filter units in the upper left cell group 171 in the minimum repeating unit in the dotted box; the reading circuit 330 combines and reads the induced currents of the five pixel units located by the five full-color filter units (assuming that the five positions are all arranged with full-color filter units) in the upper left cell group 171. For example, the reading circuit 330 combines and reads Fig. 9 The induced electrical signals of the pixel units located by the five red filter units in the lower left cell group of the minimum repeating unit in the dotted box are read, and the induced electrical signals of the pixel units located by the remaining cells in the lower left cell group (for example, the four cells may be provided with full-color filters corresponding to full-color filter units or may not be provided with any filters) are read. It should be noted that in order to read out the induced electrical signals of each pixel unit, each pixel unit needs to be addressed. As an example, Fig.15The address selection process of a certain pixel unit is: under the control of the row driving circuit 310, a certain row address line 311 is selected and under the control of the column driving circuit 320, a certain column address line 321 is selected, so as to read the current signal induced by the pixel unit at the intersection of the row address line 311 and the column address line 321 through the reading line 312 connected to the reading circuit 330. Compared with the solution of the prior art, some embodiments of the present application need to merge and read the induced current of the pixel units located by the filter units of the same color in each cell group in a minimum repeating unit. The color types of these embodiments of the present application include: red, green, blue, cyan and full color, wherein the full color can also be replaced by not arranging any filter unit, that is, the reading circuit 330 will merge and read the induced current of the pixel units corresponding to the cells arranged with the colored filter units in each cell group, regardless of whether these cells are provided with full color filter units or not provided with any filter units.

[0073] like Fig.15 As shown, in some embodiments, the image sensor further includes a conversion device 340, and the conversion device 340 is configured to convert the electrical signals sensed by the multiple photosensitive elements corresponding to the minimum repeating unit into an output signal in the Bayer format, so that the image processing circuit can analyze the image. As an example, the conversion device 340 is configured to: determine the target weight of the neural network model according to the resolution type, wherein the resolution type includes a half-size mode and a full-size mode; and convert the input electrical signal to be converted into the output signal in the Bayer format according to the neural network model.

[0074] That is to say, some embodiments of the present application require the conversion device 340 to perform signal processing before sending the induced electrical signals of the pixel units read by the reading circuit 330 to the image processing circuit. This is because, in some embodiments, the image processing circuit cannot directly parse the electrical signals induced by the pixel units corresponding to the arrangement of the filter units provided in some embodiments of the present application, so the read data needs to be converted into the following two types of processing: For the half-size mode, that is, the photosensitive currents corresponding to the pixel units corresponding to the filter units of the same color in the same cell group are merged and read. At this time, the current signals induced by the 36 pixel units corresponding to the minimum repeating unit (that is, the 36 filter units included in the matrix of six rows and six columns) are merged into 8 pixel read values ​​(specifically, after the merged reading, two induced electrical signals are read in each cell group, for example, Figure 3 The currents sensed by the four pixel units corresponding to the four green filter units of the upper left cell group 171 are combined into one readout current. Figure 3The currents sensed by the five pixel units corresponding to the filter units corresponding to the remaining five positions of the upper left cell group 171 are combined into one readout current), and then the conversion device 340 according to some embodiments of the present application can convert the 8 pixel readout values ​​into 4 rearranged standard Bayer format pixel output values; for the full-size mode, that is, the current values ​​sensed by the 36 pixel units corresponding to the minimum repeating unit are read out respectively, that is, each pixel unit reads out a sensed current value, and then the 36 sensed current values ​​read out are converted into a 6x6 standard Bayer format by the conversion device of the embodiment of the present application. For example, the conversion device 340 can be a separate bridge chip that obtains the readout current value of the pixel unit through the signal line; in some embodiments, the conversion device 340 can be stacked on a chip with the image sensor.

[0075] In some embodiments, the conversion device 340 can be implemented by a programmable neural network processor (NPU). As an example, the conversion device 340 for the full-size mode can be implemented by the NPU in the following manner: S1. Read the sensed signal values ​​(for example, sensed analog current values) of the photosensitive units of the pixel units 120 in sequence and enter the buffer area of ​​the NPU in sequence; S2. When there is enough data in the buffer area, start the NPU to use the pre-trained neural network weights to calculate the processed results; S3. The results calculated by the NPU are sent to the input and output units of the image sensor through the bus. It can be understood that for the required neural network weights, the following method can be used: prepare a large number of RGB images; arrange the colors according to the color arrangement method in the embodiment of the present application (i.e. Figure 3-Figure 14 ) arranges filter units and simulates the corresponding photosensitive unit (for example, the photodiode included in the pixel unit) readout value and the real value (ground truth) in Bayer format; adds simulated noise (for example, independent and identically distributed standard Gaussian noise) to the readout value of the photosensitive unit; takes the L1 norm of the predicted value of the neural network and the correct labeled value ground truth as the objective function, and uses the stochastic gradient descent SGD algorithm to adjust the weight of the neural network until convergence.

[0076] like Fig.16 As shown, some embodiments of the present application further provide a digital camera 400, which includes an image sensor 10 and a lens 410. The specific structure of the image sensor 10 can be referred to the above description, and will not be described in detail here to avoid repetition.

[0077] like Fig.17 As shown, the embodiment of the present application provides a mobile phone terminal 500, which includes a memory 510, a processor 520 and an image sensor 10. The specific structure of the image sensor 10 can refer to the above description, and no further details are given here to avoid repetition.

[0078] like Fig.18 As shown, the embodiment of the present application provides a video monitoring system 600, which includes a processor 610, a display 620, and an image sensor 10. The specific structure of the image sensor 10 can refer to the above description, and will not be described in detail here to avoid repetition.

[0079] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0081] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0082] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0083] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0084] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A filter array, characterized in that: The filter array includes a plurality of tiled minimal repeating units, each of which includes a cell group of six rows by six columns, and the cell group of six rows by six columns includes four cell groups of three rows by three columns; wherein, Part of the cells in the first cell group of the four cell groups are used to arrange green filter units; Part of the cells in the second cell group among the four cell groups are used to arrange red filter units; Part of the cells in the third cell group among the four cell groups are used to arrange blue filter units; Part of the cells in the fourth cell group of the four cell groups are used to arrange cyan filter units; Among them, at least some of the cells in at least some of the cell groups in the first cell group, the second cell group, the third cell group and the fourth cell group that are not provided with colored filter units are used to set full-color filter units, wherein the colored filter units include red filter units, green filter units, blue filter units and cyan filter units.

2. The filter array according to claim 1, wherein: Four non-adjacent cells among the nine cells included in the first cell group are used to set the green filter unit; Four non-adjacent cells among the nine cells included in the second cell group are used to set the red filter unit; Four non-adjacent cells among the nine cells included in the third cell group are used to set the blue filter unit; Four non-adjacent cells among the nine cells included in the fourth cell group are used to set the cyan filter unit; The four non-adjacent cells are located on cells other than the main diagonal and the secondary diagonal of the figure composed of the nine cells.

3. The filter array according to claim 1, wherein: Five non-adjacent cells among the nine cells included in the first cell group are used to set the green filter unit; Five non-adjacent cells among the nine cells included in the second cell group are used to set the red filter unit; Five non-adjacent cells among the nine cells included in the third cell group are used to set the blue filter unit; Five non-adjacent cells among the nine cells included in the fourth cell group are used to set the cyan filter unit; The five non-adjacent cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

4. The filter array according to claim 2 or 3, characterized in that: At least some of the cells in the four groups of cells of three rows by three columns that are not provided with color filter units are used to set up full-color filter units; Wherein, the colored filter unit includes: the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit.

5. An image sensor, characterized in that: The image sensor comprises: A pixel array, the pixel array comprising a plurality of pixel units; and A filter array, the filter array comprising a plurality of filter units, the filter array being positioned above the pixel array and the filter units being optically coupled to the pixel units, the minimum repeating unit of the pattern of the filter array being a cell group of six rows by six columns, the cell group of six rows by six columns comprising four cell groups of three rows by three columns; wherein, Some cells in a first cell group of the four cell groups are arranged with green filter units; Some cells in the second cell group of the four cell groups are arranged with red filter units; Some cells in the third cell group among the four cell groups are arranged with blue filter units; Some cells in a fourth cell group among the four cell groups are arranged with cyan filter units; Among them, at least some of the cells in at least some of the cell groups in the first cell group, the second cell group, the third cell group and the fourth cell group that are not provided with colored filter units are used to set full-color filter units, wherein the colored filter units include red filter units, green filter units, blue filter units and cyan filter units.

6. The image sensor according to claim 5, characterized in that Four non-adjacent cells among the nine cells included in the first cell group are used to set the green filter unit; Four non-adjacent cells among the nine cells included in the second cell group are used to set the red filter unit; Four non-adjacent cells among the nine cells included in the third cell group are used to set the blue filter unit; Four non-adjacent cells among the nine cells included in the fourth cell group are used to set the cyan filter unit; The four non-adjacent cells are located on cells other than the main diagonal and the secondary diagonal of the figure composed of the nine cells.

7. The image sensor according to claim 6, wherein: Five non-adjacent cells among the nine cells included in the first cell group are used to set the green filter unit; Five non-adjacent cells among the nine cells included in the second cell group are used to set the red filter unit; Five non-adjacent cells among the nine cells included in the third cell group are used to set the blue filter unit; Five non-adjacent cells among the nine cells included in the fourth cell group are used to set the cyan filter unit; The five non-adjacent cells are located on the main diagonal and the secondary diagonal of the figure composed of the nine cells.

8. The image sensor according to claim 6 or 7, characterized in that: At least some of the plurality of cells in the four three-row by three-column groups of cells not provided with color filter units are used to provide full-color filter units; Wherein, the colored filter unit includes: the red filter unit, the green filter unit, the blue filter unit and the cyan filter unit.

9. The image sensor according to claim 6 or 7, characterized in that: The image sensor further includes a reading circuit configured to combine and read the photosensitive currents of the photosensitive elements corresponding to the filter units of the same color in the same cell group.

10. The image sensor according to claim 6 or 7, characterized in that: The image sensor further includes a conversion device configured to convert electrical signals sensed by a plurality of photosensitive elements corresponding to the minimum repeating unit into output signals in a Bayer format for image processing circuits to analyze images.

11. The image sensor according to claim 10, wherein: The conversion device is configured to: Determining a target weight of a neural network model according to a resolution type, wherein the resolution includes a half-size mode and a full-size mode; The input electrical signal to be converted is converted into an output signal in the Bayer format according to the neural network model.

12. A digital camera, characterized in that: The camera comprises: Lenses; and The filter array according to any one of claims 1 to 4 or the image sensor according to any one of claims 5 to 11.

13. A mobile phone terminal, characterized in that: The mobile phone terminal comprises a memory, a processor, and the filter array according to any one of claims 1 to 4 or the image sensor according to any one of claims 5 to 11.

14. A video surveillance system, characterized in that: The video monitoring system comprises a memory, a display, and a filter array as claimed in any one of claims 1 to 4 or an image sensor as claimed in any one of claims 5 to 11.

Images