Imaging chip comprising multispectral imaging unit

Abstract

The utility model relates to an imaging chip comprising a multispectral imaging unit. The imaging chip comprising the multispectral imaging unit comprises imaging pixel units arranged periodically, and each imaging pixel unit comprises imaging pixels based on a predetermined imaging array and used for imaging and multispectral pixels different from the imaging pixels and used for multispectral imaging, the multi-spectral pixels replace one or more imaging pixels to form at least one multi-spectral imaging unit, and the sum of the number of the imaging pixels in the multi-spectral imaging unit and the number of the pixel types of the multi-spectral pixels is larger than or equal to five. The multispectral imaging unit forms a rectangular array composed of imaging pixels and multispectral pixels. Thus, the imaging pixel unit used for imaging is provided with the imaging pixel containing five or more kinds of pixels and the multispectral pixel, spectral imaging and color imaging are integrated in the same imaging chip, and more accurate color restoration is achieved.

Description

Technical Field

[0001] This application relates to the field of imaging technology, and more specifically, to an imaging chip comprising a multispectral imaging unit. Background Technology

[0002] Spectral imaging technology is an emerging technology that combines spectral analysis and imaging technology. It can obtain three-dimensional data cubes containing two-dimensional spatial information and spectral information, which surpasses the perception ability of the human eye. It has important application prospects in many fields such as disease diagnosis and treatment, health monitoring, and precision agriculture.

[0003] A Bayer array is a widely used color filter array used to convert light captured by an image sensor into a color image. Commonly, it consists of red, green, and blue filters, or red, yellow, and blue filters arranged in repeating four-pixel units, with the number of green filters being twice that of red and blue. A Bayer array can also contain four colors of filters. During shooting, the image sensor acquires the intensity information of different colors of light through the Bayer array. Then, a demosaicing algorithm is used to convert this information into a complete color image. Bayer array-based color imaging technology is central to modern mobile phone photography, driving the rapid development of mobile imaging technology.

[0004] On the one hand, with the development of the smart era, spectral imaging devices are moving towards miniaturization and micro-miniaturization, with their size continuously shrinking, enabling them to be integrated into consumer electronic devices such as smartphones and tablets. On the other hand, consumers have increasingly higher expectations for mobile phone image quality, especially for the accuracy of color reproduction. However, existing Bayer arrays can only acquire image information from three or four channels, making it difficult to further improve the accuracy of color reproduction.

[0005] In existing technologies, obtaining multi-channel spectral information requires adding an additional spectral sensor. However, this sensor can only acquire the average spectral information of the imaging field of view, missing image information. Furthermore, adding an extra spectral sensor affects the phone's appearance design, increasing internal space, power consumption, and cost. While existing technologies can achieve multispectral imaging, they still require an additional component and suffer from issues such as aligning the spectral image with the main camera image and synchronizing exposure time.

[0006] Therefore, there is a need to provide an imaging chip solution with improved spectral and color imaging performance. Utility Model Content

[0007] This application provides an imaging chip that includes a multispectral imaging unit. By setting imaging pixels containing five or more types of pixels and multispectral pixels in the imaging pixel unit used for imaging, spectral imaging and color imaging are integrated in the same imaging chip, thereby achieving more accurate color reproduction.

[0008] According to one aspect of this application, an imaging chip including a multispectral imaging unit is provided, comprising: periodically arranged imaging pixel units, each of the imaging pixel units comprising imaging pixels for imaging based on a predetermined imaging array and multispectral pixels of a different type for multispectral imaging, wherein the multispectral pixels form at least one multispectral imaging unit by replacing one or more imaging pixels, and the sum of the number of pixel types of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five, and the multispectral imaging unit is formed as a rectangular array composed of imaging pixels and multispectral pixels.

[0009] In the imaging chip containing the above-mentioned multispectral imaging units, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0010] In the imaging chip containing the multispectral imaging unit, the multispectral pixel of one of the multispectral imaging units is located in a different imaging pixel unit.

[0011] In the imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0012] In the imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0013] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0014] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0015] In the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral imaging unit, and less than or equal to 50%.

[0016] In the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0017] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 0.1%.

[0018] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0019] In the imaging chip containing the above-mentioned multispectral imaging units, each multispectral imaging unit contains at least one type of multispectral pixel or the number of imaging pixels is greater than 1.

[0020] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 4×4 rectangular array containing five types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

[0021] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0022] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 4×4 rectangular array containing seven types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

[0023] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0024] In the imaging chip containing the multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is 8×8. The horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel, and the ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 3.125%.

[0025] In the imaging chip containing the multispectral imaging unit, the imaging pixel is an R, G, B filter, and the multispectral pixel is a filter of other types besides the R, G, B filter.

[0026] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0027] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0028] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0029] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

[0030] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0031] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

[0032] The imaging chip containing a multispectral imaging unit provided in this application embodiment can integrate spectral imaging and color imaging in the same imaging chip by setting imaging pixels containing five or more types of pixels and multispectral pixels in the imaging pixel unit used for imaging, thereby achieving more accurate color reproduction. Attached Figure Description

[0033] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0034] Figure 1 The diagram illustrates a Bayer array in an existing imaging chip.

[0035] Figure 2 The illustration shows a variation of the Bayer array in an existing imaging chip.

[0036] Figure 3 The illustration shows a schematic diagram of the array arrangement of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0037] Figure 4 The illustration shows a schematic diagram of the imaging principle of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0038] Figure 5The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0039] Figure 6 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0040] Figure 7 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0041] Figure 8 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0042] Figure 9 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0043] Figure 10 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Detailed Implementation

[0044] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0045] As mentioned above, Bayer arrays are widely used in current imaging chips. Invented by Bryce Bayer in 1976, the Bayer array is a color filter array (CFA) widely used in digital cameras and mobile phone cameras to convert the light captured by the image sensor into a color image. The basic unit of a Bayer array is a 2×2 filter matrix, consisting of two green, one red, and one blue filters, typically arranged in an "RGGB" pattern, such as... Figure 1 As shown. Here, Figure 1 The diagram illustrates a Bayer array in an existing imaging chip. In the Bayer array, the number of green filters is twice that of red and blue filters because the human eye is more sensitive to green. This configuration improves image brightness and detail resolution.

[0046] When light passes through a Bayer array and reaches the image sensor, each pixel can only detect the intensity of one color of light (red, green, or blue). Therefore, the raw data acquired by the sensor is a black-and-white image, called a "Bayer pattern," which needs to be demosaiced to generate a complete color image. Demosaic algorithms calculate the RGB value of each pixel using interpolation methods. Common demosaic algorithms include bilinear interpolation, malvar interpolation, and more complex adaptive algorithms. The choice of different algorithms affects image sharpness, color accuracy, and noise levels.

[0047] Bayer arrays are widely used in digital cameras, mobile phone cameras, and surveillance equipment. With technological advancements, variations of Bayer arrays have increased, such as RYYB, RGBW, and RGBIR, which consist of three, four, or more filters (the blank area can also be considered a filter). Figure 2 As shown in (a), (b), and (c), this is to improve imaging performance in specific application scenarios. Here, Figure 2 The diagram illustrates variations of Bayer arrays in existing imaging chips. For example, the RYYB array replaces the green (G) filter with a yellow (Y) filter. The yellow filter allows more light to pass through, thus enabling the RYYB array to capture more light in low-light environments, improving image brightness and quality. However, the RYYB array also requires complex algorithms to accurately reproduce image colors, potentially leading to color distortion. The RGBW array replaces part of the green (G) filter with a white (W) filter. The white filter allows all light to pass through, improving image brightness and dynamic range. However, the addition of the white filter also makes color reproduction more complex. The RGBIR array adds an infrared (IR) filter to the traditional RGB filter, allowing simultaneous acquisition of visible and infrared information. In nighttime or low-light environments, infrared light provides additional information, improving image sharpness. However, specialized algorithms are needed to process and integrate visible and infrared light information. These variations of Bayer arrays, through different filter combinations and structural designs, optimize image quality and functionality for specific application scenarios.

[0048] Based on this, this application proposes an imaging chip that embeds a multispectral imaging unit in a Bayer array to improve the color accuracy of imaging and enable spectral imaging to assist applications such as mobile phone white balance. Here, the Bayer array in the imaging chip of this application can be a traditional RGGB arrangement or a variation of the Bayer array, which will not be distinguished below.

[0049] Taking a Bayer array with an "RGGB" arrangement as an example, the arrangement after embedding multispectral imaging units (also known as multispectral units) is as follows: Figure 3As shown. Here, Figure 3 The illustration shows a schematic diagram of the array arrangement of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 3 As shown, the Bayer array arrangement after embedding multispectral units is still a periodic array, and the smallest repeating unit of the periodic array is as follows: Figure 3 As shown in the solid-line rectangle, n (n≥1) sets of multispectral imaging units are embedded in each minimum repeating unit by replacing the original pixels in the Bayer array with multispectral pixels, so that each minimum repeating unit of the periodic array includes n multispectral imaging units. In the embodiments of this application, the embedded multispectral pixels are obtained by replacing the R, G, or B filters at one or more positions in the "RGGB" arrangement with filters other than those arranged in the Bayer array, such as R, G, and B filters in the "RGGB" arrangement. That is, each multispectral pixel corresponds to a filter of one color. It should be noted that two or more filter materials forming a multispectral pixel can also be considered as a filter of one color (based on the final transmission spectrum).

[0050] The multispectral imaging unit is shown as the dashed rectangle in the figure. When one or more multispectral imaging units are included, they may or may not overlap. Each multispectral imaging unit is a rectangular array of size h0×w0, containing multiple filters including R, G, and B filters, and the number of filter types in each multispectral imaging unit is m (m≥5). Thus, based on m types of filters, spectral reconstruction can be performed, thereby achieving spectral imaging. A more accurate Color Correction Matrix (CCM) can also be calculated for more accurate color reproduction, which will be explained in detail below. It should be noted that, preferably, the multispectral pixels of the same multispectral imaging unit are located within the smallest repeating unit, but this is not a limitation; filters constituting the same multispectral unit may also be located in different smallest repeating units.

[0051] Therefore, the imaging chip including a multispectral imaging unit according to embodiments of this application includes: periodically arranged imaging pixel units, each imaging pixel unit comprising imaging pixels for imaging based on a predetermined imaging array and multispectral pixels of a different type for multispectral imaging, wherein the multispectral pixels form at least one multispectral imaging unit by replacing one or more imaging pixels, and the sum of the number of pixel types of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five, and the multispectral imaging unit is formed as a rectangular array composed of imaging pixels and multispectral pixels. Wherein, the imaging pixels located in the multispectral imaging unit are also used for spectral imaging.

[0052] In other words, the imaging pixel unit here refers to the smallest repeating unit mentioned above, and it includes imaging pixels based on a predetermined imaging array, such as a Bayer array, for example, R, G, and B pixels, and multispectral pixels for multispectral imaging, such as C and Y pixels. Those skilled in the art will understand that the predetermined imaging array can also be other pixel array arrangements for imaging besides a Bayer array. In this embodiment, at least one multispectral imaging unit is formed in the imaging pixel unit by replacing one or more imaging pixels, such as R, G, and B pixels, in the predetermined imaging array with multispectral pixels of a different type than the imaging pixels, such as C and Y pixels. Furthermore, in order to achieve multispectral imaging through the multispectral imaging unit, the sum of the number of imaging pixels and the number of pixel types of the multispectral pixels in each multispectral imaging unit is greater than or equal to five, that is, the number of filter types included in the multispectral imaging unit for realizing pixels is greater than or equal to five. Moreover, the multispectral imaging unit is formed as a rectangular array composed of imaging pixels, such as R, G, and B pixels, and multispectral pixels, such as C and Y pixels.

[0053] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of the present application, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0054] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, the multispectral pixels of the multispectral imaging unit are located in different imaging pixel units.

[0055] The principle of spectral reconstruction calculation in the multispectral imaging unit of the imaging chip including the multispectral imaging unit according to the embodiments of this application will be explained below. Here, the spectrum of the incident light, that is, the intensity signal of the incident light at different wavelengths λ, is denoted as x(λ), and the transmission spectrum curves of various filters in the multispectral imaging unit are denoted as T. i (λ) (i = 1, 2, ..., m), each filter has a different transmission spectrum. The incident light, after being modulated by each filter, is detected by the corresponding image sensor pixels below, and the detected light intensity is y. i Here, each filter corresponds to one or more physical pixels of the image sensor. The image sensor's response curve is R(λ), representing the sensor's sensitivity to light of different wavelengths. The relationship between the incident light spectrum and the detected light intensity of the image sensor can be expressed by the following formula:

[0056] y i =∫x(λ)*T i (λ)*R(λ)dλ

[0057] Discretizing the wavelength domain allows the above equation to be transformed into:

[0058]

[0059] The corresponding matrix form is:

[0060]

[0061] In short: y = Ax, where

[0062] A ij =T i (λ j )*R(λ j )

[0063] Matrix A represents the system's response to incident light of different wavelengths, determined by the filter transmittance and the quantum efficiency of the image sensor, and can be obtained by calibrating the system. By solving the above equations, the incident light spectrum can be reconstructed. After calculating the spectrum at the location of each multispectral imaging unit, spectral imaging (either spectral reconstruction or spectral imaging) is achieved.

[0064] It should be noted that the process of acquiring spectral information based on the above-mentioned multispectral imaging unit is not limited to calculating spectral reconstruction. A multispectral imaging unit can also be implemented using a narrowband filter, for example, by using a dispersive element for spectral dispersion or by using a resonant cavity, metal plasma, etc., to achieve narrowband filtering. Thus, the spectral information of the incident light can be directly obtained without calculating reconstruction. In the embodiments of this application, the filter can be implemented as a metasurface, photonic crystal, nanopillar, multilayer film, filter material (dye, pigment, etc.), quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties.

[0065] Based on the principle of spectral reconstruction described above, in order to accurately reconstruct the spectrum at the location of each multispectral imaging unit, it is required that the incident light spectrum received by each filter within the multispectral imaging unit is the same. In order to meet this condition as much as possible, in the embodiments of this application, the size of a single multispectral imaging unit cannot be too large. Specifically, for example, h0≤10, w0≤10, preferably h0≤4, w0≤4, h0 and w0 can be different. Preferably, the multispectral imaging unit is rectangular (defined and divided into rectangles according to requirements; for example, when it is not a standard rectangle, it can be expanded into a rectangle, which is also covered by this utility model).

[0066] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0067] Furthermore, in the imaging chip including multispectral imaging units according to the embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0068] The following describes the principle behind the use of multispectral imaging units to calculate more accurate color correction matrices, such as... Figure 4 As shown. Figure 4 The illustration shows a schematic diagram of the imaging principle of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Specifically, based on... Figure 3The array arrangement of the imaging chip containing multispectral imaging units shown can produce a raw image of size H×W after imaging a scene. First, multispectral pixels embedded in the multispectral imaging units (which serve as filters for other types of multispectral pixels replacing the imaging pixels in the Bayer array) are removed as defective pixels. Defective pixel removal is generally achieved through neighborhood interpolation, that is, interpolating the values ​​of normal pixels in the same channel surrounding the defective pixel to replace its value. Common interpolation methods include averaging and bilinear interpolation. After defective pixel removal, de-mosaicing is performed, resulting in a raw RGB image of size H×W×3, denoted as image A. Furthermore, the pixel values ​​of the m types of filters contained in each multispectral imaging unit are extracted from the original raw image by averaging the pixel values ​​of the same type of filter in the multispectral unit, resulting in a raw multichannel image of size N×m, where N corresponds to the number of all multispectral imaging units. Furthermore, multiplying the last dimension of the raw multi-channel image by a CCM of size m×3 yields a high-fidelity XYZ image of size N×3, denoted as image B. It should be noted that the CCM can be pre-calibrated and calculated using the multispectral imaging unit, where XYZ represents tristimulus values. Since m (m>3) channels are used to calculate the tristimulus values, the calculation is more accurate than using only the RGB channels, resulting in a high-fidelity XYZ image. Additionally, taking only the RGB components of the last dimension of the raw multi-channel image yields a raw RGB multi-channel image of size N×3, denoted as image C. By extracting the XYZ values ​​and corresponding raw RGB values ​​of all pixels in images B and C, an N×3 XYZ matrix and an N×3 raw RGB matrix can be obtained. Using the least squares method, a 3×3 CCM (called the dynamic CCM) can be calculated. This dynamic CCM can map the above raw RGB matrix to an XYZ matrix. The calculation method of the dynamic CCM is as follows:

[0069]

[0070] Among them, M RGB and M XYZ These represent the raw RGB matrix and the XYZ matrix, respectively. Pinv(M) represents matrix multiplication. RGB ) is M RGBThe pseudo-inverse is then calculated. Finally, multiplying the last dimension of image A by the dynamic CCM yields a high spatial resolution, high-fidelity XYZ image of size H×W×3. This image is then converted to the sRGB color gamut using a standard transform, resulting in a more accurate sRGB image of size H×W×3. As seen in the calculations above, the dynamic CCM is the optimal CCM calculated based on the color components of the target scene. This means that the calculated dynamic CCM changes with the shooting scene, achieving higher color reproduction accuracy compared to traditional CCM.

[0071] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the imaging process includes:

[0072] Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively;

[0073] After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit (i.e., removing multispectral pixels of other types besides the imaging pixel types in the Bayer array), interpolation is performed to obtain an original RGB image of size H×W×3.

[0074] The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number in the multispectral imaging unit.

[0075] The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3;

[0076] Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3;

[0077] The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix;

[0078] Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and

[0079] The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

[0080] Furthermore, based on the above, it is also possible to determine whether a multispectral imaging unit is located at the edge of an image by calculating the color gradient within the multispectral imaging unit. The color gradient represents the degree of color change within the multi-source imaging unit and can be calculated using the following formula:

[0081] grad=max(f(S1),f(S2),…,f(S m ))

[0082] S i ∈ Multispectral Imaging Unit

[0083] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i This refers to the pixel value corresponding to the i-th filter within the multispectral imaging unit. When the color gradient is less than a certain threshold (for example, a threshold of 0.05 can be used), the multispectral imaging unit is considered not to be at the image edge. By calculating the color gradient of each multispectral imaging unit, multispectral imaging units not at the edge can be filtered out. Then, in images B and C, only those pixel values ​​not at the edge (where N ≤ the total number of multispectral imaging units) are extracted for dynamic CCM calculation. This avoids the impact of edge-induced false color on the accuracy of dynamic CCM calculation, achieving more accurate dynamic CCM calculation.

[0084] That is, in the aforementioned imaging chip containing multispectral imaging units, extracting the pixel values ​​of m types of filters (which serve as multispectral pixels) contained within the multispectral imaging units from the original image to obtain an original multichannel image of size N×m includes: calculating the color gradient of each multispectral imaging unit based on the following formula:

[0085] grad=max(f(S1),f(S2),…,f(S m ))

[0086] S i ∈ Multispectral Imaging Unit

[0087] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit;

[0088] The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and

[0089] Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

[0090] In the above calculation process, on the one hand, in order to reduce the impact of bad pixel removal on image spatial resolution, the density of bad pixels should be as small as possible, that is, the embedding density of multispectral imaging units should be as small as possible. Here, it can be specified that the proportion of multispectral pixels in the embedded multispectral imaging units (excluding imaging pixels used for imaging, such as R, G, B pixels or R, Y, B pixels used for multispectral imaging) to the total number of pixels in the imaging chip should be less than or equal to 25%. That is, the multispectral pixels are other types of filters besides the filters used for imaging included in the predetermined imaging array. For example, when the predetermined imaging array is a Bayer array, and the imaging pixels it includes are RGGB pixels, the multispectral pixels can be variants of C, Y, M, G, W, IR, etc. If the imaging pixels of the Bayer array are RYYB, the multispectral pixels can be C, M, G and their variants, W, IR, etc.; preferably, its proportion should be less than or equal to 10%. Furthermore, preferably, within each multispectral imaging unit, the proportion of multispectral pixels should be greater than or equal to 2 / (h0×w0) and less than or equal to 50%, preferably less than or equal to 25%. On the other hand, to ensure the accuracy of dynamic CCM calculation, the number of multispectral imaging units cannot be too small, i.e., N cannot be too small. Here, it can be specified that the total number of multispectral pixels in all multispectral imaging units accounts for a proportion greater than 0.1% of the total number of pixels in the imaging chip, preferably greater than 1%. In addition, to facilitate the calculation of color gradients within multispectral imaging units, it is preferably required that the number of at least one type of filter in each multispectral imaging unit is greater than 1.

[0091] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0092] Furthermore, in the imaging chip containing the multispectral imaging unit described above, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0093] Furthermore, in the imaging chip containing the multispectral imaging units, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral pixel unit, and less than or equal to 50%.

[0094] Furthermore, in the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0095] Furthermore, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels of the multispectral imaging unit to the total number of pixels of the imaging chip is greater than 0.1%.

[0096] Furthermore, in the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0097] Furthermore, in the imaging chip containing multispectral imaging units according to embodiments of this application, each multispectral imaging unit contains at least one type of pixel, i.e., the number of multispectral pixels or imaging pixels is greater than 1.

[0098] Below, specific examples of an imaging chip including a multispectral imaging unit according to embodiments of this application will be described.

[0099] Figure 5 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 5 As shown, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 4×4 rectangle (h0=4, w0=4) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, the multispectral pixels can also be other types of filters, such as C and M filters; or Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. In this arrangement, G accounts for 50%, and C and Y each account for 2 / ((2a+4)×(2b+4)). Furthermore, in this example, the multispectral imaging unit may also include a predetermined imaging array composed of other types of imaging pixels, such as imaging arrays other than Bayer arrays, including other types of imaging pixels, such as RYYB, RGBW, etc., as are the other examples described below.

[0100] Figure 6 The illustration shows a schematic diagram of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 6As shown, unlike Example 1, one set of C and Y in the multispectral imaging unit is replaced with two other filters, M and W, while the other set of C and Y remains unchanged. This way, each multispectral unit contains 7 filters (m=7), enabling more accurate spectral reconstruction and color reproduction.

[0101] Figure 7 The illustration shows a schematic diagram of Example 3 of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 7 As shown, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least two multispectral imaging units (n=2). Figure 7 The example shows two multispectral imaging units, but this example may include more than two. The smallest repeating unit is 8×8, and the horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel. In this arrangement, C and Y each account for 3.125%.

[0102] Figure 8 The illustration shows a schematic diagram of Example 4 of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 8 As shown, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, G accounts for 50%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0103] Figure 9The illustration shows a schematic diagram of Example 5 of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 9 As shown, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains two multispectral imaging units (n=2). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0104] Figure 10 The illustration shows a schematic diagram of Example Six of an imaging chip including a multispectral imaging unit according to an embodiment of this application. Figure 10 As shown, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 2 / ((2a+4)×(2b+4)).

[0105] The above Figures 5-10 The corresponding example uses RGGB as an example, but it can cover other types, such as RYYB, where the Y and G filters are considered to be replaced. Other types of Bayer arrays can also follow the above principle; therefore, although RGGB is used as an example, it covers other array types. The description of the proportions of R, G, and B filters in the embodiment can be understood as a specific description of the proportions of imaging pixels in the Bayer array, and does not limit the Bayer array to an RGGB array.

[0106] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0107] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0108] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0109] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0110] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An imaging chip comprising a multi-spectral imaging unit, characterized in that, The application comprises: Periodically arranged imaging pixel units, each of which contains imaging pixels for imaging and multispectral pixels for multispectral imaging of different kinds from the imaging pixels based on a predetermined imaging array, the multispectral pixels form at least one multispectral imaging unit by replacing one or more imaging pixels, and the sum of the pixel kinds of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five, and the multispectral imaging unit is formed into a rectangular array composed of imaging pixels and multispectral pixels.

2. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units are overlapped or not overlapped in the imaging pixel units.

3. The imaging chip comprising a multispectral imaging unit of claim 2, wherein, The multispectral pixels of one of the multispectral imaging units are located in different imaging pixel units.

4. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

5. The imaging chip comprising a multispectral imaging unit of claim 4, wherein, The length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

6. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels of the imaging chip is less than or equal to 25%.

7. The imaging chip comprising a multispectral imaging unit of claim 6, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels of the imaging chip is less than or equal to 10%.

8. The imaging chip comprising a multispectral imaging unit of claim 7, wherein, The ratio between the number of multispectral pixels in each of the multispectral imaging units and the total number of pixels of the multispectral imaging unit is greater than or equal to 2 / total number of pixels of the multispectral imaging unit, and less than or equal to 50%.

9. The imaging chip comprising a multispectral imaging unit of claim 8, wherein, The ratio between the number of multispectral pixels in each of the multispectral imaging units and the total number of pixels of the multispectral imaging unit is less than or equal to 25%.

10. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels of the imaging chip is greater than 0.1%.

11. The imaging chip comprising a multispectral imaging unit of claim 10, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels of the imaging chip is greater than 1%.

12. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The number of at least one kind of multispectral pixels or imaging pixels in each multispectral imaging unit is greater than 1.

13. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 4x4 rectangular array containing five kinds of pixels, each of the imaging pixel units contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)x(2b+4), wherein the lateral interval of adjacent multispectral imaging units is 2a pixels, and the longitudinal interval is 2b pixels, the ratio of the number of each multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)x(2b+4)).

14. The imaging chip comprising a multispectral imaging unit of claim 13, wherein, The imaging pixels are R, G, B pixels, the multispectral pixels are pixels of other kinds than R, G, B pixels, and the ratio of the G pixels to the total number of pixels of the imaging chip is 50%.

15. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 4x4 rectangular array comprising seven kinds of pixels, each of the imaging pixel units comprises one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)x(2b+4), wherein the lateral interval of adjacent multispectral imaging units is 2a pixels, the longitudinal interval is 2b pixels, and the ratio of the number of pixels of each multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)x(2b+4)).

16. The imaging chip comprising a multispectral imaging unit of claim 15, wherein, The imaging pixels are R, G, and B pixels, the multispectral pixels are other kinds of pixels other than R, G, and B pixels, and the ratio of the G pixels to the total number of pixels of the imaging chip is 50%.

17. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 3x3 rectangular array comprising five kinds of pixels, each of the imaging pixel units comprises two multispectral imaging units, and the size of the imaging pixel unit is 8x8, wherein the lateral interval and the longitudinal interval of adjacent multispectral imaging units are both 1 pixel, and the ratio of the number of pixels of each multispectral pixel to the total number of pixels of the imaging chip is 3.125%.

18. The imaging chip comprising a multispectral imaging unit of claim 17, wherein, The imaging pixels are R, G, and B filters, and the multispectral pixels are other kinds of filters other than R, G, and B filters.

19. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 3x3 rectangular array comprising five kinds of pixels, each of the imaging pixel units comprises at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)x(2b+4), wherein the lateral interval of adjacent multispectral imaging units is 2a+1 pixels and the longitudinal interval is 2b+1 pixels, and the ratio of the number of pixels of each multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)x(2b+4)).

20. The imaging chip comprising a multispectral imaging unit of claim 19, wherein, The imaging pixels are R, G, and B pixels, the multispectral pixels are other kinds of pixels other than R, G, and B pixels, and the ratio of the G pixels to the total number of pixels of the imaging chip is 50%.

21. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 3x3 rectangular array comprising five kinds of pixels, each of the imaging pixel units comprises two multispectral imaging units, and the size of the imaging pixel unit is (2a+4)x(2b+4), wherein the lateral interval of adjacent multispectral imaging units is 2a+1 pixels and the longitudinal interval is 2b+1 pixels, and the ratio of the number of pixels of each multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)x(2b+4)).

22. The imaging chip comprising a multispectral imaging unit of claim 21, wherein, The imaging pixels are R, G, and B pixels, the multispectral pixels are other kinds of pixels other than R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels of the imaging chip is 25%.

23. The imaging chip comprising a multispectral imaging unit of claim 1, wherein, The multispectral imaging unit is a 3x3 rectangular array comprising five kinds of pixels, each of the imaging pixel units comprises at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)x(2b+4), wherein the lateral interval of adjacent multispectral imaging units is 2a+1 pixels and the longitudinal interval is 2b+1 pixels, and the ratio of the number of pixels of each multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)x(2b+4)).

24. The imaging chip comprising a multispectral imaging unit of claim 23, wherein, The imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other kinds than R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels of the imaging chip is 25%.

Images