Pixel, image sensor, imaging system, calibration method, device and medium

Abstract

The present application relates to a pixel, an image sensor, an imaging system, a calibration method, a device and a medium, the pixel comprising: a photoelectric conversion module comprising n photoelectric converters arranged adjacently in an array form, n being a positive integer greater than or equal to 3; the asymmetric optical module is arranged on the photoelectric conversion module, the asymmetric optical module is used for performing focusing diffraction on incident light to generate a three-dimensional asymmetric light intensity pattern, and then n different space signals are correspondingly recorded and formed on the n photoelectric converters in the photoelectric conversion module in a one-to-one mode. According to the method, calibration is carried out based on different space signals associated with a plurality of photoelectric converters and a plurality of calibration color bands in a visible spectral range, incident light is analyzed into spectral components corresponding to the plurality of calibration color bands, brightness detection and color detection of the incident light are realized, a micro lens and an optical filter in a pixel can be effectively replaced, and the resolution ratio of the pixel is improved. The sensitivity of the image sensor is improved.

Description

Technical Field

[0001] The present application relates to the field of optoelectronic technologies, and particularly to a pixel, an image sensor, an imaging system, a calibration method, a device, and a medium. Background Art

[0002] In an image sensor, in order to generate a color image, each pixel is equipped with one of a red filter, a green filter, or a blue filter to transmit a corresponding color band and block the remaining light within the visible light spectrum.

[0003] However, in the image sensor of the related art, since each filter only transmits a narrow light band of the spectrum and the filter will absorb part of the light, the light signal is weakened, and the sensitivity of the image sensor is reduced. Summary of the Invention

[0004] Embodiments of the present application provide a pixel, an image sensor, an imaging system, a calibration method, a device, and a medium to solve the above problems.

[0005] To achieve the above object, according to the first aspect of the present application, there is provided a pixel, which includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module, disposed on the photoelectric conversion module, where the asymmetric optical module is configured to perform focusing diffraction on incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module one by one; Wherein, the pixel is configured to: based on decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through pre-calibration parameters obtained by pre-calibrating and calibrating the n calibration color bands and the n spatial signals, so as to obtain spectral components of the incident light under each of the n calibration color bands.

[0006] Optionally, the photoelectric conversion module only includes photoelectric converters of one size, and the n photoelectric converters in the photoelectric conversion module are arranged in a regular array.

[0007] Optionally, the photoelectric conversion module includes at least two different sizes of photoelectric converters, and the n photoelectric converters in the photoelectric conversion module are arranged in an irregular array.

[0008] Optionally, the asymmetric optical module includes: A background structure layer, which is composed of a first material having a first refractive index; A diffractive structure layer embedded in the background structure layer, the diffractive structure layer including a plurality of components made of a second material having a second refractive index for focusing and diffracting the incident light, the first refractive index being lower than the second refractive index.

[0009] Optionally, the first material and the second material are inorganic materials.

[0010] Optionally, the components include at least two different sizes of diffractive cylinders and at least two different sizes of diffractive toroidal cylinders, and the plurality of diffractive cylinders and the plurality of diffractive toroidal cylinders are staggered and asymmetrically arranged, so that the diffraction of the incident light by the diffractive structure layer is asymmetric diffraction.

[0011] According to a second aspect of the present application, an embodiment of the present application further provides an image sensor, the image sensor including a plurality of pixels arranged in an array; The pixel includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module, disposed on the photoelectric conversion module, where the asymmetric optical module is configured to focus and diffract the incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module one by one; Wherein, the pixel is configured to: based on decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through pre-calibration parameters obtained by pre-calibrating the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each calibration color band.

[0012] Optionally, the image sensor only includes pixels of one structure, and the pixels in the image sensor are arranged in a regular array.

[0013] Optionally, the image sensor includes at least two different structures of pixels, and the pixels in the image sensor are arranged in a regular array.

[0014] Optionally, in the image sensor, n has only one value, however, the array arrangement structures of the n photoelectric converters in at least two of the photoelectric conversion modules are different.

[0015] Optionally, in the image sensor, n has at least two different values, so that the array arrangement structures of the n photoelectric converters in at least two of the photoelectric conversion modules are different.

[0016] Optionally, in the image sensor, n has only one value, and the array arrangement structures of the n photoelectric converters in each of the photoelectric conversion modules are the same. However, the asymmetric optical modules in at least two of the pixels are different.

[0017] Optionally, the pixel further includes a back-illuminated silicon substrate. The photoelectric conversion module is disposed on the back-illuminated silicon substrate, and trenches are provided around the photoelectric conversion module on the back-illuminated silicon substrate.

[0018] According to a third aspect of the present application, an embodiment of the present application further provides an imaging system, which includes: An imaging lens for converging and imaging the light of an object to form incident light; An image sensor for focusing and diffracting the incident light based on the pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, and then correspondingly recording and forming n different spatial signals on the n photoelectric converters in the pixels; where n is a positive integer greater than or equal to 3; Wherein, the pixel is configured to: based on decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through the pre-calibration parameters obtained by pre-calibrating and calibrating the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each of the calibration color bands.

[0019] According to a fourth aspect of the present application, an embodiment of the present application further provides a calibration method for an image sensor, which is applied to an image sensor composed of multiple pixels. The method includes: For each pixel, select n calibration color bands to decompose the incident light; For each pixel, perform pre-calibration and calibration based on the n calibration color bands to obtain pre-calibration parameters; For each pixel, obtain n different spatial signals obtained by the pixel performing photoelectric acquisition on the incident light; For each pixel, perform analytical calibration on the incident light through the pre-calibration parameters and the n spatial signals, and calculate the spectral components of the incident light under each of the calibration color bands; Wherein, the pixel includes: A photoelectric conversion module including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module is disposed on the photoelectric conversion module. Wherein, the asymmetric optical module is used to focus and diffract the incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module one by one.

[0020] Optionally, the selected n calibration color bands at least include a red spectral color band, a green spectral color band, and a blue spectral color band to meet the unfolding and decomposition requirements of the incident light of unknown color.

[0021] Optionally, for each pixel, pre-calibration and calibration are performed based on the n calibration color bands to obtain pre-calibration and calibration parameters, including: For each pixel, based on the spatial signal responses of the incident light decomposed by the n calibration color bands on the n photoelectric converters in the pixel, an n - element linear equation system is constructed. Wherein, the n - element linear equation system is composed of a calibration coefficient matrix, a color band fraction vector, and a spatial signal vector. For each pixel, traverse the n calibration color bands, and based on the spatial signal response of a single calibration color band, simplify and calculate the n - element linear equation system to obtain the calibration coefficient vector of a single calibration color band. Then, synthesize the calibration coefficient vectors of the n calibration color bands to obtain the calibration coefficient matrix. For each pixel, perform an inverse operation on the calibration coefficient matrix to obtain the inverse matrix of the calibration coefficient matrix, and the inverse matrix of the calibration coefficient matrix is the pre - calibration and calibration parameter.

[0022] Optionally, the matrix form of the n - element linear equation system is as follows: , Wherein, C is the n×n calibration coefficient matrix, which is composed of the calibration coefficient vectors corresponding to the n calibration color bands; X is the color band fraction vector, which is the color band fraction vector of the incident light that contributes to the measured spatial signal; M is the spatial signal vector, which is composed of the n spatial signals measured by the incident light.

[0023] Optionally, for each pixel, the incident light is analytically calibrated through the pre - calibration and calibration parameters and the n spatial signals, and the spectral components of the incident light under each calibration color band are calculated, including: For each pixel, based on the n - element linear equation system, derivation and solution are performed to obtain an analytical calibration formula: , in, C -1 is the inverse matrix of the calibration coefficient matrix; For each of the pixels, the spatial signal vector is constructed based on the n spatial signals, and the inverse matrix of the calibration coefficient matrix and the spatial signal vector are substituted into the analytical calibration formula to calculate the color band score vector to obtain the spectral component of the incident light in each of the calibration color bands.

[0024] According to a fifth aspect of the present application, an embodiment of the present application further provides an optoelectronic device, including: a memory having a computer program stored thereon; A processor is used to execute the computer program in the memory to implement the steps of any one of the methods provided in the embodiments of the present application.

[0025] According to the sixth aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the methods provided in the embodiments of the present application are implemented.

[0026] Some embodiments of the present application have at least the following beneficial effects: calibration is performed based on the associated different spatial signals on multiple photoelectric converters and multiple calibration color bands in the visible spectrum range, and the incident light is parsed into spectral components corresponding to the multiple calibration color bands to achieve brightness detection and color detection of the incident light. It can combine asymmetric optical modules with multi-color band calibration technology to replace the microlenses and filters of pixels in related technologies, and can effectively avoid problems such as reduced white light broadband sensitivity and reduced spectral resolution caused by the microlenses and filters in the pixels.

[0027] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0029] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0030] Figure 1 This is a schematic diagram of the structure of a pixel in a traditional image sensor; Figure 2It is a schematic structural diagram of a traditional RGGB image sensor; Figure 3 It is a schematic diagram comparing the quantum efficiency (QE) curves of a monochromatic image sensor and a traditional RGGB image sensor in the visible spectral range; Figure 4 It is a schematic structural diagram of an imaging system in the related art; Figure 5 It is a curve distribution diagram of the chief ray angle of the imaging lens of the imaging system in the related art; Figure 6 It is a schematic process flow diagram of a traditional RGGB image sensor; Figure 7 It is a schematic structural diagram of a pixel in an optional embodiment of the present application; Figure 8 It is a three-dimensional view of a pixel in an optional embodiment of the present application; Figure 9 It is a schematic diagram of asymmetric diffraction of different incident lights by an asymmetric optical module in an optional embodiment of the present application; Figure 10 It is a schematic diagram of 4 selected calibration color bands in an optional embodiment of the present application; Figure 11 It is a schematic diagram of the calibration and calibration process of a pixel in an optional embodiment of the present application; Figure 12 It is a schematic structural diagram of a pixel in an optional embodiment of the present application; Figure 13 It is a schematic structural diagram of another pixel in an optional embodiment of the present application; Figure 14 It is a schematic structural diagram of another pixel in an optional embodiment of the present application; Figure 15 It is a schematic structural diagram of an imaging system in an optional embodiment of the present application; Figure 16 It is a curve distribution diagram of the chief ray angle of the imaging lens of the imaging system in an optional embodiment of the present application; Figure 17 It is a schematic structural diagram of an optoelectronic device in an optional embodiment of the present application. Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0032] To facilitate the understanding of the implementation solutions provided by the embodiments of the present application, first, the relevant application backgrounds of technical solutions such as pixels, image sensors, imaging systems, calibration methods of image sensors, optoelectronic devices, and computer-readable storage media provided by the embodiments of the present application will be described.

[0033] As Figure 1 shown, in a traditional image sensor, each pixel includes at least a microlens, a color filter, and a photoelectric converter (such as a photodiode). The microlens is used to converge light, ensuring that the light can effectively pass through the color filter and then irradiate onto the photoelectric converter, thereby improving the light collection efficiency of the photoelectric converter. The photoelectric converter generates charges or currents according to the incident light. Among them, in order to generate a color image, each pixel is equipped with one of a red color filter, a green color filter, or a blue color filter. The color filter transmits the corresponding color band and blocks the remaining light within the visible light spectrum. For example, the color filter can be arranged repeatedly in the image sensor in the form of an RGGB pattern (also known as a Bayer pattern) as Figure 2 shown.

[0034] Since each color filter only transmits a narrow spectral band while the remaining light is blocked. Therefore, at any given position of the photoelectric converter or pixel, only one color band signal transmitted by the color filter can be recorded. In an alternative embodiment of the present application, a comparison test is conducted between a monochromatic image sensor without any color filters and a traditional RGGB image sensor, and the quantum efficiency (QE) curves of the two within the visible light spectrum are as Figure 3 shown. As Figure 3 can be seen, compared with the monochromatic image sensor without any color filters, the spectral filtering of the traditional RGGB image sensor reduces the white light broadband sensitivity by about 3 - 4 times.

[0035] At the same time, in order to infer the block color signal at any given pixel position in the RGGB image sensor, it is necessary to interpolate (also known as demosaicing) the signals of adjacent color pixels to obtain the block color signal at the given position, which reduces the spectral resolution of the color image sensor.

[0036] As Figures 4 - 5 shown, in many existing application scenarios, the imaging system is equipped with a multi-element imaging lens. The imaging lens sends the incident light for imaging to the image sensor at a very high angle (the chief ray angle up to 35 degrees). In the image sensor of the related technology, the pixels have a pixel stack height of 2 - 4 μm between the top surface of the microlens and the photosensitive substrate interface. The light incident on the microlens at a high angle is focused by the microlens and undergoes a large displacement during the propagation through the pixel stack. Therefore, in each pixel, the microlens and the color filter need to be displaced relative to the photoelectric converter to collect the light into the appropriate photoelectric converter.

[0037] Specifically, from Figure 4 and Figure 5 it can be seen that the displacements of the microlens and the filter relative to the optoelectronic converter depend on many factors, such as the pixel stack height, the number of pixel stack layers, the optical properties of the materials, the pixel size, the chief ray angular distribution curve of the imaging lens of the image sensor, and the focal length f of the imaging lens. Therefore, the offset design for the microlens and the filter in a specific imaging lens and image sensor is usually a very long and costly process. Any modification to the imaging lens design, pixel size, or pixel stack height also incurs additional costs because the manufacturing process has changed substantially and the extremely high stability of the new process for producing high-yield parts needs to be verified.

[0038] Therefore, by reducing the pixel stack and ideally avoiding the need for any part of the pixel stack to shift, it would be very beneficial for significantly reducing the cost of the image sensor. At the imaging system assembly stage, the multi-element imaging lens needs to be three-dimensionally aligned with the image sensor array to achieve the lateral coincidence of the optical centers of the imaging lens and the image sensor. Additionally, it is also crucial to prevent the imaging lens from tilting relative to the sensor plane so that the image sensor with offset microlens and filter can sense the optical signal at the correct incident angle. Therefore, it is desirable to provide an image sensor that can tolerate the misalignment problem between the imaging lens and the image sensor to reduce the process complexity.

[0039] In addition, the deposition process of the microlens and the color filter based on non-CMOS organic polymer materials requires additional non-CMOS production equipment for setting the absorptive filter on the image sensor array. In an alternative embodiment of the present application, the number of process steps, production cycle, and production yield of the existing RGGB image sensor are as Figure 6 shown.

[0040] Exemplarily, as Figure 6 shown, the addition of the non-CMOS process lengthens the production process of the image sensor, and the yield of the non-CMOS production equipment is relatively low. Therefore, the addition of the non-CMOS process will increase the cost of the image sensor. At the same time, the traditional polymer-based absorptive filter is also affected by weathering and has a shorter lifespan than the inorganic materials using CMOS production equipment. Therefore, it is desirable to integrate the color imaging function of the image sensor into the CMOS manufacturing process to reduce the cost of the image sensor and improve the yield of the image sensor.

[0041] In view of this, an embodiment of the present application provides an image sensor based on a pure CMOS process: designed in units of pixels, within each pixel, an asymmetric optical module shared by n photoelectric converters is designed based on at least two inorganic materials with different refractive indexes, focusing and diffracting incident light to generate a three-dimensional asymmetric light intensity pattern, and then different spatial signals are recorded and formed on the n photoelectric converters based on the three-dimensional asymmetric light intensity pattern; the three-dimensional asymmetric light intensity pattern is formed by light diffraction occurring on the asymmetric optical module and subsequent self-interference, and the interference property of light makes the detected spatial signals highly correlated, so that narrow spectral bands of various colors (hereinafter referred to as color bands) form n different signal component distributions corresponding one by one on the n photoelectric converters. In this way, n representative calibration color bands can be selected within the visible spectral range, the incident light of unknown color can be expanded into the sum of the components of the n calibration color bands, and then based on the spatial signal responses of the incident light expanded and decomposed into the n calibration color bands on the n photoelectric converters, a linear equation system of n variables is constructed. Among them, the linear equation system of n variables consists of a calibration coefficient matrix, a color band fraction vector, and a spatial signal vector, and the calibration coefficient matrix can be calibrated by the independent spatial signals of each calibration color band.

[0042] Therefore, based on the linear equation system of n variables, the color band components of the incident light can be solved based on the pre-calibrated calibration coefficient matrix and the n spatial signals measured in real time, and the incident light can be analyzed into spectral components corresponding to the n calibration color bands, realizing the color measurement of the incident light while realizing the brightness detection of the incident light. By combining the asymmetric optical module with the multi-color band calibration technical means to replace the microlenses and filters in the existing pixels, it helps to reduce the production cost and process complexity of the image sensor and improve the sensitivity and spectral resolution of the detection of incident light.

[0043] Among them, n is a positive integer greater than or equal to 3; a variety of different representative calibration color bands are selected within the entire visible spectral range. For example, representative calibration color bands can be selected within the narrow spectral bands corresponding to different colors. Exemplarily, at least three calibration color bands of red, green, and blue are covered, that is, the selected calibration color bands at least include a red spectral color band, a green spectral color band, and a blue spectral color band to meet the expansion and decomposition requirements of the incident light of unknown color.

[0044] First, an embodiment of the present application provides a pixel, which includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module is disposed on a photoelectric conversion module. The asymmetric optical module is configured to focus and diffract incident light to generate a three-dimensional asymmetric light intensity pattern, and then record one-to-one on n photoelectric converters in the photoelectric conversion module to form n different spatial signals. The pixel is configured to: based on decomposing the incident light by n calibration color bands, resolve and calibrate the incident light through pre-calibration parameters obtained from pre-calibration of the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each calibration color band.

[0045] It should be noted that the three-dimensional asymmetric light intensity pattern means that after the incident light undergoes asymmetric focusing diffraction by the asymmetric optical module, a three-dimensional asymmetric light intensity pattern is formed in the three-dimensional space of the photoelectric converter, and this three-dimensional asymmetric light intensity pattern is then projected and mapped onto the photosensitive plane of the photoelectric converter, forming a two-dimensional asymmetric light intensity distribution on each photoelectric converter at different spatial positions in the photosensitive plane. The light intensity distributions collected by the photoelectric converters at different positions are different, and then based on the photoelectric effect, n different spatial signals are recorded one-to-one on the n photoelectric converters.

[0046] Exemplarily, as Figures 7 - 8 shown, the value of n is 4, and the asymmetric optical module is disposed on a photoelectric conversion module composed of 4 photoelectric converters. For example, the asymmetric optical module can be disposed on the corresponding 2×2 photoelectric converters. The asymmetric optical module focuses and diffracts the incident light to generate a three-dimensional asymmetric light intensity pattern, so as to record and form 4 different spatial signals on the 2×2 photoelectric converters in the photoelectric conversion module.

[0047] In some embodiments, the asymmetric optical module includes: A background structure layer, which is composed of a first material having a first refractive index; A diffraction structure layer embedded in the background structure layer. The diffraction structure layer includes a plurality of components composed of a second material having a second refractive index for focusing and diffracting incident light, and the first refractive index is lower than the second refractive index.

[0048] It can be understood that the geometric shape, size, material properties, and arrangement of the components can be flexibly designed according to actual needs to achieve asymmetric focusing and diffraction of incident light. Among them, the incident light can be the emitted light directly emitted by the light source, or the reflected light after the light source emits light and irradiates an object, which will not be elaborated here.

[0049] In some embodiments, the components can include cylindrical, rectangular, V-shaped, annular, etc.

[0050] In some embodiments, the components may be arranged in a periodic, quasi-periodic or random manner.

[0051] In some embodiments, the components include at least two different-sized diffractive cylinders and at least two different-sized diffractive ring cylinders, and the multiple diffractive cylinders and the multiple diffractive ring cylinders are staggered and asymmetrically arranged so that the diffraction of the incident light by the diffractive structure layer is asymmetric diffraction.

[0052] Exemplarily, as Figures 7 - 8 shown, the asymmetric optical module includes: A background structure layer having a first refractive index; An asymmetric diffractive structure layer having a second refractive index, the asymmetric diffractive structure layer being embedded in the background structure layer, and the second refractive index being different from the first refractive index.

[0053] Exemplarily, as Figures 7 - 8 shown, the asymmetric diffractive structure layer is a transparent structure layer and is embedded at the bottom of the background structure layer. The asymmetric diffractive structure layer includes multiple diffractive cylinders and multiple diffractive ring cylinders, and the multiple diffractive cylinders of different sizes and the multiple diffractive ring cylinders of different sizes are staggered so that the diffraction of the incident light by the asymmetric diffractive structure layer is asymmetric diffraction.

[0054] Exemplarily, as Figures 7 - 8 shown, the diameters of the multiple diffractive cylinders are different from each other, the inner diameters and outer diameters of the multiple diffractive ring cylinders are different from each other, and the heights of the diffractive cylinders and the diffractive ring cylinders are both less than the height of the background structure layer.

[0055] In some embodiments, the first material and the second material are inorganic materials.

[0056] It should be noted that the asymmetric optical module is completely made of inorganic materials. Compared with traditional image sensors using organic materials, due to the use of inorganic materials, it does not involve CMOS process flows such as organic materials, and the entire image sensor can be completely manufactured in a CMOS foundry, reducing the cost of the image sensor. At the same time, due to the process precision and harsh requirements of organic materials, the yield of traditional image sensors with microlenses and filters is relatively low, about 80%, while the yield of the all-inorganic material CMOS image sensor in the embodiments of the present application is very high, close to 100%, and the compatibility between inorganic materials is better. By eliminating non-CMOS materials, the possibility of dark current can be further reduced. Therefore, the all-CMOS image sensor provides higher lifespan and higher reliability.

[0057] Among them, the background structure layer can be composed of low-refractive-index materials, such as silicon oxide with a refractive index of 1.46, or air with a refractive index of 1; the asymmetric diffraction structure layer is composed of high-refractive-index materials, such as silicon nitride, titanium oxide, tantalum oxide, silicon carbide, and silicon, etc., and its refractive index can be in the range of 1.8 to 4.

[0058] It can be understood that in other alternative embodiments, the asymmetric optical module can be composed of three or more inorganic materials, which are not limited herein. For example, it can be a first background layer with a lower refractive index, a second background layer with a medium refractive index, and an asymmetric diffraction structure layer with a higher refractive index that are nested layer by layer from the inside out, or a first background layer with a lower refractive index, a first asymmetric diffraction structure layer with a medium refractive index embedded in the first background layer, and a second asymmetric diffraction structure layer with a higher refractive index embedded in the first background layer, etc.

[0059] In some embodiments, the photoelectric conversion module includes only one size of photoelectric converter, and the n photoelectric converters in the photoelectric conversion module are arranged in a regular array.

[0060] Exemplarily, as Figures 7 - 8 shown, Figure 7 for the pixels enclosed by the dashed line in, the sizes of the 4 photoelectric converters in the corresponding photoelectric conversion module are the same, all being square photoelectric converters, the corresponding photosensitive surface is designed as a square, and the 4 photoelectric converters are arranged in a 2×2 regular array.

[0061] Exemplarily, as Figure 12 shown, Figure 12 for the pixels enclosed by the dashed line in, the sizes of the 3 photoelectric converters in the corresponding photoelectric conversion module are the same, all being rectangular photoelectric converters, the corresponding photosensitive surface is designed as a rectangle, and the 3 photoelectric converters are arranged in a 1×3 regular array.

[0062] In some embodiments, the regular array arrangement can be a linear arrangement, a two-dimensional matrix arrangement, etc.

[0063] Exemplarily, as Figures 7 - 8 shown, the 4 photoelectric converters with the same size in the corresponding photoelectric conversion module are arranged in a 2×2 two-dimensional matrix.

[0064] Exemplarily, as Figure 12 shown, the 3 photoelectric converters with the same size in the corresponding photoelectric conversion module are arranged in a 1×3 matrix, which can also be said to be a horizontal linear arrangement.

[0065] In some embodiments, the photoelectric conversion module includes at least two different sizes of photoelectric converters, and the n photoelectric converters in the photoelectric conversion module are arranged in an irregular array.

[0066] Exemplarily, as Figures 13 - 14 shown, the photoelectric conversion module in at least some pixels includes two different sizes of photoelectric converters. For example, in the pixels enclosed by the dotted line in the upper left corner of Figure 13 and in the pixels enclosed by the dotted line in the upper left corner of Figure 14 , the photoelectric conversion module corresponding to the pixel includes 1 first rectangular photoelectric converter and 2 second rectangular photoelectric converters, and the 3 photoelectric converters are arranged in an irregular array; and the photoelectric conversion module in at least some pixels only includes one size of photoelectric converter. In the pixels enclosed by the dotted line in the lower right corner of Figure 14 , the 4 photoelectric converters in the photoelectric conversion module corresponding to the pixel have the same size, all of which are square photoelectric converters, the corresponding photosensitive surfaces are designed as squares, and the 4 photoelectric converters are arranged in a 2×2 regular array.

[0067] In some embodiments, the specific sizes or irregular arrangements of the photoelectric converters within the pixel can be customized according to specific requirements. For example, larger-sized photoelectric converters can be arranged in certain areas to improve local sensitivity, or smaller-sized photoelectric converters can be used in other areas to achieve higher spatial resolution.

[0068] Secondly, the embodiment of the present application also provides an image sensor, and the image sensor includes a plurality of pixels arranged in an array; The pixel includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module, disposed on the photoelectric conversion module, where the asymmetric optical module is used to focus and diffract incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module one by one; Among them, the pixel is configured to: based on decomposing the incident light by n calibration color bands, analyze and calibrate the incident light through the pre-calibration parameters obtained by pre-calibrating and calibrating the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each calibration color band.

[0069] Exemplarily, as Figures 7 - 8 shown, the image sensor includes: A photoelectric conversion array, and the photoelectric conversion array includes a plurality of photoelectric conversion modules arranged in an array; Asymmetric optical array. The asymmetric optical array includes a plurality of asymmetric optical modules arranged in an array on a photoelectric conversion array. Each asymmetric optical module is disposed on a corresponding photoelectric conversion module. The value of n is 4. Each photoelectric conversion module includes 4 photoelectric converters arranged in a 2×2 regular array. The asymmetric optical module focuses and diffracts incident light to generate a three-dimensional asymmetric light intensity pattern, so as to record and form 4 different spatial signals on the 2×2 photoelectric converters within the corresponding photoelectric conversion module.

[0070] Exemplarily, as Figures 7 - 8 shown, 2×2 photoelectric converters and an asymmetric optical module thereon constitute a pixel (which can also be called a pixel unit). The image sensor includes a plurality of pixels arranged in an array. Incident light forms a three-dimensional asymmetric light intensity pattern within each pixel based on the asymmetric optical module, and then 4 different spatial signals are correspondingly formed on the 4 photoelectric converters one by one. Combining with 4 monochromatic calibration color bands within the subsequent visible light spectrum range, it can effectively perform brightness detection and color detection on the incident light within each pixel, and resolve the incident light into spectral components corresponding to the 4 calibration color bands.

[0071] It should be noted that, compared with traditional RGGB sensors, in the above image sensor, instead of filtering out multiple color components through color filters and then transmitting the remaining single color band component to the photoelectric converter, full-spectrum color band component transmission is performed based on the asymmetric optical module, and its photosensitivity within the visible light spectrum range is increased by 2 to 3 times. At the same time, the brightness detection and color detection of incident light are completely based on the pre-calibration and measured spatial signals of multi-color band components, without the need for interpolation calculation based on adjacent pixels, which helps to improve the spectral resolution of the image sensor.

[0072] It can be understood that since the pixels in the embodiments of the present application can combine the asymmetric optical module and the multi-color band calibration technology to replace the microlens and filter in the existing pixels, the pixels in the embodiments of the present application are not limited by the structural dimensions of the microlens and filter, especially not limited by the structural dimensions of the microlens, and the corresponding pixel size can be adaptively reduced.

[0073] It should be noted that, in addition to the above photoelectric conversion array and the above asymmetric optical array, the image sensor also includes other structures such as a pixel processing circuit, a controller, and an image processor. For details, reference can be made to the prior art and will not be elaborated here.

[0074] In some embodiments, the image sensor only includes pixels of one structure, and the pixels in the image sensor are arranged in a regular array.

[0075] Exemplarily, as Figure 7 、 Figure 12 or Figure 13As shown, the image sensor includes a plurality of pixels, and the structure of each pixel is the same, that is, the structure of the photoelectric conversion module in each pixel is the same, and the structure of the asymmetric optical module in each pixel is also the same. Moreover, the pixels in the image sensor are arranged in a regular two-dimensional matrix.

[0076] In some embodiments, the image sensor includes at least two different structures of pixels, and the pixels in the image sensor are arranged in a regular array.

[0077] Exemplarily, the image sensor includes a plurality of pixels, and the structures of at least two pixels are different. The structures of the photoelectric conversion modules in these two pixels are different, and / or the structures of the asymmetric optical modules are different.

[0078] In some embodiments, in the image sensor, n has only one value. However, the array arrangement structures of the n photoelectric converters in at least two photoelectric conversion modules are different.

[0079] Exemplarily, the image sensor includes a plurality of pixels. The number of photoelectric converters included in each pixel is the same, and the structure of the asymmetric optical module in each pixel is the same. However, the structures of the photoelectric conversion modules in some pixels are different, that is, the arrangement structures of the multiple photoelectric converters in some pixels are different. For example, the value of n in each pixel in the image sensor is 3. The structure of the photoelectric conversion module in some pixels is as Figure 12 shown, which is formed by arranging 3 rectangular photoelectric converters of the same size in a 1×3 array. The structure of the photoelectric conversion module in some pixels is as Figure 13 shown, which is formed by arranging 3 rectangular photoelectric converters of two different sizes in an irregular array.

[0080] In some embodiments, in the image sensor, n has at least two different values, so that the array arrangement structures of the n photoelectric converters in at least two photoelectric conversion modules are different.

[0081] Exemplarily, the image sensor includes a plurality of pixels. The number of photoelectric converters included in at least two pixels is different, and the array arrangement structures of the multiple photoelectric converters in these two pixels are different. As Figure 14 shown, the value of n in some pixels is 3, and the value of n in some pixels is 4. Even if the structure of the asymmetric optical module in each pixel is the same, on the premise that the sizes of the photoelectric conversion modules are the same, since the number of photoelectric converters included in some pixels is different, this makes the arrangement structures of the respective photoelectric converters in the photoelectric conversion modules of the corresponding pixels necessarily different.

[0082] In some embodiments, in an image sensor, n has only one value, and the array arrangement structures of the n photoelectric converters in each photoelectric conversion module are the same. However, the asymmetric optical modules in at least two pixels are different.

[0083] Exemplarily, the image sensor includes a plurality of pixels. The number n of photoelectric converters included in each pixel is the same, and the array arrangement structures of the n photoelectric converters in the photoelectric conversion module corresponding to each pixel are the same. However, the asymmetric optical modules in at least two pixels are different. For example, the asymmetric optical modules in some of the pixels shown in Figure 13 are rotated 90° to the right in the vertical projection plane of the incident light. The focusing diffraction orientation of the rotated asymmetric optical module is different from that of the unrotated asymmetric optical module. In this way, two different asymmetric optical modules are obtained. Or directly change the structure of the asymmetric optical modules in some of the pixels shown in Figure 13 based on the transformation of the internal component shapes and materials.

[0084] In some embodiments, the value of the number n of photoelectric converters in a pixel is 4. A pixel is formed based on a photoelectric conversion module composed of 2×2 photoelectric converters and an asymmetric optical module, and a total of 4 calibration color bands, namely R, G1, G2, and B, are uniformly selected in the visible spectral range. Pre-calibration and calibration are performed based on the spatial signal responses corresponding to the monochromatic calibration color bands. Subsequently, incident light of an unknown color can be resolved into spectral components corresponding to the 4 calibration color bands. This is applicable to an image sensor in which a single photoelectric converter is arranged in a square shape and multiple photoelectric converters are arranged in a regular array. For example, Figure 7 the 4×6 photoelectric conversion array shown in includes 24 photoelectric converters arranged in a 4×6 array, which can be configured as 2×3 pixels, and an asymmetric optical module is formed corresponding to 2×2 photoelectric converters.

[0085] In addition, as shown in Figure 7 the 4×6 photoelectric conversion array can also be configured as 1×2 pixels, and an asymmetric optical module is formed corresponding to 3×3 photoelectric converters to form a pixel; as the photoelectric conversion array expands, an asymmetric optical module can also be formed corresponding to 4×4, 5×5,..., N×N photoelectric converters to form a pixel.

[0086] Correspondingly, 3×3, 4×4, 5×5,..., N×N calibration color bands need to be selected in the visible spectral range, and the calibration color bands cover at least the three primary colors of red, green, and blue to effectively meet the requirements for color detection of incident light of various unknown colors. Here, N is an integer greater than or equal to 2.

[0087] It can be understood that the more uniform and the larger the number of the color distribution of the calibration color ribbon, the more detailed the color decomposition and analysis of the incident light, the higher the accuracy of the decomposition and analysis, and the more accurate the result of the decomposition and analysis. However, the corresponding calculation amount is relatively large, and it can be selected according to the actual situation.

[0088] Of course, as Figure 7 shown, the 4×6 photoelectric conversion array can also be configured into 2×2 pixels, and a 2×3 photoelectric converter in one pixel corresponds to form an asymmetric optical module to form one pixel. Among them, the shape and structure of the asymmetric optical module need to be adjusted correspondingly, and the diffraction focusing effect is poor in some directions or on a certain photoelectric converter.

[0089] Therefore, in another embodiment of the present application, as Figure 12 shown, for an image sensor in which a plurality of photoelectric converters are arranged in an array regularly but each photoelectric converter is rectangularly arranged, an asymmetric optical module can be correspondingly formed on the 1×3 photoelectric converters shown in the dotted line box. Correspondingly, 3 different calibration color ribbons need to be selected within the visible light spectrum range. For example, a red spectral color ribbon of 600 nm, a green spectral color ribbon of 525 nm, and a blue spectral color ribbon of 450 nm can be used for pre-calibration and calibration to obtain the inverse matrix C -1 of the corresponding calibration coefficient matrix C. The specific calibration and calibration process can refer to the following embodiments and will not be elaborated here.

[0090] In some embodiments, as Figure 7 or Figure 12 shown, the specification sizes of each photoelectric converter in the image sensor are all the same and a plurality of photoelectric converters are arranged in an array regularly.

[0091] In some embodiments, there may be at least two different specification sizes of photoelectric converters in the image sensor, and the corresponding plurality of photoelectric converters are arranged in a crisscross pattern, as Figure 13 and Figure 14 shown.

[0092] In another embodiment of the present application, as Figure 13 shown, for a plurality of photoelectric converters arranged in a crisscross pattern, an asymmetric optical module can be correspondingly formed on the 3 photoelectric converters shown in the dotted line box to form one pixel. Correspondingly, 3 different calibration color ribbons need to be selected within the visible light spectrum range. For example, a red spectral color ribbon of 600 nm, a green spectral color ribbon of 525 nm, and a blue spectral color ribbon of 450 nm can be used for pre-calibration and calibration to obtain the inverse matrix C -1 of the corresponding calibration coefficient matrix C. The specific calibration and calibration process can refer to the following embodiments and will not be elaborated here.

[0093] In another embodiment of the present application, as Figure 14As shown, for multiple photoelectric converters arranged in a partially crisscross pattern and partially in an array pattern, a first asymmetric optical module can be correspondingly formed on the 3 photoelectric converters shown in the first dashed box, and a second asymmetric optical module can be correspondingly formed on the 2×2 photoelectric converters shown in the second dashed box. Among them, the shape and structure of the first asymmetric optical module can be the same as or different from those of the second asymmetric optical module. Thus, a first pixel is formed based on the 3 photoelectric converters and the shared first asymmetric optical module thereon, and a second pixel is formed based on the 2×2 photoelectric converters and the shared second asymmetric optical module thereon.

[0094] Among them, the first pixel needs to select 3 different calibration color bands within the visible light spectrum range, and the second pixel needs to select 4 different calibration color bands within the visible light spectrum range. For example, the first pixel can be pre-calibrated using the red spectral color band at 600 nm, the green spectral color band at 525 nm, and the blue spectral color band at 450 nm, and the second pixel can be pre-calibrated using 4 calibration color bands of R, G1, G2, and B; or, 4 calibration color bands of R, G1, G2, and B can be used to pre-calibrate the first pixel and the second pixel respectively, and the first pixel can use 3 calibration color bands of R, G1, B or 3 calibration color bands of R, G2, B during pre-calibration. The specific calibration process can refer to the following embodiments and will not be elaborated here.

[0095] In some embodiments, the pixel further includes a back-illuminated silicon substrate, the photoelectric converter is disposed on the back-illuminated silicon substrate, and trenches are provided around the multiple photoelectric converters on the back-illuminated silicon substrate.

[0096] Exemplarily, as Figure 8 shown, the pixel further includes a back-illuminated silicon substrate (Back-Side illuminated Si, BSI Si), the photoelectric conversion module is disposed on the back-illuminated silicon substrate, the asymmetric optical module is disposed on the photoelectric conversion module, and the pixel further includes a cell deep trench isolation structure (Cell Deep Trench Isolation, CDTI) disposed in the back-illuminated silicon substrate. The cell deep trench isolation structure surrounds the 2×2 photoelectric converters in the photoelectric conversion module to physically isolate the 2×2 photoelectric converters under each asymmetric optical module and prevent light crosstalk between adjacent asymmetric optical modules.

[0097] Among them, the cell deep trench isolation structure can be made of materials such as silicon oxide, which is not limited herein.

[0098] Again, the embodiments of the present application further provide a calibration method for an image sensor, which is applied to an image sensor composed of multiple pixels. The method includes the steps: S1. For each pixel, select n calibration color bands to decompose the incident light by expansion. S2. For each pixel, perform pre-calibration based on the n calibration color bands to obtain pre-calibration parameters. S3. For each pixel, obtain n different spatial signals resulting from the optoelectronic acquisition of the incident light by the pixel. S4. For each pixel, perform analytical calibration on the incident light through the pre-calibration parameters and the n spatial signals, and calculate the spectral components of the incident light under each calibration color band. Among them, the pixel includes: An optoelectronic conversion module, including n optoelectronic converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3. An asymmetric optical module, disposed on the optoelectronic conversion module, where the asymmetric optical module is used to focus and diffract the incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n optoelectronic converters in the optoelectronic conversion module in one-to-one correspondence.

[0099] It should be noted that the spatial signal refers to the signal measured on each optoelectronic converter in the optoelectronic conversion array, which is formed by the three-dimensional asymmetric light intensity pattern generated by the asymmetric focusing and diffraction of the incident light of unknown color by the asymmetric optical module. Since the light intensity pattern is a three-dimensional asymmetric light intensity pattern, after it is projected onto the photosensitive planes of the optoelectronic converters in the pixel, there are differences in the light intensity patterns on the photosensitive planes of the optoelectronic converters, and the sizes of the spatial signals corresponding to the optoelectronic induction are different. Therefore, the spatial signal reflects the spatial distribution characteristics of the incident light.

[0100] It can be understood that the spectral components may include the contribution ratio of each calibration color band to the total incident light intensity. For example, the contribution ratio of the 600nm red spectral color band is 20%, the contribution ratio of the 525nm green spectral color band is 30%, and the contribution ratio of the 450nm blue spectral color band is 50%.

[0101] In some embodiments, the n calibration color bands selected in step S1 at least include a red spectral color band, a green spectral color band, and a blue spectral color band to meet the requirements for the decomposition of the incident light of unknown color.

[0102] Among them, the red spectral color band is used to calibrate and analyze the red component in the incident light, the green spectral color band is used to calibrate and analyze the green component in the incident light, and the blue spectral color band is used to calibrate and analyze the blue component in the incident light.

[0103] Spectral decomposition refers to the process of decomposing incident light of an unknown color into components of different wavelengths. By using a red spectral color band, a green spectral color band, and a blue spectral color band as calibration color bands, the light of an unknown color can be decomposed into three basic components: a red spectral color band, a green spectral color band, and a blue spectral color band, thereby accurately describing its spectral characteristics.

[0104] To ensure the accurate decomposition and analysis of incident light of an unknown color, the calibration color bands selected in step S1 must be able to cover the spectral range of the unknown color. The red spectral color band, the green spectral color band, and the blue spectral color band are commonly used reference color bands in spectral analysis, which can cover most of the visible light range and can effectively decompose and describe the spectral characteristics of the incident light of an unknown color.

[0105] It should be noted that in step S1, when selecting calibration color bands, the more calibration color bands of different types are selected, the finer and more accurate the spectral decomposition of the incident light by multiple calibration color bands will be. Therefore, on the basis of covering the red spectral color band, the green spectral color band, and the blue spectral color band, the selected calibration color bands can include more spectral color bands of other colors, such as a yellow spectral color band, a purple spectral color band, etc., and multiple spectral color bands of the same color system with similar colors but different wavelengths can also be selected, such as a green spectral color band with a wavelength of 490 nm and a green spectral color band with a wavelength of 550 nm. However, the corresponding calibration calculation amount is relatively large and needs to be considered in a balanced way.

[0106] In step S2, the pre-calibration and calibration parameters refer to the parameters obtained through a series of pre-calibration steps for spectral decomposition of incident light, which can be obtained by combining the calculation results of the spatial signal responses of individual color bands of each calibration color band.

[0107] In some embodiments, for each pixel, the step S2 of obtaining pre-calibration and calibration parameters by pre-calibrating based on n calibration color bands further includes: S21. For each pixel, based on the spatial signal responses of the incident light decomposed by n calibration color bands on n photoelectric converters within the pixel, construct an n - variable linear equation system, where the n - variable linear equation system consists of a calibration coefficient matrix, a color band fraction vector, and a spatial signal vector; S22. For each pixel, traverse the n calibration color bands, and based on the spatial signal response of a single calibration color band, simplify and calculate the n - variable linear equation system to obtain the calibration coefficient vector of a single calibration color band, and then combine the calibration coefficient vectors of the n calibration color bands to obtain the calibration coefficient matrix; S23. For each pixel, perform an inverse operation on the calibration coefficient matrix to obtain the inverse matrix of the calibration coefficient matrix, and the inverse matrix of the calibration coefficient matrix is the pre-calibration and calibration parameter.

[0108] In step S21, the calibration coefficient matrix represents the calibration coefficients of the spatial signals measured by each calibration color band on each photoelectric converter. Specifically, multiple elements in each row of the calibration coefficient matrix respectively represent the calibration coefficients of each calibration color band for the spatial signal on a certain photoelectric converter, and multiple elements in each column of the calibration coefficient matrix respectively represent the calibration coefficients of a certain calibration color band for the spatial signals on each photoelectric converter. That is to say, each element a in the calibration coefficient matrix ij represents the calibration coefficient of the i-th spatial signal under the j-th calibration color band.

[0109] Exemplarily, for each pixel, the calibration method of the above image sensor can be further refined to include: Step S01: Determine the number n of photoelectric converters sharing the same asymmetric optical module; Step S02: Select n calibration color bands within the visible spectral range that can effectively decompose incident light of various unknown colors; Step S03: Based on the spatial signal responses of the n calibration color bands on the n photoelectric converters sharing the same asymmetric optical module, construct a system of linear equations with n unknowns; Step S04: Based on the spatial signal response of a single calibration color band, simplify the system of linear equations with n unknowns and solve the calibration coefficient matrix corresponding to the system of linear equations with n unknowns; Step S05: Solve the inverse matrix of the calibration coefficient matrix; Step S06: For incident light of unknown color, according to the measured spatial signals on the n photoelectric converters sharing the same asymmetric optical module and the inverse matrix of the calibration coefficient matrix, decompose the incident light into spectral components corresponding to the n calibration color bands.

[0110] Among them, the detailed step flow of the calibration method of the above image sensor can refer to the following embodiments and will not be elaborated here.

[0111] Within each pixel, the spatial signal response refers to the photoelectric effect response of each photoelectric converter to the asymmetric light intensity spatial distribution formed after the incident calibration color band undergoes asymmetric focusing diffraction by the asymmetric optical module, forming a corresponding electrical signal. That is to say, multiple spatial signal responses within each pixel can describe the spatial distribution of incident light on the photoelectric conversion module.

[0112] Exemplarily, within each pixel, the spatial signal response can be obtained by irradiating a known calibration color band (such as a red spectral color band, a green spectral color band, a blue spectral color band, etc.) onto the photoelectric conversion module and recording the electrical signals output by each photoelectric converter.

[0113] It can be understood that the spatial signal response within each pixel can be represented as a vector or matrix, where each element corresponds to the electrical signal intensity on a photoelectric converter. For example, for a 2×2 photoelectric conversion array, the spatial signal response can be represented as a 4×1 vector.

[0114] Exemplarily, in the embodiments of the present application, for an Figures 7 - 8 asymmetric optical module as shown, when the value of n is 4 and narrow spectral bands with wavelengths of 430 nm, 490 nm, 550 nm, and 610 nm are selected as calibration color bands within the visible spectral range, the asymmetric optical module can effectively perform asymmetric focusing diffraction on the incident light of the above 4 calibration color bands (B = 430 nm, G2 = 490 nm, G1 = 550 nm, and R = 610 nm), as Figure 9 shown. Therefore, the asymmetric optical module can also effectively perform asymmetric diffraction on various incident lights based on the above 4 calibration color bands, forming different spatial signals on 2×2 photoelectric converters.

[0115] In this way, based on the different spatial signal responses on 4 photoelectric converters within a pixel, 4 representative calibration color bands can be selected within the visible spectral range to construct a system of four linear equations with one unknown. Then, calibration is performed based on the spatial signal response of a single calibration color band to determine the calibration coefficient matrix corresponding to the system of four linear equations with one unknown; the color band components of the incident light are solved based on the calibration coefficient matrix and the 4 spatial signals measured in real time, that is, the 2×2 spatial signals are resolved into 2×2 spectral components of the incident light using 2×2 calibration color bands.

[0116] In some embodiments, for the general case where the photoelectric conversion module includes n photoelectric converters, the matrix form of the system of n linear equations with one unknown formed based on the different spatial signal responses within a pixel can be expressed as follows: , where C is an n×n calibration coefficient matrix, which is composed of calibration coefficient vectors corresponding to n calibration color bands; X is the color band fraction vector, which is the color band fraction vector of the incident light that contributes to the measured spatial signal; M is the spatial signal vector, which is composed of n spatial signals measured from the incident light.

[0117] Exemplarily, in the embodiments of the present application, for a Figures 7 - 8 pixel or image sensor as shown, each pixel performs a spatial signal response on the incident light expanded and decomposed based on the above 4 calibration color bands, forming the following system of four linear equations with one unknown.

[0118] ; ; ; ; Among them, represents the calibration coefficient of the spatial signal Mij measured on the photoelectric converter PDij for the bk component of the calibration color band of the incident light, represents the contribution ratio of the bk component of the calibration color band of the incident light to the spatial signal Mij, and Mij represents the spatial signal measured on the photoelectric converter PDij for the incident light of unknown color; i and j are integers from 1 to 2, and k is an integer from 1 to 4.

[0119] Exemplarily, in the embodiments of the present application, in the visible spectral range of 400-640 nm, as Figures 9 - 10 shown, blue light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 430 nm is used as the calibration color band Band 1 (abbreviated as b1), green light with a FWHM of 60 nm and a wavelength of 490 nm is used as the calibration color band Band 2 (abbreviated as b2), green light with a FWHM of 60 nm and a wavelength of 550 nm is used as the calibration color band Band 3 (abbreviated as b3), and red light with a FWHM of 60 nm and a wavelength of 610 nm is used as the calibration color band Band 4 (abbreviated as b4).

[0120] It can be understood that the above quaternary linear equation system can be written in the following form.

[0121] ; Among them, C is a 4×4 calibration coefficient matrix, X is a color band fraction vector of the calibration color band that contributes to the measured spatial signal in the incident light of unknown color, and M is a spatial signal vector measured for the incident light of unknown color; , , .

[0122] Multiply both sides of the above matrix equation by the inverse matrix of C to obtain the following relationship, which is the solution of the linear equation system: ; As can be seen from the above matrix equation, after the pixel structure and the selected 4 calibration color bands are determined, the calibration vector of each calibration color band for the pixel structure is determined accordingly, and then the calibration coefficient matrix is uniquely determined. Thus, the spatial signal response of the pixel structure to the incident light of unknown color can be understood as "encoding the incident light decomposed according to the 4 calibration color bands based on the calibration coefficient matrix to obtain 4 different spatial signals". At the same time, as can be seen from the above matrix equation, the spatial signal response simplification or encoding simplification can be performed each time using only one of the 4 calibration color bands, that is, making one element in X equal to 1 and the other three elements equal to 0 to simplify the matrix equation, and then some elements or vectors in the calibration coefficient matrix C can be obtained based on the corresponding measured spatial signals. Combining with the calibration coefficient elements or calibration coefficient vectors obtained by solving the spatial signal responses corresponding to the 4 calibration color bands, the complete calibration coefficient matrix C can be obtained, and the inverse matrix C of the calibration coefficient matrix C can be solved. -1 。

[0123] Finally, as can be seen from the above matrix equation, for the incident light X of unknown color, based on the inverse matrix C of the known calibration coefficient matrix C -1 and the spatial signal vector M corresponding to the incident light X, the incident light X can be directly analytically calculated based on the inverse matrix C of the calibration coefficient matrix C -1 and the measured spatial signal vector M. The incident light X is resolved into the spectral components corresponding to the above 4 calibration color bands, and while detecting the brightness of the incident light, the color measurement of the incident light is realized. This process can be understood as "decoding based on the inverse operation parameters of the encoding parameters (the inverse matrix C of the calibration coefficient matrix C -1 ) and the encoding results (4 different spatial signals) to obtain the decomposition results of the incident light under the 4 calibration color bands".

[0124] Exemplarily, in the embodiment of the present application, let , and only use the red light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 610 nm as the calibration color band Band 4 for monochromatic band pre-calibration simplification. Then the corresponding system of four linear equations with four unknowns is simplified to: ; ; ; ; At the same time, according to the actual measured values of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are obtained as: QE PD11 = 16.7, QE PD12 = 12.7, QE PD21 = 12.7, QE PD22 = 14.8.

[0125] Exemplarily, in the embodiment of the present application, let , only the green light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 550 nm is used as the calibration color band Band 3 for monochromatic band pre-calibration and simplification. Then, the corresponding system of four linear equations with four unknowns is simplified to: ; ; ; ; Meanwhile, according to the actual measured values of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are obtained as: QE PD11 = 20.2, QE PD12 = 14, QE PD21 = 16, QE PD22 = 17.5.

[0126] Exemplarily, in the embodiment of the present application, let , only the green light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 490 nm is used as the calibration color band Band 2 for monochromatic band pre-calibration and simplification. Then, the corresponding system of four linear equations with four unknowns is simplified to: ; ; ; ; Meanwhile, according to the actual measured values of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are obtained as: QE PD11 = 19.8, QE PD12 = 15, QE PD21 = 15.4, QE PD22 = 17.3.

[0127] Exemplarily, in the embodiment of the present application, let , only the blue light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 430 nm is used as the calibration color band Band 1 for monochromatic band pre-calibration and simplification. Then, the corresponding system of four linear equations with four unknowns is simplified to: ; ; ; ; Meanwhile, according to the actual measured values of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are obtained as: QE PD11 = 18.1, QE PD12 = 16.9, QE PD21 = 14.4, QE PD22 = 19.9.

[0128] Thus, by comprehensively pre-calibrating and simplifying the above four single-color calibration color bands, the corresponding calibration coefficient matrix C is obtained as shown in Table 1 below. Among them, the four calibration coefficients in each column correspond to the pre-calibration and simplification results of a calibration color band.

[0129] Table 1

[0130] At the same time, for calculation considerations, the four calibration coefficients corresponding to the pre-calibration and simplification results of a single calibration color band are normalized to obtain the calibration coefficient matrix C as shown in Table 2 below.

[0131] Table 2

[0132] In some embodiments, for each pixel, the incident light is analytically calibrated by pre-calibrated calibration parameters and n spatial signals, and the spectral components of the incident light under each calibration color band are calculated, including: For each pixel, based on the n-ary linear equation system, derivation and solution are carried out to obtain the analytical calibration formula: , where, C -1 is the inverse matrix of the calibration coefficient matrix; For each pixel, a spatial signal vector is constructed based on the n spatial signals, and the inverse matrix of the calibration coefficient matrix and the spatial signal vector are brought into the analytical calibration formula to calculate the color band fraction vector, and the spectral components of the incident light under each calibration color band are obtained.

[0133] Exemplarily, the inverse calculation of the calibration coefficient matrix C shown in the above table is performed to obtain the inverse matrix C -1 of the calibration coefficient matrix C as shown in Table 3 below.

[0134] Table 3

[0135] Thus, the pre-calibration or pre-calibration of the corresponding pixel or image sensor is realized. Subsequently, for the incident light of unknown color, based on the formula, the incident light X can be directly calculated according to the inverse matrix C -1 of the calibration coefficient matrix C and the measured spatial signal vector M, and the incident light X is resolved into the spectral components corresponding to 4 calibration color bands, that is , while realizing the detection of the incident light brightness, the color measurement of the incident light is realized.

[0136] ]>In the embodiments of the present invention, by Figure 9It can be seen that the asymmetric distribution patterns of the spatial signals of incident light of different colors on the four photoelectric converters are different. The asymmetric spatial patterns of the spatial signals recorded on the four photoelectric converters can be regarded as the fingerprints of the incident spectrum / color. Thus, by writing a system of linear equations connecting the spatial signals of the photoelectric converters with the spectral components of the measured incident light signals, the spectral components corresponding to the four calibration color bands can be analyzed and decoded. To decode the four measured spatial signals, the pixel sharing an asymmetric optical module can be pre-calibrated by irradiating multiple photoelectric converters with calibration color bands of different wavelength bands in sequence, and the corresponding spatial signal responses are recorded to obtain a calibration coefficient matrix. Then, the inverse matrix of the calibration coefficient matrix is used to calculate the incident light of the unknown color and solve the system of linear equations. Each element of the calibration coefficient matrix is used as the coefficient of the system of linear equations connecting the four spatial measurement signals to the four calibration color bands.

[0137] In the embodiment of the present application, by combining the asymmetric optical module with the multi-color band calibration technical means, the microlens and the filter in the traditional pixel are replaced, and the processes of incident light brightness detection and color detection are as follows Figure 11 As shown, after resolving the incident light X into the spectral components corresponding to the four calibration color bands based on the above process, each obtained spectral component is multiplied by the total measurement signal of the four photoelectric converters, so that each spectral component is converted into the least significant bit (LSB) of the total measurement signal. Finally, based on the least significant bit, the spectral components corresponding to the four calibration color bands are converted into the standard RGB color space for display.

[0138] For example, in order to convert the measured spectral components into standardized values, each spectral component needs to be normalized, that is, each spectral component is divided by the total measurement signal of the four photoelectric converters to eliminate the influence of the light source intensity change and ensure the relative ratio of the measured values is consistent.

[0139] Next, the embodiment of the present application further provides an imaging system, and the imaging system includes: An imaging lens for converging and imaging the light of an object to form incident light; An image sensor for focusing and diffracting the incident light based on the pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, and then correspondingly recording and forming n different spatial signals on n photoelectric converters in the pixels; where n is a positive integer greater than or equal to 3; Wherein, the pixel is configured to: based on the decomposition of the incident light by n calibration color bands, analyze and calibrate the incident light through the pre-calibration calibration parameters obtained from the pre-calibration of the n calibration color bands and the n spatial signals, and obtain the spectral components of the incident light under each calibration color band.

[0140] Exemplarily, as Figure 15As shown, the imaging system includes: An imaging lens that converges the light of an object and forms an image; The above-mentioned image sensor asymmetrically collects spatial signals of incident light based on the above-mentioned pixels, and combines the pre-calibration of 4 calibration color bands within the visible light spectrum range to resolve the incident light into spectral components corresponding to the 4 calibration color bands, so as to perform brightness detection and color detection on the incident light within each pixel.

[0141] Exemplarily, such as Figure 15 and Figure 16 As shown, in the embodiment of the present application, since the asymmetric optical module and the subsequent multi-color band calibration technology within the visible light spectrum range can effectively replace the microlens and filter structure in the relevant pixels, therefore, the image sensor in the embodiment of the present application does not require any structural offset to adapt to a high chief ray angle, and the image sensor in the embodiment of the present application can also adapt to an imaging lens with any aperture number F. As Figure 16 As shown, experiments prove that imaging lenses with two different high chief ray angles CRA and different aperture numbers F, namely lens 1 and lens 2, can both effectively adapt to the structure of the image sensor in the embodiment of the present application.

[0142] In this way, the assembly of the imaging system can also be simplified. According to the needs, different specifications of imaging lenses + different specifications of image sensors are assembled to obtain an imaging system, and the assembled imaging system is calibrated based on the pre-calibration of multiple calibration color bands within the visible light spectrum range.

[0143] Finally, as Figure 17 As shown, the embodiment of the present application also provides an optoelectronic device 1700.

[0144] Specifically, the optoelectronic device 1700 includes a processor 1701 with one or more processing cores, a memory 1702 with one or more computer-readable storage media, and a computer program stored on the memory 1702 and executable on the processor 1701. Among them, the processor 1701 is electrically connected to the memory 1702. Those skilled in the art can understand that the structure of the optoelectronic device shown in the figure does not constitute a limitation on the optoelectronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0145] The processor 1701 is the control center of the optoelectronic device 1700, connecting various parts of the entire optoelectronic device 1700 through various interfaces and lines. By running or loading software programs and / or units stored in the memory 1702, and calling the data stored in the memory 1702, it executes various functions of the optoelectronic device 1700 and processes data, thereby monitoring the optoelectronic device 1700 as a whole. The processor 1701 can be a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0146] In the embodiments of the present application, the processor 1701 in the optoelectronic device 1700 will, according to the corresponding steps, load the instructions corresponding to the processes of one or more application programs into the memory 1702, and the processor 1701 will run the application programs stored in the memory 1702, thereby implementing various functions, such as executing the steps of the above-mentioned calibration method for the image sensor.

[0147] For the specific implementation of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0148] As Figure 17 shown, the optoelectronic device 1700 further includes an image sensor (not shown in the figure). The processor 1701 is electrically connected to the image sensor respectively, and the image sensor can be the image sensor involved in the above embodiments. Those skilled in the art can understand that Figure 17 the optoelectronic device structure shown in

[0149] does not constitute a limitation on the optoelectronic device, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0150] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0151] Therefore, the embodiments of the present application also provide a computer-readable storage medium, in which multiple computer programs are stored. These computer programs can be loaded by a processor to execute any one of the calibration methods for the image sensor provided in the embodiments of the present application. These computer programs can execute the steps of the above-mentioned calibration method for the image sensor.

[0152] For the specific implementation of each of the above operations, reference may be made to the foregoing embodiments and will not be elaborated herein.

[0153] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0154] Since the computer program stored in the computer-readable storage medium can execute any one of the image sensor calibration methods provided in the embodiments of the present application, the beneficial effects achievable by any one of the image sensor calibration methods provided in the embodiments of the present application can be achieved. For details, reference may be made to the foregoing embodiments and will not be elaborated herein.

[0155] The foregoing has introduced in detail a pixel, an image sensor, an image sensor calibration method, an optoelectronic device, a computer-readable storage medium, and an imaging system provided in the embodiments of the present application. In the embodiments of the present application, specific examples are used to elaborate the principle and implementation manner. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the embodiments of the present application.

[0156] It should be noted that the embodiments in the content of the present application are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments may be referred to each other.

[0157] It should also be noted that in the content of the present application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0158] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the content of this application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the content of this application can be implemented in other embodiments without departing from the spirit or scope of the content of this application. Therefore, the content of this application will not be limited to these embodiments shown in the content of this application, but rather to the broadest scope consistent with the principles and novel features disclosed in the content of this application.

Claims

1. A pixel, characterized in that, The pixel includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module, disposed on the photoelectric conversion module, where the asymmetric optical module is configured to perform focusing diffraction on incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module in one-to-one correspondence; Wherein, the pixel is configured to: on the basis of decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through the pre-calibration parameters obtained from the pre-calibration of the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each calibration color band.

2. The pixel according to claim 1, characterized in that, The photoelectric conversion module only includes photoelectric converters of one size, and the n photoelectric converters in the photoelectric conversion module are arranged in a regular array.

3. The pixel according to claim 1, characterized in that, The photoelectric conversion module includes at least two different sizes of photoelectric converters, and the n photoelectric converters in the photoelectric conversion module are arranged in an irregular array.

4. The pixel according to claim 1, wherein The asymmetric optical module includes: A background structure layer, which is composed of a first material having a first refractive index; A diffraction structure layer embedded in the background structure layer, the diffraction structure layer includes a plurality of components composed of a second material having a second refractive index for performing focusing diffraction on the incident light, and the first refractive index is lower than the second refractive index.

5. The pixel according to claim 4, characterized in that, The first material and the second material are inorganic materials.

6. The pixel according to claim 5, characterized in that, The component includes at least two different sizes of diffraction cylinders and at least two different sizes of diffraction ring cylinders, and the plurality of diffraction cylinders and the plurality of diffraction ring cylinders are staggered and asymmetrically arranged, so that the diffraction of the incident light by the diffraction structure layer is asymmetric diffraction.

7. An image sensor, characterized in that, The image sensor includes a plurality of pixels arranged in an array; The pixel includes: A photoelectric conversion module, including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3; An asymmetric optical module, disposed on the photoelectric conversion module, where the asymmetric optical module is configured to perform focusing diffraction on incident light to generate a three-dimensional asymmetric light intensity pattern, and then record and form n different spatial signals on the n photoelectric converters in the photoelectric conversion module in one-to-one correspondence; Wherein, the pixel is configured to: on the basis of decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through the pre-calibration parameters obtained from the pre-calibration of the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each calibration color band.

8. The image sensor according to claim 7, wherein, The image sensor only includes pixels of one structure, and the pixels in the image sensor are arranged in a regular array.

9. The image sensor according to claim 7, characterized in that, The image sensor includes at least two different structures of pixels, and the pixels in the image sensor are arranged in a regular array.

10. The image sensor according to claim 9, wherein In the image sensor, n has only one value. However, the array arrangement structures of the n photoelectric converters in at least two of the photoelectric conversion modules are different.

11. The image sensor according to claim 9, characterized in that, In the image sensor, n has at least two different values, so that the array arrangement structures of the n photoelectric converters in at least two of the photoelectric conversion modules are different.

12. The image sensor according to claim 9, wherein In the image sensor, n has only one value, and the array arrangement structures of the n photoelectric converters in each of the photoelectric conversion modules are the same. However, the asymmetric optical modules in at least two of the pixels are different.

13. The image sensor according to claim 8, characterized in that, The pixel further includes a back-illuminated silicon substrate. The photoelectric conversion module is disposed on the back-illuminated silicon substrate, and trenches are provided around the photoelectric conversion module on the back-illuminated silicon substrate.

14. An imaging system, characterized in that, The imaging system includes: An imaging lens for converging and imaging the light of an object to form incident light. An image sensor for focusing and diffracting the incident light based on the pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, and then correspondingly recording and forming n different spatial signals on the n photoelectric converters in the pixels; where n is a positive integer greater than or equal to 3. Wherein, the pixel is configured to: based on decomposing the incident light by n calibration color bands, perform analytical calibration on the incident light through the pre-calibration parameters obtained from the pre-calibration of the n calibration color bands and the n spatial signals, so as to obtain the spectral components of the incident light under each of the calibration color bands.

15. A calibration method for an image sensor, characterized in that, Applied to an image sensor composed of multiple pixels, the method includes: For each pixel, select n calibration color bands to decompose the incident light. For each pixel, perform pre-calibration based on the n calibration color bands to obtain pre-calibration parameters. For each pixel, obtain n different spatial signals obtained by the pixel through photoelectric acquisition of the incident light. For each pixel, perform analytical calibration on the incident light through the pre-calibration parameters and the n spatial signals, and calculate the spectral components of the incident light under each of the calibration color bands. Wherein, the pixel includes: A photoelectric conversion module including n photoelectric converters arranged adjacent to each other in an array form, where n is a positive integer greater than or equal to 3. [[ID= 16. The method according to claim 15, characterized in that, ​ 17. The method according to claim 15, wherein ​ For each of the pixels, based on the spatial signal responses of the incident light decomposed by the n calibration color bands on the n photoelectric converters within the pixel, a system of n linear equations with one unknown is constructed, where the system of n linear equations with one unknown consists of a calibration coefficient matrix, a color band fraction vector, and a spatial signal vector; For each of the pixels, traverse the n calibration color bands, and based on the spatial signal response of a single calibration color band, simplify and calculate the system of n linear equations with one unknown to obtain the calibration coefficient vector of a single calibration color band, and then synthesize the calibration coefficient vectors of the n calibration color bands to obtain the calibration coefficient matrix; For each of the pixels, perform an inverse operation on the calibration coefficient matrix to obtain the inverse matrix of the calibration coefficient matrix, and the inverse matrix of the calibration coefficient matrix is the pre-calibration and calibration parameter.

18. The method according to claim 17, wherein The matrix form of the system of n linear equations with one unknown is as follows: ， Among them, C is the calibration coefficient matrix of n×n, which is composed of the calibration coefficient vectors corresponding to n calibration color bands; X is the color band fraction vector, which is the color band fraction vector in the incident light that contributes to the measured spatial signal; M is the spatial signal vector, which is composed of n spatial signals measured by the incident light.

19. The method according to claim 18, characterized in that, For each of the pixels, perform analytical calibration on the incident light through the pre-calibration and calibration parameter and the n spatial signals, and calculate the spectral components of the incident light under each calibration color band, including: For each of the pixels, perform derivation and solution based on the system of n linear equations with one unknown to obtain an analytical calibration formula: ， Among them, C -1 is the inverse matrix of the calibration coefficient matrix; For each of the pixels, construct the spatial signal vector based on the n spatial signals, and substitute the inverse matrix of the calibration coefficient matrix and the spatial signal vector into the analytical calibration formula to calculate the color band fraction vector, and obtain the spectral components of the incident light under each calibration color band.

20. An optoelectronic device, characterized in that, Including: A memory, on which a computer program is stored; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 15 to 19.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 15 to 19 are implemented.

Images