Pixels, image sensors, imaging systems, methods of operation, apparatuses, and media

Abstract

The application relates to a pixel, an image sensor, an imaging system, an operation method, equipment and a medium, the pixel comprising: a photoelectric conversion module comprising n photoelectric converters arranged adjacently in an array form, wherein n is a positive integer greater than or equal to 3; and an asymmetric optical module arranged on the photoelectric conversion module, wherein the asymmetric optical module is used for focusing diffraction on incident light to generate a three-dimensional asymmetric light intensity pattern, and then form n different spatial signals one by one on the n photoelectric converters in the photoelectric conversion module; wherein the pixel is configured to determine the source of the n spatial signals according to the spatial correlation between the n spatial signals in the photoelectric conversion module, so as to exclude the interference of internal noise. The method is helpful to reduce the influence of noise on the image quality and improve the quality of the image by analyzing the spatial correlation of the spatial signals of the photoelectric converter.

Description

Technical Field

[0001] The present application relates to the field of optoelectronic technology, and in particular to a pixel, an image sensor, an imaging system, an operating method, a device, and a medium. Background Art

[0002] In a traditional image sensor, each pixel consists of at least a microlens, a filter, and a photoelectric converter (such as a photodiode). The microlens focuses light, ensuring it effectively passes through the filter and strikes the photoelectric converter. The photoelectric converter then generates an electric charge or current based on the incident light. However, during the photoelectric conversion process, the electrical signal is inevitably affected by noise, which in turn affects image quality and subsequent processing. Summary of the Invention

[0003] The embodiments of the present application provide a pixel, an image sensor, an imaging system, an operating method, a device, and a medium to solve the above-mentioned problems.

[0004] In order to achieve the above-mentioned objective, according to a first aspect of the present application, a pixel is provided, comprising:

[0005] A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3;

[0006] an asymmetric optical module, the asymmetric optical module being disposed on the photoelectric conversion module, wherein the asymmetric optical module is configured to focus and diffract incident light to generate a three-dimensional asymmetric light intensity pattern, thereby recording n different spatial signals in a one-to-one correspondence on the n photoelectric converters in the photoelectric conversion module;

[0007] The pixel is configured to determine sources of the n spatial signals according to spatial correlations among the n spatial signals in the photoelectric conversion module, so as to eliminate interference from internal noise.

[0008] Optionally, the pixel is also configured to: decompose the incident light with n calibration color bands after eliminating internal noise interference, and analytically calibrate the incident light using pre-calibration calibration parameters obtained by pre-calibration calibration of the n calibration color bands and n spatial signals to obtain the spectral components of the incident light under each calibration color band.

[0009] Optionally, the pixel is further configured to: superimpose and sum the n spatial signals after eliminating internal noise interference to obtain a spatial superposition signal, and detect brightness changes of the incident light based on the spatial superposition signal.

[0010] Optionally, the asymmetric optical module includes:

[0011] a background structure layer composed of a first material having a first refractive index;

[0012] a diffractive structure layer embedded in the background structure layer, the diffractive structure layer comprising a plurality of components composed of a second material having a second refractive index for focusing diffraction of the incident light, the first refractive index being lower than the second refractive index.

[0013] Optionally, the first material and the second material are inorganic materials.

[0014] Optionally, the components comprise at least two different sizes of diffractive cylinders and at least two different sizes of diffractive circular cylinders, a plurality of the diffractive cylinders and a plurality of the diffractive circular cylinders being staggered and asymmetrically arranged so that the diffractive structure layer performs asymmetric diffraction of the incident light.

[0015] According to a second aspect of the present application, embodiments of the present application further provide an image sensor, the image sensor comprising a plurality of pixels arranged in an array; the pixel comprising:

[0016] a photoelectric conversion module comprising n photoelectric converters arranged adjacent in an array, wherein n is a positive integer greater than or equal to 3;

[0017] an asymmetric optical module arranged on the photoelectric conversion module, wherein the asymmetric optical module is configured to perform focusing diffraction of incident light to generate a three-dimensional asymmetric light intensity pattern, and further to form n different spatial signals one-to-one corresponding on the n photoelectric converters in the photoelectric conversion module;

[0018] wherein the pixel is configured to determine the source of the n spatial signals according to the spatial correlation between the n spatial signals in the photoelectric conversion module to exclude the interference of internal noise.

[0019] Optionally, the image sensor comprises at least two different structures of the pixel, and each of the pixels in the image sensor is arranged in a regular array.

[0020] Optionally, the pixel is further configured to, under the condition that the internal noise interference is excluded, decompose the incident light into n calibration color bands, and calibrate the incident light by using pre-calibration parameters obtained by pre-calibration of the n calibration color bands and the n spatial signals to obtain spectral components of the incident light under each of the calibration color bands.

[0021] Optionally, the pixel is further configured to: superimpose and sum the n spatial signals after eliminating internal noise interference to obtain a spatial superposition signal, and detect brightness changes of the incident light based on the spatial superposition signal.

[0022] Optionally, the pixel further includes a back-illuminated silicon substrate, the photoelectric conversion module is arranged on the back-illuminated silicon substrate, and a groove is provided on the back-illuminated silicon substrate surrounding the photoelectric conversion module.

[0023] According to a third aspect of the present application, an embodiment of the present application further provides an imaging system, comprising:

[0024] Imaging lens, used to converge the light of the object into an image to form incident light;

[0025] An image sensor configured to focus and diffract the incident light based on pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, and then record n ​​different spatial signals on n photoelectric converters in the pixels in a one-to-one correspondence; wherein n is a positive integer greater than or equal to 3;

[0026] Wherein, the pixels are configured as follows:

[0027] determining sources of the n spatial signals according to spatial correlations between the n spatial signals to eliminate interference from internal noise;

[0028] Under the condition of eliminating internal noise interference, the incident light is decomposed by using n calibration color bands, and the incident light is analytically calibrated using pre-calibrated calibration parameters obtained by pre-calibrating the n calibration color bands and the n spatial signals to obtain the spectral components of the incident light in each of the calibration color bands;

[0029] And / or, when internal noise interference is eliminated, the n spatial signals are superimposed and summed to obtain a spatial superposition signal, and the brightness change of the incident light is detected based on the spatial superposition signal.

[0030] According to a fourth aspect of the present application, an embodiment of the present application further provides an operating method of an image sensor, which is applied to an image sensor composed of multiple pixels, and the method includes:

[0031] Acquiring an imaging mode of the image sensor, wherein the imaging mode includes a spectral imaging mode, an event imaging mode, and a fusion imaging mode;

[0032] Based on the imaging mode, controlling the image sensor to collect corresponding incident light in units of the pixel, to obtain n different spatial signals corresponding to each pixel;

[0033] Determining the source of the spatial signal using the pixel as a unit to eliminate interference from internal noise;

[0034] When interference from the internal noise is eliminated, based on the imaging mode, detection is performed in units of the pixels, and the color of the incident light is detected according to the n spatial signals within the pixels, and / or the brightness change of the incident light is detected according to the n spatial signals within the pixels;

[0035] The pixels include:

[0036] A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3;

[0037] An asymmetric optical module is arranged on the photoelectric conversion module, wherein the asymmetric optical module is used to focus and diffract the incident light to produce a three-dimensional asymmetric light intensity pattern, and then record the n different spatial signals on the n photoelectric converters in the photoelectric conversion module in a one-to-one correspondence.

[0038] Optionally, the determining the source of the spatial signal in units of pixels to eliminate interference from internal noise includes:

[0039] For each pixel, obtaining corresponding n different spatial signals;

[0040] For each of the pixels, analyzing the spatial correlation between the n spatial signals according to the signal values ​​of the n spatial signals;

[0041] For each of the pixels, determining sources of the n spatial signals according to the spatial correlation;

[0042] For each of the pixels, when the n spatial signals corresponding to the pixel are derived from the internal noise, the n spatial signals corresponding to the pixel are removed, and / or the n spatial signals corresponding to the pixel are compensated and corrected to eliminate the interference of the internal noise.

[0043] Optionally, the spatial correlation includes a first correlation, a second correlation, and a third correlation, and analyzing the spatial correlation between the n spatial signals according to the signal values ​​of the n spatial signals for each pixel includes:

[0044] For each of the pixels, constructing a spatial signal matrix based on the signal values ​​of the n spatial signals;

[0045] For each of the pixels, analyzing the spatial correlation between the n spatial signals based on the values ​​and distribution patterns of the signal values ​​in the spatial signal matrix;

[0046] Wherein, when the n signal values ​​in the spatial signal matrix are different and all are non-zero values, the n spatial signals exhibit the first correlation;

[0047] When the signal values ​​of at least one row or column in the spatial signal matrix are the same non-zero value, the n spatial signals present the second correlation;

[0048] In the case that some of the n signal values ​​in the spatial signal matrix are zero and some are non-zero values, and the non-zero values ​​are randomly distributed, the third correlation exists between the n spatial signals.

[0049] Optionally, for each pixel, determining sources of the n spatial signals according to the spatial correlation includes:

[0050] In a case where the first correlation exists between the n spatial signals, determining that the n spatial signals corresponding to the pixel originate from the incident light;

[0051] When the second correlation is present between the n spatial signals, determining that the n spatial signals corresponding to the pixel are derived from the internal noise, and the internal noise is row stripe noise or column stripe noise;

[0052] In a case where the third correlation exists between the n spatial signals, it is determined that the n spatial signals corresponding to the pixel are derived from the internal noise, and the internal noise is random noise.

[0053] Optionally, detecting the color of the incident light according to the n spatial signals within the pixel includes:

[0054] For each pixel, selecting n calibration color bands to decompose the incident light;

[0055] For each of the pixels, pre-calibration is performed based on the n calibration ribbons to obtain pre-calibration parameters;

[0056] For each pixel, obtaining n different spatial signals obtained by the pixel performing photoelectric collection on the incident light;

[0057] For each of the pixels, the incident light is analytically calibrated using the pre-calibrated calibration parameters and the n spatial signals, and the spectral components of the incident light in each of the calibration color bands are calculated.

[0058] Optionally, detecting a brightness change of the incident light according to the n spatial signals within the pixel includes:

[0059] Superimposing and summing the n spatial signals to obtain a spatial superposition signal;

[0060] A brightness change of the incident light is detected based on the spatial superposition signal.

[0061] According to a fifth aspect of the present application, an embodiment of the present application further provides an optoelectronic device, including:

[0062] a memory having a computer program stored thereon;

[0063] A processor is used to execute the computer program in the memory to implement the steps of the method described in any one of the embodiments of the present application.

[0064] According to the sixth aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.

[0065] Some embodiments of the present application have at least the following beneficial effects: based on the spatial correlation analysis between each spatial signal within each pixel, it can accurately identify and quantify the internal noise sources of the image sensor, such as dark current noise, etc., which helps to eliminate the interference of the internal noise of the sensor and eliminate the spatial superposition signal generated by the superposition of noise intensity, which helps to more accurately distinguish the intensity of external light sources, improve the accuracy of incident light brightness detection and color detection, enhance the sensing capability of distant objects, and provide strong support for high-performance imaging of image sensors in complex environments.

[0066] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0068] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0069] Figure 1 This is a schematic diagram of the pixel structure in a traditional image sensor;

[0070] Figure 2 It is a schematic diagram of the structure of a traditional RGGB image sensor;

[0071] Figure 3 This is a schematic diagram comparing the quantum efficiency (QE) curves of monochrome image sensors and traditional RGGB image sensors in the visible spectrum range;

[0072] Figure 4 This is a schematic diagram of light sensing of a distant object point by an imaging system in the related art;

[0073] Figure 5 It is a structural diagram of an imaging system in related art;

[0074] Figure 6 is a graph showing the distribution of the chief ray angle of an imaging lens in an imaging system in the related art;

[0075] Figure 7 This is a schematic diagram of the process flow of a traditional RGGB image sensor;

[0076] Figure 8 This is a schematic structural diagram of a pixel in an optional embodiment of the present application;

[0077] Figure 9 is a three-dimensional view of a pixel in an optional embodiment of the present application;

[0078] Figure 10 This is a schematic diagram of diffraction and self-interference of an imaging system in an optional embodiment of the present application;

[0079] Figure 11 This is a schematic diagram of imaging of image sensor noise and external light sources by pixels in an optional embodiment of the present application;

[0080] Figure 12 This is a schematic diagram of imaging of image sensor noise and external light sources by an imaging system in an optional embodiment of the present application;

[0081] Figure 13 This is a schematic diagram of the imaging process of an imaging system with a noise recognition function in an optional embodiment of the present application;

[0082] Figure 14 1 is a schematic diagram of a matrix determinant of a spatial signal corresponding to a single-color band calibration using 610 nm red light in an optional embodiment of the present application;

[0083] Figure 15 1 is a schematic diagram of a matrix determinant of a spatial signal corresponding to a single-color band calibration using 550nm green light in an optional embodiment of the present application;

[0084] Figure 161 is a schematic diagram of a matrix determinant of a spatial signal corresponding to a single-color band calibration using 490nm green light in an optional embodiment of the present application;

[0085] Figure 17 1 is a schematic diagram of a matrix determinant of a spatial signal corresponding to a single-color band calibration using 430nm blue light in an optional embodiment of the present application;

[0086] Figure 18 This is a schematic diagram of noise recognition based on a 2×2 spatial signal matrix in an optional embodiment of the present application;

[0087] Figure 19 This is another schematic diagram of noise recognition based on a 2×2 spatial signal matrix in an optional embodiment of the present application;

[0088] Figure 20 Schematic diagram of asymmetric diffraction of different incident light by an asymmetric optical module in an optional embodiment of the present application;

[0089] Figure 21 is a schematic diagram of four calibration ribbons selected in an optional embodiment of the present application;

[0090] Figure 22 1 is a schematic diagram of a pixel calibration process in an optional embodiment of the present application;

[0091] Figure 23 This is a flowchart of an operating method of an image sensor in an optional embodiment of the present application;

[0092] Figure 24 This is a schematic structural diagram of an imaging system in an optional embodiment of the present application;

[0093] Figure 25 is a graph showing the distribution of the chief ray angle of the imaging lens of the imaging system in an optional embodiment of the present application;

[0094] Figure 26 This is a schematic structural diagram of an optoelectronic device in an optional embodiment of the present application. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0096] To facilitate understanding of the implementation scheme provided in the embodiments of the present application, the relevant application background of the pixels, image sensors, imaging systems, image sensor operation methods, optoelectronic devices, and computer-readable storage media provided in the embodiments of the present application are first described.

[0097] like Figure 1 As shown in the figure, in a traditional image sensor, each pixel includes at least a microlens, a filter and a photoelectric converter (such as a photodiode). The microlens is used to converge light to ensure that the light can effectively pass through the filter and illuminate the photoelectric converter, thereby improving the light collection efficiency of the photoelectric converter. The photoelectric converter generates charge or current based on the incident light. In order to produce a color image, each pixel is equipped with a red filter, a green filter or a blue filter, which transmits the corresponding color band and blocks the rest of the light in the visible spectrum. For example, the filter can be used as follows Figure 2 The RGGB pattern (also called the Bayer pattern) shown is repeatedly arranged in the image sensor.

[0098] Since each filter transmits only a narrow spectral band and blocks the rest of the light, at any given photoelectric converter position or pixel position, only one of the color band signals transmitted by the filter can be recorded. In an optional embodiment of the present application, a monochrome image sensor without any color filter is compared with a traditional RGGB image sensor to obtain the quantum efficiency (QE) curves of the two in the visible spectrum range, as shown in Figure 2. Figure 3 As shown. Figure 3 It can be seen that the spectral filtering of traditional RGGB image sensors reduces the white light broadband sensitivity by about 3-4 times compared with monochrome image sensors that do not contain any color filters.

[0099] At the same time, in order to infer the block color signal at any given pixel position in the RGGB image sensor, the signals of adjacent color pixels must be interpolated (also called demosaicing) to obtain the block color signal at the given position, which reduces the spectral resolution of the color image sensor.

[0100] Furthermore, the photosensitivity of the image pixels is particularly important in the event imaging mode of operation, which detects the continuity of events based on the evaluation of the acquired signal over time, with high photosensitivity (for any given illumination level, without loss due to absorption by the pixel filter). Conventional microlenses only provide unstructured light intensity information to the photoelectric converter, such as Figure 4As shown, when using a traditional pixel microlens to place an object at a long distance (e.g., over 50 meters), distinguishing dark from bright signals becomes particularly difficult. The probability of light from distant objects being collected by the traditional pixel microlens is low, and background noise or sensor-intrinsic noise (i.e., sensor-internal noise, such as signal readout noise and dark current noise) is relatively high. Therefore, the collected low-light signal is difficult to distinguish from background noise and sensor-intrinsic noise. In this case, it is desirable to provide a more robust and universal method to verify the source of the signal, effectively identifying whether the signal originates from external incident light or internal sensor noise, thereby eliminating interference from internal sensor noise.

[0101] like Figure 5-Figure 6 As shown in the figure, in many existing application scenarios, the imaging system is equipped with a multi-element imaging lens, which sends the incident light for imaging to the image sensor at a very high angle (up to 35 degrees of the chief ray angle). However, in the image sensor of the related art, the pixels have a pixel stack height of 2-4μm between the top surface of the microlens and the interface with the photosensitive substrate. Light incident on the microlens at a high angle is focused by the microlens and experiences a large displacement during its propagation through the pixel stack. Therefore, in each pixel, the microlens and filter need to be shifted relative to the photoelectric converter to collect light into the appropriate photoelectric converter.

[0102] Specifically, by Figure 5 and Figure 6 It can be seen that the displacement of the microlens and filter relative to the photoelectric converter depends on many factors, such as the pixel stack height, the number of pixel stack layers, the optical properties of the material, the pixel size, the image sensor's imaging lens chief ray angle distribution curve, and the focal length f of the imaging lens. Therefore, designing the offset of the microlens and filter for a specific imaging lens and image sensor is typically a very lengthy and expensive process. Any modification to the imaging lens design, pixel size, or pixel stack height also incurs additional costs due to the substantial changes in the manufacturing process and the need to verify the extremely high stability of the new process to produce high-yield parts.

[0103] Therefore, by reducing pixel stacking and, ideally, avoiding the need to shift any part of the pixel stack, it would be very beneficial to significantly reduce the cost of image sensors. During the assembly stage of the imaging system, the multi-element imaging lens needs to be three-dimensionally aligned with the image sensor array to achieve lateral coincidence of the optical centers of the imaging lens and the image sensor. In addition, it is also crucial to prevent the imaging lens from tilting relative to the sensor plane so that the image sensor with offset microlenses and filters aligned with the multi-element imaging lens can sense the light signal at the correct incident angle. Therefore, it is desirable to provide an image sensor that is not affected by the misalignment problem between the imaging lens and the image sensor, so as to reduce the process complexity during the subsequent assembly of the imaging system.

[0104] In addition, the deposition process of microlenses and color filters based on non-CMOS organic polymer materials requires additional non-CMOS production equipment for placing absorptive filters on the image sensor array. In an optional embodiment of the present application, the number of process steps, production cycle and production yield of the existing RGGB image sensor are as follows: Figure 7 shown.

[0105] For example, Figure 7 As shown, the addition of non-CMOS process technology lengthens the image sensor production process, and the yield of non-CMOS production equipment is low, thus increasing the cost of image sensors. Furthermore, traditional polymer-based absorptive filters are also susceptible to weathering, resulting in a shorter lifespan than inorganic materials used in CMOS production equipment. Therefore, it is desirable to integrate the color imaging function of image sensors into the CMOS manufacturing process to reduce the cost of image sensors and improve their yield.

[0106] In view of this, an embodiment of the present application proposes an image sensor based on a pure CMOS process: it is designed in pixels, and within each pixel, an asymmetric optical module shared by n photoelectric converters is designed based on at least two inorganic materials with different refractive indices, and the incident light is focused and diffracted to generate a three-dimensional asymmetric light intensity pattern, and then different spatial signals are recorded on the n photoelectric converters based on the three-dimensional asymmetric light intensity pattern; the three-dimensional asymmetric light intensity pattern is formed by the light diffraction and subsequent self-interference occurring on the asymmetric optical module, and the interference properties of light make the detected spatial signals highly correlated, that is, there is a high spatial correlation between the multiple different spatial signals obtained by each pixel collecting the incident light.

[0107] The internal noise of the sensor can be mainly divided into two categories: random noise and fixed noise. Random noise refers to noise that appears randomly, has no rules, and cannot be reproduced, such as dark current noise and thermal noise. When the source of the spatial signal within the pixel is not external incident light but random noise inside the sensor, there is no spatial correlation between the spatial signals within the pixel, showing a random distribution in which some spatial signals have values ​​and some spatial signals have no values; fixed noise is reproducible, usually manifested as stripes or regional brightness changes, showing stripe noise or regional noise. When the source of the spatial signal within the pixel is not external incident light but fixed noise inside the sensor, there is a certain spatial correlation between the spatial signals within the pixel, showing a distribution in which some spatial signals have values ​​and some spatial signals have no values, or a distribution in which all spatial signals have values, and multiple spatial signals with values ​​are distributed in rows, columns or regions with consistent signal values.

[0108] That is to say, under the pixel structure of the present application, when the spatial signals in the pixel are derived from three different signal sources, i.e., external incident light, random noise inside the sensor, and fixed noise inside the sensor, there are great differences in the spatial correlation between the corresponding multiple spatial signals. Thus, in each pixel, the source of the spatial signal can be identified based on the spatial correlation between the respective spatial signals, to determine whether the spatial signal is derived from external incident light, random noise inside the sensor, or fixed noise inside the sensor, so as to exclude the interference of the internal noise of the sensor.

[0109] In the case of excluding the interference of the internal noise, for the detection of the light-dark change or the brightness change of the external incident light, since the multiple spatial signals in the pixel are obtained by photoelectric induction of the asymmetrically focused diffraction of the same incident light, there is a very high spatial correlation between the multiple spatial signals. At this time, each spatial signal in the pixel can be directly superimposed and combined to obtain a spatial superposition signal with greater intensity, and the spatial superposition signal can be taken as an event sensing signal to detect the light-dark change or the brightness change of the incident light, thereby improving the event sensing capability of the distant object.

[0110] In the case of excluding the interference of the internal noise, for the color sensing detection of the external incident light, the n spatial signals obtained by the n photoelectric converters sharing the same asymmetric optical module in the pixel are different and combined to form a characteristic pattern of a specific incident light color and spectrum, so that the narrow spectral bands (hereinafter referred to as color bands) of various colors form n different spatial signal component distributions on the n photoelectric converters. Thus, n representative calibration color bands can be selected in the visible spectral range, the incident light of unknown color is expanded and decomposed into the sum of the components of the n calibration color bands, and based on the spatial signal responses of the incident light expanded and decomposed by the n calibration color bands on the n photoelectric converters, an n-element linear equation set is constructed, wherein the n-element linear equation set is composed of a calibration coefficient matrix, a color band score vector, and a spatial signal vector, and the calibration coefficient matrix can be pre-calibrated by the independent spatial signal responses of each calibration color band.

[0111] Therefore, based on the n-element linear equation set, the calibration coefficient matrix pre-calibrated and the n spatial signals measured in real time can be used to solve the color band components of the incident light, so as to analyze the incident light into the spectral components corresponding to the n calibration color bands, thereby realizing the brightness detection of the incident light and the color measurement of the incident light. By combining the asymmetric optical module and the multi-color band calibration technology to replace the microlens and the optical filter in the existing pixel, the production cost and the process complexity of the image sensor can be reduced, and the sensitivity and the spectral resolution of the incident light detection can be improved.

[0112] Where n is a positive integer greater than or equal to 3; multiple different representative calibration color bands are selected from the entire visible spectrum. For example, representative calibration color bands can be selected from narrow spectral bands corresponding to different colors. Exemplarily, at least three calibration color bands of red, green, and blue are included. That is, the selected calibration color bands include at least a red spectral band, a green spectral band, and a blue spectral band to meet the requirements for expanding and decomposing incident light of unknown color.

[0113] First, an embodiment of the present application provides a pixel, the pixel comprising:

[0114] A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3;

[0115] An asymmetric optical module is disposed on the photoelectric conversion module, wherein the asymmetric optical module is used to focus and diffract the incident light to produce a three-dimensional asymmetric light intensity pattern, and then record the n different spatial signals on the n photoelectric converters in the photoelectric conversion module in a one-to-one correspondence;

[0116] The pixel is configured to determine the sources of the n spatial signals according to the spatial correlation between the n spatial signals in the photoelectric conversion module, so as to eliminate the interference of internal noise.

[0117] It should be noted that the three-dimensional asymmetric light intensity pattern refers to the three-dimensional asymmetric light intensity pattern formed in the three-dimensional space on the photoelectric converter after the incident light is asymmetricly focused and diffracted by the asymmetric optical module. The three-dimensional asymmetric light intensity pattern is then projected onto the photosensitive plane of the photoelectric converter, forming a two-dimensional asymmetric light intensity distribution on each photoelectric converter at different spatial positions in the photosensitive plane. The light intensity distribution collected on the photoelectric converters at different positions is different, and then based on the photoelectric effect, the light is recorded one by one on n photoelectric converters to form n spatial signals of different sizes.

[0118] For example, Figure 8-Figure 9 As shown, the value of n is 4, the asymmetric optical module is arranged on a photoelectric conversion module composed of 4 photoelectric converters, and the 4 photoelectric converters in the photoelectric conversion module are arranged in a 2×2 regular array. The asymmetric optical module focuses and diffracts the incident light to generate a three-dimensional asymmetric light intensity pattern, which is recorded one-to-one on the 2×2 photoelectric converters in the photoelectric conversion module to form 4 different spatial signals.

[0119] In some embodiments, the asymmetric optical module comprises:

[0120] A background structure layer, the background structure layer is made of a first material having a first refractive index;

[0121] The diffraction structure layer is embedded in the background structure layer, and the diffraction structure layer includes a plurality of components made of a second material with a second refractive index for focusing and diffracting incident light, and the first refractive index is lower than the second refractive index.

[0122] It is understood that the geometric shape, size, material properties, and arrangement of the components can be flexibly designed according to actual needs to achieve asymmetric focusing and diffraction of the incident light. The incident light can be the light emitted directly by the light source or the light reflected from the object after the light source emits the light, which will not be described in detail here.

[0123] In some embodiments, the component may include a cylindrical, rectangular, V-shaped, annular, or the like.

[0124] In some embodiments, components may be arranged in a periodic, quasi-periodic, or random manner.

[0125] In some embodiments, the component includes at least two diffraction cylinders of different sizes and at least two diffraction ring cylinders of different sizes, and multiple diffraction cylinders and multiple diffraction ring cylinders are staggered and asymmetrically arranged so that the diffraction structure layer diffracts the incident light asymmetricly.

[0126] For example, Figure 8-Figure 9 As shown, the asymmetric optical module includes:

[0127] a background structure layer having a first refractive index;

[0128] An asymmetric diffraction structure layer having a second refractive index is embedded in the background structure layer, and the second refractive index is different from the first refractive index.

[0129] For example, Figure 8-Figure 9 As shown, the asymmetric diffraction structure layer is a transparent structure layer and is embedded in the bottom of the background structure layer. The asymmetric diffraction structure layer includes multiple diffraction cylinders and multiple diffraction ring cylinders. Multiple diffraction cylinders of different sizes and multiple diffraction ring cylinders of different sizes are staggered, so that the diffraction of the incident light by the asymmetric diffraction structure layer is asymmetric diffraction.

[0130] For example, Figure 8-Figure 9 As shown, the diameters of the multiple diffraction cylinders are different, the inner diameters and outer diameters of the multiple diffraction ring cylinders are different, and the heights of the diffraction cylinders and the heights of the diffraction ring cylinders are both smaller than the height of the background structure layer.

[0131] In some embodiments, the first material and the second material are inorganic materials.

[0132] It should be noted that the asymmetric optical module is made entirely of inorganic materials. Compared to traditional image sensors using organic materials, the use of inorganic materials eliminates the need for CMOS process flows, such as organic materials. This allows the entire image sensor to be manufactured entirely in a CMOS fab, reducing the cost of the image sensor. Furthermore, due to the high manufacturing precision and stringent requirements for organic materials, traditional image sensors with microlenses and filters have a low yield of approximately 80%. However, the all-inorganic CMOS image sensor in the embodiments of this application has a very high yield, approaching 100%. Furthermore, the inorganic materials exhibit greater compatibility, and the elimination of non-CMOS materials further reduces the potential for dark current. Consequently, the all-CMOS image sensor offers a longer lifespan and higher reliability.

[0133] Among them, the background structure layer can be composed of a low-refractive-index material (i.e., the first material), such as silicon oxide, whose refractive index is 1.46, or air, whose refractive index is 1; the asymmetric diffraction structure layer is composed of a high-refractive-index material (i.e., the second material), such as silicon nitride, titanium oxide, tantalum oxide, silicon carbide and silicon, and its refractive index can be in the range of 1.8~4.

[0134] It is understandable that in other optional embodiments, the asymmetric optical module can be composed of three or more inorganic materials, which are not limited here, such as a first background layer with a lower refractive index, a second background layer with a medium refractive index, and an asymmetric diffraction structure layer with a higher refractive index, which are nested in sequence from the inside to the outside; or a first background layer with a lower refractive index, a first asymmetric diffraction structure layer with a medium refractive index embedded in the first background layer, and a second asymmetric diffraction structure layer with a higher refractive index embedded in the first background layer, etc.

[0135] In some embodiments, the photoelectric conversion module includes only photoelectric converters of one size, and the n photoelectric converters in the photoelectric conversion module are arranged in a regular array.

[0136] For example, Figure 8-Figure 9 As shown, the four photoelectric converters in the photoelectric conversion module corresponding to the pixel have the same size, are all square photoelectric converters, the corresponding photosensitive surface is square in design, and the four photoelectric converters are arranged in a 2×2 regular array.

[0137] In some embodiments, the regular array arrangement may be a linear arrangement, a two-dimensional matrix arrangement, or the like.

[0138] For example, Figure 8-Figure 9 As shown, Figure 8 For the pixel framed by the dotted line, four photoelectric converters of the same size in the corresponding photoelectric conversion module are arranged in a 2×2 two-dimensional matrix.

[0139] It can be understood that the size and shape of the photoelectric converter in the pixel is not limited to a square. In other optional embodiments, the size and shape of the photoelectric converter can also be a rectangle, a diamond, a triangle, etc., and the corresponding photosensitive surface is designed in a rectangular, diamond, triangular, etc. shape. Multiple photoelectric converters of the same size can be arranged linearly along the horizontal direction, vertical direction or other diagonal directions, which is not limited here.

[0140] In some embodiments, the photoelectric conversion module includes photoelectric converters of at least two different sizes, and the n photoelectric converters in the photoelectric conversion module are arranged in an irregular array.

[0141] For example, the intra-pixel photoelectric conversion module includes one first rectangular photoelectric converter and two second rectangular photoelectric converters, and the three photoelectric converters are arranged in an irregular array; for another example, the intra-pixel photoelectric conversion module includes one first rectangular photoelectric converter, three second rectangular photoelectric converters and three square photoelectric converters, and the seven photoelectric converters are arranged in an irregular array.

[0142] It should be noted that the specific size or irregular arrangement of each photoelectric converter within a pixel can be customized according to specific needs. For example, larger photoelectric converters can be arranged in certain areas to improve local sensitivity, or smaller photoelectric converters can be used in other areas to achieve higher spatial resolution.

[0143] In some embodiments, the pixel further includes a back-illuminated silicon substrate, the photoelectric conversion module is disposed on the back-illuminated silicon substrate, and a groove is disposed on the back-illuminated silicon substrate surrounding the photoelectric conversion module.

[0144] For example, Figure 9 As shown, the pixel also includes a back-illuminated silicon substrate (Back-Side illuminated Si, BSI Si), the photoelectric conversion module is arranged on the back-illuminated silicon substrate, the asymmetric optical module is arranged on the photoelectric conversion module, and the pixel also includes a cell deep trench isolation structure (Cell Deep Trench Isolation, CDTI) arranged in the back-illuminated silicon substrate, and the cell deep trench isolation structure is arranged around the 2×2 photoelectric converters in the photoelectric conversion module to physically isolate the 2×2 photoelectric converters under each asymmetric optical module to prevent light crosstalk from adjacent asymmetric optical modules.

[0145] The cell deep trench isolation structure may be made of materials such as silicon oxide, which is not limited here.

[0146] In some embodiments, the pixel is further configured to decompose the incident light into n calibration bands, eliminating internal noise interference, and analytically calibrate the incident light using pre-calibrated calibration parameters obtained from pre-calibration of the n calibration bands and n spatial signals to obtain the spectral components of the incident light in each calibration band. For more information about the spectral components of the incident light in each calibration band, please refer to the relevant description below.

[0147] In some embodiments, the pixel is further configured to: superimpose and sum the n spatial signals, excluding internal noise interference, to obtain a spatial superposition signal, and detect brightness changes of incident light based on the spatial superposition signal. For more information on detecting brightness changes of incident light, please refer to the relevant description below.

[0148] Secondly, an embodiment of the present application further provides an image sensor, the image sensor comprising a plurality of pixels arranged in an array;

[0149] Pixels include:

[0150] A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3;

[0151] An asymmetric optical module is disposed on the photoelectric conversion module, wherein the asymmetric optical module is used to focus and diffract the incident light to produce a three-dimensional asymmetric light intensity pattern, and then record the n different spatial signals on the n photoelectric converters in the photoelectric conversion module in a one-to-one correspondence;

[0152] The pixel is configured to determine the sources of the n spatial signals according to the spatial correlation between the n spatial signals in the photoelectric conversion module, so as to eliminate the interference of internal noise.

[0153] For example, Figure 8-Figure 9 As shown, the image sensor includes:

[0154] A photoelectric conversion array, comprising a plurality of photoelectric conversion modules arranged in an array;

[0155] An asymmetric optical array is arranged on a photoelectric conversion array, and the asymmetric optical array includes multiple asymmetric optical modules arranged in an array, each asymmetric optical module is arranged on a corresponding photoelectric conversion module, the value of n is 4, and each photoelectric conversion module includes 4 photoelectric converters arranged in a regular 2×2 array. The asymmetric optical module focuses and diffracts the incident light to generate a three-dimensional asymmetric light intensity pattern, which is recorded on the 2×2 photoelectric converters in the corresponding photoelectric conversion module to form 4 different spatial signals.

[0156] For example, Figure 8-Figure 9 As shown, the photoelectric conversion array includes a plurality of photoelectric converters arranged in a regular array, the asymmetric optical array includes a plurality of asymmetric optical modules arranged in a regular array, every 2×2 photoelectric converters constitute a photoelectric conversion module, and every 2×2 photoelectric converters and an asymmetric optical module thereon constitute a pixel (also referred to as a pixel unit). Thus, the image sensor includes a plurality of the above-mentioned pixels arranged in a regular array.

[0157] In each pixel, the incident light forms a three-dimensional asymmetric light intensity pattern based on the asymmetric optical module, and then forms four different spatial signals on the four photoelectric converters. Combined with the four monochrome calibration bands in the subsequent visible spectrum range, it can effectively detect the brightness and color of the incident light in each pixel, and resolve the incident light into the spectral components corresponding to the four calibration bands.

[0158] It should be noted that compared to traditional RGGB sensors, these image sensors no longer use color filters to remove multiple color components before transmitting the remaining single color band components to a photoelectric converter. Instead, they utilize an asymmetric optical module to transmit the full spectrum of color band components, resulting in a 2-3x improvement in photosensitivity within the visible spectrum. Furthermore, by detecting the brightness and color of incident light entirely based on pre-calibrated multi-color band components and measured spatial signals, they eliminate the need for interpolation calculations based on adjacent pixels, helping to improve the image sensor's spectral resolution.

[0159] It can be understood that since the pixels in the embodiments of the present application can combine asymmetric optical modules with multi-color band calibration technology to replace the microlenses and filters in existing pixels, the pixels in the embodiments of the present application are not limited by the structural dimensions of the microlenses and filters, especially not by the structural dimensions of the microlenses, and the corresponding pixel size can be adaptively reduced.

[0160] It should be noted that, in addition to the above-mentioned photoelectric conversion array and the above-mentioned asymmetric optical array, the image sensor also includes other structures such as a pixel processing circuit, a controller and an image processor. For details, please refer to the existing technology and will not be repeated here.

[0161] In some embodiments, the image sensor includes pixels of only one structure, that is, the structure of each pixel in the image sensor is exactly the same, and each pixel in the image sensor is arranged in a regular array.

[0162] For example, Figure 8As shown, the image sensor includes multiple pixels, and the structure of each pixel is the same, that is, the structure of the photoelectric conversion module in each pixel is the same, the structure of the asymmetric optical module in each pixel is also the same, and the pixels in the image sensor are arranged in a regular two-dimensional matrix.

[0163] In some embodiments, the image sensor includes at least two pixels with different structures. The at least two pixels have different structures, meaning that the number and / or size of photosensors included in the at least two pixels differ, and / or the asymmetric optical modules within the at least two pixels differ in structure. For example, the components within the different asymmetric optical modules may differ in at least one of shape, size, and material.

[0164] For example, in an image sensor, n has only one value, but the array arrangement structure of n photoelectric converters in at least two photoelectric conversion modules is different. For example, the number of photoelectric converters included in each pixel is the same, and the structure of the asymmetric optical module in each pixel is the same, but the structure of the photoelectric conversion modules in some pixels is different, that is, the arrangement structure of multiple photoelectric converters in some pixels is different, and there are differences in the size and arrangement of the corresponding photoelectric converters.

[0165] Exemplarily, in an image sensor, n has at least two different values, so that the array arrangement structure of n photoelectric converters in at least two photoelectric conversion modules is different. For example, the number of photoelectric converters included in at least two pixels is different, and the array arrangement structure of the multiple photoelectric converters in the two pixels is different. Even if the structure of the asymmetric optical module in each pixel is the same, under the premise that the size of the photoelectric conversion module is the same, due to the different number of photoelectric converters included in some pixels, the arrangement structure of each photoelectric converter in the photoelectric conversion module in the corresponding pixel must be different.

[0166] For example, in the image sensor, n has only one value, and the array arrangement structure of n photoelectric converters in each photoelectric conversion module is the same, but the asymmetric optical modules in at least two pixels are different, for example, Figure 8 The asymmetric optical module in some pixels shown is rotated 45° to the left in the vertical projection plane of the incident light. The focus diffraction orientation of the rotated asymmetric optical module is different from that of the unrotated asymmetric optical module. In this way, two different asymmetric optical modules are obtained, or the focus diffraction orientation can be changed directly based on the transformation of the shape and material of the internal components. Figure 8 The structure of the asymmetric optical module in a portion of the pixel is shown.

[0167] In some embodiments, as Figure 8As shown, the number n of photoelectric converters in a pixel is 4, a photoelectric conversion module composed of 2x2 photoelectric converters and an asymmetric optical module form a pixel, and 4 calibration color bands R, G1, G2 and B are uniformly selected in the visible spectrum range, and the spatial signal response corresponding to the single-color calibration color band is pre-calibrated, and then the incident light of unknown color can be resolved into 4 spectral components corresponding to the calibration color bands, which is suitable for an image sensor in which a single photoelectric converter is arranged in a square and multiple photoelectric converters are arranged in an array, such as Figure 8 As shown, the 4x6 photoelectric conversion array includes 24 photoelectric converters arranged in a 4x6 array, which can be configured as 2x3 pixels, and 2x2 photoelectric converters form an asymmetric optical module.

[0168] In addition, as shown in the 4x6 photoelectric conversion array Figure 8 As shown, the 4x6 photoelectric conversion array can also be configured as 1x2 pixels, and 3x3 photoelectric converters form an asymmetric optical module to form a pixel; as the photoelectric conversion array expands, an asymmetric optical module can be formed on 4x4, 5x5, …, NxN photoelectric converters to form a pixel.

[0169] Correspondingly, 3x3, 4x4, 5x5, …, NxN calibration color bands need to be selected in the visible spectrum range, and the calibration color bands at least cover red, green and blue primary colors to effectively meet the needs of expanding and resolving the incident light of unknown color and color detection. Wherein, N is an integer greater than or equal to 2.

[0170] It can be understood that the more uniform the color distribution of the calibration color bands and the more the number of the calibration color bands, the more detailed the color resolution of the incident light, the higher the resolution accuracy, and the more accurate the resolution result, but the corresponding calculation amount is large, which can be selected according to the actual situation.

[0171] Thirdly, the embodiment of the application also provides an operating method of an image sensor, applied to an image sensor composed of multiple pixels, the method comprising the steps of:

[0172] S1, acquiring an imaging mode of the image sensor, wherein the imaging mode includes a spectral imaging mode, an event imaging mode and a fusion imaging mode;

[0173] S2, based on the imaging mode, controlling the image sensor to collect corresponding incident light in units of pixels to obtain n different spatial signals corresponding to each pixel;

[0174] S3, judging the source of the spatial signal in units of pixels to exclude the interference of internal noise;

[0175] S4, detecting, in a pixel unit, based on the imaging mode, a color of the incident light according to the n spatial signals in the pixel, and / or a luminance variation of the incident light according to the n spatial signals in the pixel, under the condition that internal noise interference is excluded;

[0176] The pixel comprises:

[0177] The photoelectric conversion module comprises n photoelectric converters arranged adjacently in an array form, wherein n is a positive integer greater than or equal to 3.

[0178] The asymmetric optical module is arranged on the photoelectric conversion module, wherein the asymmetric optical module is used for focusing and diffracting the incident light to generate a three-dimensional asymmetric light intensity pattern, and then recording and forming n different spatial signals one by one on the n photoelectric converters in the photoelectric conversion module.

[0179] It should be noted that the spatial signal refers to a signal measured on each photoelectric converter in the photoelectric conversion module by the three-dimensional asymmetric light intensity pattern formed by the incident light of unknown color after asymmetric focusing and diffraction of the asymmetric optical module. Since the light intensity pattern is a three-dimensional asymmetric light intensity pattern, the light intensity pattern on the photosensitive plane of each photoelectric converter is different after the light intensity pattern is projected into the pixel. The spatial signal corresponding to the photoelectric response is different in size. Therefore, the spatial signal reflects the distribution characteristics of the incident light in space.

[0180] Since each pixel in the image sensor can be used to detect the color of the incident light and the luminance variation of the incident light, in step S1, the imaging mode of the image sensor corresponding to the plurality of pixels is flexible and optional. It can be a spectral imaging mode in which all working pixels detect the color of the incident light, an event imaging mode in which all working pixels detect the luminance variation of the incident light, or a fusion imaging mode in which part of the working pixels detect the luminance variation of the incident light and part of the working pixels detect the color of the incident light.

[0181] The working pixel refers to a pixel enabled to enter a working state. In various imaging modes of the image sensor, all pixels can be working pixels for image acquisition, or part of the pixels in a region can be working pixels for image acquisition. Details are not described herein.

[0182] It should be noted that for each pixel, when the detection purpose of the pixel is different, the working mode of the corresponding photoelectric converter is different: when the pixel detects the color of the incident light, the photoelectric converter works in the first mode, and the corresponding pixel circuit needs to enter the reset clear state, exposure integration state and readout state in sequence. During exposure integration, the photoelectric converter generates charge based on the incident light, and the corresponding spatial signal collected is the integral voltage; when the pixel detects the change in the brightness of the incident light, the photoelectric converter works in the second mode, and the corresponding pixel circuit has only one working state. The photoelectric converter generates current based on the incident light, and the corresponding spatial signal collected is a logarithmic voltage.

[0183] Therefore, in step S2, it is necessary to control the image sensor to collect the corresponding incident light in units of pixels based on the imaging mode, so that the photoelectric converter in the pixel that detects the color of the incident light operates in the first mode, and the photoelectric converter in the pixel that detects the brightness change of the incident light operates in the second mode, and each pixel collects corresponding n different spatial signals.

[0184] In some embodiments, step S3 of determining the source of the spatial signal in units of pixels to eliminate interference from internal noise further includes:

[0185] S31. For each pixel, obtain corresponding n different spatial signals;

[0186] S32. For each pixel, analyzing the spatial correlation between the n spatial signals according to the signal values ​​of the n spatial signals;

[0187] S33. For each pixel, determine the sources of n spatial signals according to spatial correlation;

[0188] S34. For each pixel, if the n spatial signals corresponding to the pixel are derived from internal noise, remove the n spatial signals corresponding to the pixel, and / or perform compensation correction on the n spatial signals corresponding to the pixel to eliminate interference from the internal noise.

[0189] In a photoelectric conversion module, the distribution of spatial signals is affected by a variety of factors, such as the characteristics of the light source, the characteristics of the optical components, the characteristics of the photoelectric converter, and noise. By analyzing the spatial correlation between the spatial signals of multiple photoelectric converters, the source of the spatial signal can be inferred.

[0190] Spatial correlation between spatial signals refers to the relationship between the values, temporal characteristics, or changing trends of the spatial signals output by different photoelectric converters. For example, the values ​​of the spatial signals from different photoelectric converters may exhibit a certain proportional or linear relationship. Another example is that when the intensity or wavelength of the incident light changes, the spatial signals from different photoelectric converters exhibit similar changing trends.

[0191] In some embodiments, correlation coefficients (eg, cross-correlation functions, etc.) of multiple spatial signals within a pixel may be calculated to determine spatial correlations between the multiple spatial signals.

[0192] In some embodiments, a matrix composed of n spatial signals within a pixel is normalized, and a covariance matrix of the matrix is ​​calculated. By analyzing the variance contribution rate and time-frequency characteristics of the principal components, it is determined whether the multiple spatial signals have noise.

[0193] In some embodiments, the sources of the spatial signal may include external incident light, random noise inside the sensor, and fixed noise inside the sensor. The random noise inside the sensor may be dark current noise, readout noise, and shot noise of the photoelectric converter, etc. The fixed noise inside the sensor may be noise caused by non-ideal characteristics of optical elements such as lens distortion and non-uniformity of the reflector. The fixed noise inside the sensor may also be noise caused by size differences of the photoelectric converter, deviations in transistor characteristics, or inconsistent amplifier gain.

[0194] Among them, dark current noise refers to the weak current generated by the photoelectric converter in the absence of light.

[0195] It should be noted that general pixels and image sensors focus more on detecting the color of incident light, while Figure 4 The light collected by traditional image sensors for distant objects is low light, even if it is replaced by Figure 8 The image sensor based on the asymmetric optical module shown above effectively reduces the loss of imaging light after eliminating light absorption. However, the light collected from distant objects is still relatively low. The spatial signal based on a single photoelectric converter or the superposition of spatial signals from a few photoelectric converters still cannot effectively eliminate interference from background noise or internal noise of the sensor, which is very likely to cause false detection of signals.

[0196] At this time, considering that the n photoelectric converters in the pixel and image sensor of the above embodiment share the same asymmetric optical module, the spatial signals of the n photoelectric converters in the pixel are formed by the diffraction and self-interference of the same imaging light, so that the spatial signals detected by the n photoelectric converters are highly correlated, such as Figure 10As shown, if the spatial signal comes from external incident light (i.e., an external light source), then when the light originates from the same light source, after diffraction and self-interference, a pattern with a specific light intensity distribution is formed in space. That is, interrelated spatial signals are generated on the photoelectric converters at different positions in space, and each spatial signal reflects the light intensity change at that position. If the spatial signal comes from the internal noise of the sensor (i.e., an internal dark source, such as dark current noise, readout noise, etc.), then there is no corresponding light signal for each spatial signal, and there is no diffraction and self-interference processing by the asymmetric optical module. Therefore, the corresponding n spatial signals do not have the high spatial correlation constrained by the asymmetric optical module, but instead exhibit low spatial correlation based on the fixed noise inside the sensor, or have no spatial correlation at all based on the random noise inside the sensor.

[0197] Therefore, in some embodiments, for each pixel, it is possible to determine whether the collected spatial signal comes from an external light source or sensor noise based on the spatial correlation between the spatial signals on n photoelectric converters, so as to eliminate sensor noise interference and improve the accuracy of signal detection.

[0198] The spatial correlation between the spatial signals on the n photoelectric converters within a pixel can be determined by the signal values ​​of the n spatial signals. For example, if the spatial signal comes from an external light source, there is a high spatial correlation between the n spatial signals, and the corresponding n signal values ​​are generally non-zero values ​​of varying sizes. If the spatial signal is caused by sensor noise, due to the randomness of the noise or a partially fixed pattern (such as streak noise), there is a low spatial correlation or no correlation at all between the n spatial signals, and some of the corresponding n signal values ​​are zero (no external light source and no internal noise), some are non-zero (no external light source but random noise), or some of the corresponding n signal values ​​have non-zero signal values ​​of the same size distributed in rows, columns, or blocks (no external light source but fixed noise).

[0199] It should be noted that, in the presence of internal sensor noise, the signal values ​​of the n spatial signals are generally not all non-zero.

[0200] For example, in one embodiment of the present application, Figure 11As shown, the four photoelectric converters in the pixel share the same asymmetric optical module and are arranged in a 2x2 array: when there is no external light source, due to the local random characteristics of dark current, the dark current only acts in one photoelectric converter, and only three of the corresponding four spatial signals are zero, and the other is not zero, i.e. S1=S3=S4=0, S2>0; when there is an external light source of 610 nm, through the diffraction and self-interference of light, an asymmetric spatial signal pattern is formed on the 2x2 photoelectric converter, and four spatial signals that are mutually related and all not zero are obtained, i.e. S1>0, S2>0, S3>0, S4>0.

[0201] In some embodiments, the spatial correlation includes a first correlation, a second correlation and a third correlation, and the step S32 of analyzing the spatial correlation between the n spatial signals according to the signal values of the n spatial signals for each pixel further includes:

[0202] S321, for each pixel, constructing a spatial signal matrix based on the signal values of the n spatial signals;

[0203] S322, for each pixel, analyzing the spatial correlation between the n spatial signals based on the numerical values and distribution rules of the signal values in the spatial signal matrix;

[0204] Wherein, in the case that the n signal values in the spatial signal matrix are all different and all non-zero values, the n spatial signals present the first correlation;

[0205] In the case that at least one row or one column of signal values in the spatial signal matrix is the same non-zero value, the n spatial signals present the second correlation;

[0206] In the case that the n signal values in the spatial signal matrix are partly zero and partly non-zero values and the non-zero values are randomly distributed, the n spatial signals present the third correlation.

[0207] It should be noted that in step S321, the spatial signal matrix refers to a matrix constructed based on the signal values of the n spatial signals in the pixel, and each element in the spatial signal matrix represents the signal value of the spatial signal at the corresponding specific position.

[0208] At the same time, in step S322, the first correlation is a higher spatial correlation, which means that the n signal values ​​are different non-zero values, there are no repeated or zero values, and the corresponding source is external incident light; the second correlation is a lower spatial correlation, which means that the signal values ​​of at least one column or row are repeated non-zero values, and the signal values ​​of the same or basically the same size are concentrated in rows, columns or blocks, and the corresponding source is the fixed noise inside the sensor; the third correlation is no spatial correlation at all, which means that there are zero values ​​and non-zero values ​​in multiple signal values, and the non-zero values ​​are randomly distributed without rules, and the corresponding source is the random noise inside the sensor.

[0209] In some embodiments, for each pixel, step S33 of determining the sources of n spatial signals according to spatial correlation further includes:

[0210] S331. When a first correlation is present between the n spatial signals, determine that the n spatial signals corresponding to the pixel originate from incident light;

[0211] S332: When a second correlation is present between the n spatial signals, determine that the n spatial signals corresponding to the pixel are derived from internal noise, and the internal noise is fixed noise such as row stripe noise, column stripe noise, or regional noise;

[0212] S333: When a third correlation is present between the n spatial signals, determine that the n spatial signals corresponding to the pixel are derived from internal noise, and the internal noise is random noise.

[0213] For example, in one embodiment of the present application, Figure 12 As shown, when pixel-level sensing is performed based on a 2×2 photoelectric converter sharing the same asymmetric optical module: if the signal value of at least one spatial signal is zero (one spatial signal is zero, two spatial signals are zero, or three spatial signals are zero), it can be determined that the spatial signal within the pixel is caused by the internal noise of the sensor, such as the dark current signal, and there is no external incident light event; if the signal values ​​of the four spatial signals are all non-zero and different, it can be determined that the source of the spatial signal within the pixel is an external light source, and there is an external incident light event.

[0214] Among them, the signal values ​​of the four spatial signals are all zero, which defaults to no external light source incident and no sensor internal noise interference.

[0215] In some embodiments, for a 2×2 photoelectric converter sharing the same asymmetric optical module within a pixel, after obtaining a 2×2 spatial signal matrix corresponding to four spatial signals, the correlation between the spatial signals can be directly judged based on the determinant value of the 2×2 spatial signal matrix to determine the source of the spatial signal, such as from an external light source or from internal noise of the sensor.

[0216] For example, Figure 13 As shown in the figure, when the elements of the 2×2 spatial signal matrix in the pixel are not all zero, its determinant is determined to be zero to determine the source of the spatial signal: if the determinant is zero, that is, three elements are zero, two elements in a row are zero, two elements in a column are zero, two elements in each row are equal, two elements in each column are equal, or four elements are completely equal, then the source of the spatial signal is the internal noise of the sensor (which can be recorded as a dark signal), and there is no corresponding inverse matrix and no related incident spectrum; if the determinant is not zero, that is, all four elements are not zero and the sizes are different, two elements on one diagonal are zero and the other diagonal is zero, then the source of the spatial signal is the internal noise of the sensor (which can be recorded as a dark signal), and there is no corresponding inverse matrix and no related incident spectrum; if the determinant is not zero, that is, all four elements are non-zero and the sizes are different, two elements on one diagonal are zero and two elements on the other diagonal are zero. If two elements on the matrix are not zero, or one element is zero and the other three elements are not zero, then it is necessary to further determine whether there is a zero element in the 2×2 spatial signal matrix. If not, the source of the spatial signal is an external light source (which can be recorded as a light signal), and there is a corresponding inverse matrix and a related incident spectrum. If so, the source of the spatial signal is the internal noise of the sensor (dark signal). When it is identified that the source of the spatial signal in the pixel is an external light source, event sensing can be further performed based on the corresponding four spatial signals, and the incident light can be parsed into spectral components of four calibration color bands and converted into a standard red, green, and blue color space for display.

[0217] It should be noted that when the pixel senses and collects external incident light, due to the asymmetric focusing, diffraction and self-interference of the incident light by the asymmetric optical module, there is a high spatial correlation between the corresponding four spatial signals. The signal values ​​of the four spatial signals are non-zero values ​​of different sizes, and the determinant of the corresponding 2×2 spatial signal matrix is ​​generally not zero.

[0218] For example, in one embodiment of the present application, Figure 14 As shown in , when the pixel senses and collects red light in the 610nm band, the determinant value of the corresponding 2×2 spatial signal matrix is ​​16.7×14.8-12.7×12.7=86, and its determinant value is non-zero; Figure 15 As shown in , when the pixel senses and collects green light in the 550nm band, the determinant value of the corresponding 2×2 spatial signal matrix is ​​20.2×17.5-14×16=130, and its determinant value is non-zero; Figure 16 As shown in , when the pixel senses and collects green light in the 490nm band, the determinant value of the corresponding 2×2 spatial signal matrix is ​​19.8×17.3-15×15.4=112, and its determinant value is non-zero; Figure 17 As shown, when the pixel senses and collects blue light in the 430nm band, the determinant value of the corresponding 2×2 spatial signal matrix is ​​18.1×19.9-16.9×14.4=102, and its determinant value is non-zero.

[0219] For example, in the embodiments of the present application, Figure 18 As shown, a 2×2 spatial signal matrix corresponding to four spatial signals is formed, and the source of the spatial signal is determined based on the determinant value of the spatial signal matrix, such as from an external light source or from sensor noise. There are at least the following situations:

[0220] Case A: Only one spatial signal is non-zero, and the other three spatial signals are all zero. The corresponding determinant is zero. In this case, the spatial signal is caused by the internal noise of the sensor (i.e., dark signal), and further caused by random noise inside the sensor.

[0221] Case B: The four spatial signals are all non-zero and different, and the corresponding determinants are non-zero, then the spatial signals are caused by external light sources (i.e., optical signals);

[0222] Case C: The two spatial signals in the first column of the spatial signal matrix are both zero, and the two spatial signals in the second column are both non-zero, and the corresponding determinant is zero. In this case, the spatial signal is caused by the internal noise of the sensor. At this time, the values ​​of the two spatial signals in the second column can be further compared. If the two values ​​are the same, the spatial signal is caused by the column fringe noise inside the sensor. If the two values ​​are different, the spatial signal is caused by the random noise inside the sensor.

[0223] In case D, the two spatial signals in the first row of the spatial signal matrix are both zero, and the two spatial signals in the second row are both non-zero, and the corresponding determinant is zero. In this case, the spatial signal is caused by the internal noise of the sensor. At this time, the values ​​of the two spatial signals in the second row can be further compared. If the two values ​​are the same, the spatial signal is caused by the row stripe noise inside the sensor. If the two values ​​are different, the spatial signal is caused by random noise inside the sensor.

[0224] It should be noted that there are other situations corresponding to the 2×2 spatial signal matrix.

[0225] For example, Figure 19 As shown in , when all four spatial signals are not zero but some of the spatial signals are equal, the determinant may also be zero. In this case, it can be determined that the spatial signal is caused by the internal noise of the sensor, and the type of noise signal can be further determined by the signal value distribution of the spatial signal. Figure 19 As shown in 2200a, the two spatial signals on each column are equal and the magnitudes of the signals in each column are not equal, and the corresponding determinant is zero, then the spatial signal is caused by column stripe noise; Figure 19 As shown in 2200b, the two spatial signals on each row are equal and the two row signals are not equal in size, and the corresponding determinant is zero, then the spatial signal is caused by row stripe noise.

[0226] For example, in Figure 19 Based on the above, there is a special case where the four spatial signals are all the same size, so that the determinant is also zero. In this case, it can be determined that the spatial signal is caused by the internal noise of the sensor and is caused by block noise.

[0227] In addition, for the case where the determinant of the 2×2 spatial signal matrix is ​​not zero, in addition to case B, there are other cases, such as two spatial signals on one diagonal are zero and two spatial signals on the other diagonal are not zero, or one spatial signal is zero and the other three spatial signals are not zero (such as Figure 12 As shown in Case A of Figure 2, it can be determined that the spatial signal is caused by the internal noise of the sensor. At this point, the relative sizes of the non-zero spatial signals can be further compared. If the sizes of the non-zero spatial signals are equal, the spatial signal is most likely caused by some fixed noise. If the sizes of the non-zero spatial signals vary, the spatial signal is most likely caused by some random noise.

[0228] From the above analysis, it can be seen that in the present application, a 2×2 spatial signal matrix composed of 4 spatial signals can be used to filter noise interference on the spatial signals collected by the pixels based on the determinant value of the spatial signal matrix. For example, when the 4 spatial signals are not all zero, if the value of the determinant is zero, the source of the spatial signal is the internal noise of the image sensor. Furthermore, the type of noise can be determined based on the distribution of the signal values ​​of the 4 spatial signals: if the position and size of the non-zero spatial signal are randomly and irregularly distributed, it is determined that random noise exists; if the two spatial signals on a certain column in the spatial signal matrix are equal, then there is column stripe noise; if the two spatial signals on a certain row in the spatial signal matrix are equal, then there is row stripe noise; if the three spatial signals or the four spatial signals in the spatial signal matrix are equal, then there is block noise or regional noise.

[0229] In addition, when all four spatial signals are non-zero, if the value of the determinant is non-zero, it is necessary to determine whether there is a zero element in the 2×2 spatial signal matrix; if there is no zero element, the source of the spatial signal is an external light source, and there is a corresponding inverse matrix and a related incident spectrum; if there is a zero element, the source of the spatial signal is the internal noise of the sensor, and the noise type can be further determined based on the relative size of each non-zero spatial signal, whether it is random noise or fixed noise.

[0230] It can be understood that the above embodiment only illustrates the source identification process of the spatial signal in a pixel with an n value of 4, so as to eliminate the internal noise interference of the image sensor. For other pixels with other values ​​of n and different matrix distribution forms of the n photoelectric converters in the photoelectric conversion module, noise interference analysis can be performed similar to the above-mentioned 2×2 spatial signal matrix. However, it may not be possible to directly analyze and judge based on the value of the determinant, especially for pixels with n photoelectric converters in the photoelectric conversion module that are distributed in a matrix with unequal numbers of rows and columns or an irregular matrix distribution. For this reason, there is no determinant, but it is necessary to further analyze the signal values ​​of each spatial signal. When the spatial signals are not all zero, the spatial distribution pattern and relative size of the non-zero spatial signal are analyzed. For example, nine photoelectric converters within a pixel form a 3×3 spatial signal matrix, or eight photoelectric converters within a pixel form a 2×4 spatial signal matrix. If some values ​​in the multiple spatial signals are zero and some values ​​are non-zero, the source of the spatial signal is the internal noise of the image sensor. If the spatial signals on a column in the multiple spatial signals are equal, column stripe noise exists. If the spatial signals on a row in the multiple spatial signals are equal, row stripe noise exists. Similar analysis can be performed with reference to the 2×2 spatial signal matrix, and will not be repeated here.

[0231] It should be noted that when the source of the spatial signal within the pixel is identified through steps S31 to S33, when it is identified that the n spatial signals corresponding to the pixel are derived from the internal noise of the image sensor, in step S34, the n spatial signals corresponding to the pixel can be removed and the pixel can be ignored, or the n spatial signals corresponding to the pixel can be compensated and corrected, and the pixel can be calibrated based on operations such as neighborhood interpolation calculation to effectively eliminate the interference of internal noise.

[0232] In some embodiments, after eliminating internal noise interference, step S4 is executed, and detection is performed in pixels based on the imaging mode. Within each pixel, the color of the incident light can be detected based on the n spatial signals within the pixel, and the brightness change of the incident light can be detected based on the n spatial signals within the pixel.

[0233] In some embodiments, detecting the color of incident light based on n spatial signals within a pixel includes:

[0234] S41. For each pixel, select n calibration color bands to decompose the incident light;

[0235] S42, performing pre-calibration on each pixel based on n calibration color bands to obtain pre-calibration parameters;

[0236] S43. For each pixel, obtain n different spatial signals obtained by the pixel performing photoelectric collection of incident light;

[0237] S44. For each pixel, analytically calibrate the incident light using pre-calibrated calibration parameters and n spatial signals, and calculate the spectral components of the incident light in each calibration color band.

[0238] Within each pixel, the spatial signal response refers to the photoelectric effect of each photoelectric converter in response to the asymmetric spatial distribution of light intensity created by the asymmetric focusing and diffraction of the incident calibration ribbon by the asymmetric optical module, generating a corresponding electrical signal. In other words, the multiple spatial signal responses within each pixel can describe the spatial distribution of the incident light on the photoelectric converter module.

[0239] For example, the spatial signal response can be obtained by irradiating a known calibration color band (such as a red spectral color band, a green spectral color band, a blue spectral color band, etc.) onto the photoelectric conversion module and recording the electrical signal output by each photoelectric converter.

[0240] The spatial signal response can be represented as a vector or matrix, where each element corresponds to the electrical signal strength of a photoelectric converter. For example, for a 2×2 photoelectric converter array, the spatial signal response can be represented as a 4×1 vector.

[0241] It will be appreciated that the spectral components may include the contribution of each calibration color band to the total incident light intensity, such as the contribution of the 600nm red spectral band is 20%, the contribution of the 525nm green spectral band is 30%, and the contribution of the 450nm blue spectral band is 50%.

[0242] In some embodiments, the n calibration color bands selected in step S41 include at least a red spectral color band, a green spectral color band, and a blue spectral color band to meet the requirements of unfolding and decomposing incident light of unknown color.

[0243] Among them, the red spectral band is used to calibrate and analyze the red component in the incident light, the green spectral band is used to calibrate and analyze the green component in the incident light, and the blue spectral band is used to calibrate and analyze the blue component in the incident light.

[0244] Unfolding decomposition is the process of breaking down incident light of unknown color into its wavelength components. By using the red, green, and blue spectral bands as calibration bands, the unknown color light can be decomposed into its three basic components: the red, green, and blue spectral bands, allowing its spectral characteristics to be accurately described.

[0245] To ensure accurate decomposition and analysis of incident light of unknown color, the calibration color band selected in step S41 must cover the spectral range of the unknown color. The red, green, and blue spectral bands are commonly used reference color bands in spectral analysis. They cover most of the visible light range and can effectively decompose and describe the spectral characteristics of incident light of unknown color.

[0246] It should be noted that in step S41, when selecting the calibration color band, the more different color bands the selected calibration color band covers, the more detailed and accurate the decomposition of the incident light by the multiple calibration color bands will be. Therefore, the selected calibration color band can include more spectral color bands of other colors, such as yellow spectral color band, purple spectral color band, etc. on the basis of covering the red spectral color band, green spectral color band and blue spectral color band. It is also possible to select multiple spectral color bands of the same color system with similar colors but different wavelengths, such as a green spectral color band with a wavelength of 490nm and a green spectral color band with a wavelength of 550nm. However, the corresponding calibration calculation amount is large and needs to be compromised.

[0247] In step S42, the pre-calibrated calibration parameters refer to parameters obtained through a series of pre-calibration steps for decomposing the incident light, which can be obtained by combining the calculation results of the monochrome spatial signal responses of each calibration color band.

[0248] In some embodiments, for each pixel, pre-calibration is performed based on n calibration strips to obtain pre-calibration parameters S42, further comprising:

[0249] S421. For each pixel, construct a system of linear equations of n variables based on spatial signal responses of incident light decomposed by the n calibration color bands on n photoelectric converters within the pixel, where the system of linear equations of n variables is composed of a calibration coefficient matrix, a color band fraction vector, and a spatial signal vector.

[0250] S422. For each pixel, traverse n calibration color bands, simplify and calculate a system of n-variable linear equations based on the spatial signal response of a single calibration color band to obtain a calibration coefficient vector for the single calibration color band, and then combine the calibration coefficient vectors of the n calibration color bands to obtain a calibration coefficient matrix;

[0251] S423 . For each pixel, perform an inverse operation on the calibration coefficient matrix to obtain an inverse matrix of the calibration coefficient matrix. The inverse matrix of the calibration coefficient matrix is ​​the pre-calibration calibration parameter.

[0252] In step S421, the calibration coefficient matrix represents the calibration coefficients of the spatial signals measured by each calibration color band on each photoelectric converter. Specifically, the plurality of elements of each row of the calibration coefficient matrix represent the calibration coefficients of the spatial signals of a respective calibration color band on a certain photoelectric converter, and the plurality of elements of each column of the calibration coefficient matrix represent the calibration coefficients of the spatial signals of a certain calibration color band on each photoelectric converter, that is, each element a ij in the calibration coefficient matrix represents the calibration coefficient of the ith spatial signal under the jth calibration color band.

[0253] For example, in the embodiments of the present application, for the asymmetric optical module as shown in Figure 8-Figure 9 , when the value of n is 4, and the narrow spectral color bands with wavelengths of 430 nm, 490 nm, 550 nm and 610 nm in the visible spectral range are selected as the calibration color bands, the asymmetric optical module can effectively asymmetrically focus and diffract the incident light of the above-mentioned four calibration color bands (B = 430 nm, G2 = 490 nm, G1 = 550 nm and R = 610 nm), as shown in Figure 20 , and thus the asymmetric optical module can effectively asymmetrically diffract various incident light based on the above-mentioned four calibration color bands, and form different spatial signals on the 2x2 photoelectric converters.

[0254] In this way, based on the different spatial signal responses of the four photoelectric converters in a pixel, four representative calibration color bands in the visible spectral range can be selected, a four-element linear equation set is constructed, and the calibration coefficient matrix corresponding to the four-element linear equation set is determined based on the spatial signal responses of a single calibration color band. Finally, the color band components of the incident light are solved based on the calibration coefficient matrix and the four spatial signals measured in real time, that is, based on the 2x2 spatial signals, the incident light is analyzed into 2x2 spectral components corresponding to the 2x2 calibration color bands on the basis of the expansion and decomposition of the incident light based on the above-mentioned four calibration color bands.

[0255] For example, in the embodiments of the present application, for the pixel or image sensor as shown in Figure 8-Figure 9 , each pixel responds to the spatial signals of the incident light expanded and decomposed based on the above-mentioned four calibration color bands, and the following four-element linear equation set is constructed.

[0256] ;

[0257] ;

[0258] ;

[0259] ;

[0260] wherein, represents a calibration coefficient of the spatial signal Mij measured on the photoelectric converter PDij for the calibration color band bk component of the incident light, represents a contribution factor of the calibration color band bk component of the incident light to the spatial signal Mij, and Mij represents a spatial signal measured on the photoelectric converter PDij for the incident light of unknown color; i, j are integers from 1 to 2, and k is an integer from 1 to 4.

[0261] For example, in the embodiments of the present application, in the visible spectral range of 400-640 nm, as shown in Figure 20-21 a blue light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 430 nm is used as the calibration color band Band 1 (b1 for short), a green light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 490 nm is used as the calibration color band Band 2 (b2 for short), a green light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 550 nm is used as the calibration color band Band 3 (b3 for short), and a red light with a full width at half maximum (FWHM) of 60 nm and a wavelength of 610 nm is used as the calibration color band Band 4 (b4 for short).

[0262] It can be understood that the above four-element linear equation set can be written as follows.

[0263] ;

[0264] wherein C is a 4x4 calibration coefficient matrix, X is a color band fraction vector of the incident light of unknown color that contributes to the measured spatial signal, and M is a spatial signal vector measured for the incident light of unknown color;

[0265] , , .

[0266] The above matrix equation is multiplied by the inverse matrix of C on both sides to obtain the following relationship, which is the solution of the linear equation set:

[0267] ;

[0268] It can be seen from the above matrix equation that after the pixel structure and the selected four calibration color bands are determined, the calibration vector of the pixel structure for each calibration color band is determined accordingly, thereby making the calibration coefficient matrix unique. In this way, the spatial signal response of the pixel structure to the incident light of unknown color can be understood as "encoding the incident light expanded and decomposed according to the four calibration color bands based on the calibration coefficient matrix to obtain four different spatial signals"; at the same time, it can be seen from the above matrix equation that only one of the four calibration color bands can be used each time to simplify the spatial signal response or simplify the encoding, that is, let one element of X be 1 and the other three elements be 0, so as to achieve the simplification of the matrix equation, and then based on the corresponding measured spatial signal, some elements or vectors in the calibration coefficient matrix C can be obtained, and then the calibration coefficient elements or calibration coefficient vectors solved by combining the spatial signal responses corresponding to the four calibration color bands can be obtained to obtain the complete calibration coefficient matrix C, and the inverse matrix C of the calibration coefficient matrix C can be solved. -1 .

[0269] Finally, from the above matrix equation, we can know that for the incident light X of unknown color, the inverse matrix C of the known calibration coefficient matrix C is -1 Based on the spatial signal vector M corresponding to the incident light X, the inverse matrix C of the calibration coefficient matrix C can be directly calculated. -1 The incident light X is calculated by analyzing the measured spatial signal vector M, and the incident light X is analyzed into the spectral components corresponding to the above four calibration color bands. While detecting the brightness of the incident light, the color of the incident light is measured. This process can be understood as "the inverse operation parameters based on the encoding parameters (the inverse matrix C of the calibration coefficient matrix C) -1 ) is decoded with the encoded result (4 different spatial signals) to obtain the unfolding decomposition result of the incident light under 4 calibration color bands."

[0270] For example, Figure 14 As shown, in the embodiment of the present application, , only red light with a full width at half maximum (FWHM) of 60nm and a wavelength of 610nm is used as the calibration band Band 4 for pre-calibration of the monochrome band. The corresponding four-variable linear equation system is simplified to:

[0271] ;

[0272] ;

[0273] ;

[0274] ;

[0275] At the same time, according to the actual measured value of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are: QE PD11 = 16.7, QE PD12 = 12.7, QE PD21 = 12.7, and QE PD22 = 14.8.

[0276] For example, Figure 15 As shown, in the embodiment of the present application, , only green light with a full width at half maximum (FWHM) of 60nm and a wavelength of 550nm is used as the calibration band Band 3 for pre-calibration of the monochrome band. The corresponding four-variable linear equation system is simplified to:

[0277] ;

[0278] ;

[0279] ;

[0280] ;

[0281] At the same time, according to the actual measured value of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are: QE PD11 = 20.2, QE PD12 = 14, QE PD21 = 16, and QE PD22 = 17.5.

[0282] For example, Figure 16 As shown, in the embodiment of the present application, , only the green light with a full width at half maximum (FWHM) of 60nm and a wavelength of 490nm is used as the calibration band Band 2 for the monochrome band pre-calibration simplification. The corresponding four-variable linear equation system is simplified to:

[0283] ;

[0284] ;

[0285] ;

[0286] ;

[0287] At the same time, according to the actual measured value of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are: QE PD11 = 19.8, QE PD12 = 15, QE PD21 = 15.4, and QE PD22 = 17.3.

[0288] For example, Figure 17 As shown, in the embodiment of the present application, , only the blue light with a full width at half maximum (FWHM) of 60nm and a wavelength of 430nm is used as the calibration band Band 1 for the monochrome band pre-calibration simplification, then the corresponding four-variable linear equation system is simplified to:

[0289] ;

[0290] ;

[0291] ;

[0292] ;

[0293] At the same time, according to the actual measured value of the spatial signal vector M, the quantum efficiencies of the corresponding four photoelectric converters are: QE PD11 = 18.1, QE PD12 = 16.9, QE PD21 = 14.4, and QE PD22 = 19.9.

[0294] Thus, by combining the pre-calibration simplification of the above four single-color calibration strips, the corresponding calibration coefficient matrix C is obtained as shown in Table 1 below, where the four calibration coefficients in each column correspond to the pre-calibration simplification results of one calibration strip.

[0295] Table 1

[0296]

[0297] At the same time, for calculation considerations, the four calibration coefficients corresponding to the pre-calibration simplified result of a single calibration ribbon are normalized to obtain the calibration coefficient matrix C shown in Table 2 below.

[0298] Table 2

[0299]

[0300] For example, the calibration coefficient matrix C shown in the table above is inverted to obtain the inverse matrix C of the calibration coefficient matrix C. -1 As shown in Table 3 below.

[0301] Table 3

[0302]

[0303] In this way, the corresponding pixels or image sensors are pre-calibrated or pre-calibrated, and then the incident light of unknown color is detected based on the Formula, can be directly based on the inverse matrix C of the calibration coefficient matrix C -1 The incident light X is calculated by the measured spatial signal vector M and the incident light X is parsed into spectral components corresponding to the four calibration bands, namely , while realizing the brightness detection of incident light, it also realizes the color measurement of incident light.

[0304] In the embodiment of the present application, Figure 20 It can be seen that there are differences in the asymmetric distribution patterns of the spatial signals of incident light of different colors on the four photoelectric converters. The asymmetric spatial patterns of the spatial signals recorded on the four photoelectric converters can be regarded as the fingerprints of the incident spectrum / color. In this way, the spectral components corresponding to the four calibration color bands can be analyzed and decoded by writing a set of linear equations connecting the spatial signals of the photoelectric converters and the spectral components of the measured incident light signals. In order to decode the four measured spatial signals, the pixels of an asymmetric optical module shared by multiple photoelectric converters can be pre-calibrated by sequentially irradiating the pixels with calibration color bands of different bands, and recording the corresponding spatial signal responses to obtain a calibration coefficient matrix. The inverse matrix of the calibration coefficient matrix is ​​then used to calculate the incident light of unknown color and solve the linear equation system, where each element of the calibration coefficient matrix is ​​used as the coefficient of the linear equation system connecting the four spatial measurement signals to the four calibration color bands.

[0305] Therefore, in the embodiment of the present application, the combination of asymmetric optical modules and multi-color band calibration can effectively replace the microlenses and filters in existing pixels, and realize incident light brightness detection and color detection. The process is as follows: Figure 22 As shown, after the incident light X is parsed into spectral components corresponding to the four calibration color bands based on the above process, each spectral component is multiplied by the total measurement signal in the four photoelectric converters to convert the signal into the least significant bit (LSB) of the measurement value. Finally, based on the LSB, the spectral components corresponding to the four calibration color bands are converted into the standard red, green, and blue color space for display.

[0306] For example, in order to convert the measured spectral components into standardized values, each spectral component needs to be normalized, that is, each spectral component is divided by the total measurement signal of the four photoelectric converters to eliminate the influence of the light source intensity change and ensure the relative proportion of the measured values ​​is consistent.

[0307] It should be understood that the above embodiment merely illustrates the process of expanding and analyzing incident light of unknown color within four pixels when n is set to 4. By selecting four calibration bands to expand and decompose the incident light, the incident light is analyzed based on the pre-calibrated results of the four calibration bands and the four spatial signals actually measured from the incident light, thereby analyzing the incident light into spectral components corresponding to the four calibration bands, thereby achieving color detection of the incident light. The color detection process for incident light with other numbers of pixels, such as n, can be similarly analyzed and will not be further described here.

[0308] In some embodiments, detecting a brightness change of incident light according to n spatial signals within a pixel includes:

[0309] S401, superimposing and summing n spatial signals to obtain a spatial superposition signal;

[0310] S402: Detect brightness changes of incident light based on the spatial superposition signal.

[0311] Among them, the spatial signal is the electrical signal measured by each photoelectric converter, reflecting the intensity of the incident light at that position; the n spatial signals are superimposed and summed, that is, the signal values ​​measured by each photoelectric converter in the pixel are added to obtain a total spatial superposition signal.

[0312] It should be noted that by combining the spatial signals from n photoelectric converters within a pixel that share the same asymmetric optical module, a spatial superposition signal is generated. This spatial superposition signal eliminates the sensor's internal noise interference and is formed by superimposing multiple spatial signals. This can improve the dynamic range of the photoelectric conversion module, more effectively distinguishing the intensity of external light sources and discerning bright and dark signals, enabling more accurate measurement of brightness changes from low light to strong light. When used to monitor changes in incident light intensity in real time, it can significantly improve the ability to perceive events based on brightness or light intensity changes of distant objects. In addition, superimposing multiple spatial signals can average out the effects of random noise, thereby improving the signal-to-noise ratio.

[0313] In some embodiments, when the signal value of the spatial superposition signal increases, it indicates that the brightness of the incident light increases; when the signal value of the spatial superposition signal decreases, it indicates that the brightness of the incident light decreases.

[0314] For example, in some embodiments, Figure 23 As shown, the operating method of the above-mentioned image sensor includes the steps of:

[0315] Step S2301: determining an imaging mode of the image sensor, where the imaging mode includes a spectral imaging mode and an event imaging mode;

[0316] Step S2302: Perform imaging processing based on the imaging mode of the image sensor, and at least perform noise analysis on the spatial signal in the event imaging mode.

[0317] For example, Figure 23 As shown, if the imaging mode is the spectral imaging mode, then in step S2302, the incident light can be directly parsed into spectral components of multiple calibration color bands based on the incident light color detection process shown in steps S41 to S44. For details, please refer to the previous description and will not be repeated here.

[0318] For example, Figure 23As shown, if the imaging mode is the event imaging mode, then in step S2302, it is necessary to first perform noise identification on the collected spatial signals based on the judgment and identification process of the spatial signal source shown in steps S31 to S34, and perform signal filtering or signal correction processing and other operations on the basis of noise identification to obtain the processed spatial signal, and then merge and superimpose the processed spatial signals in the pixel to obtain the spatial superposition signal, and finally perform subsequent image processing based on the spatial superposition signal. In this way, a spatial superposition signal with the internal noise interference of the sensor eliminated and the intensity superposition performed can be obtained, which can more effectively distinguish the intensity of the external light source and distinguish bright signals from dark signals, so as to improve the event sensing capability of distant objects.

[0319] For example, Figure 23 As shown, the noise identification of the spatial signal in the event imaging mode can also be fed back to the spectral imaging mode. Based on the noise identification, signal filtering or signal correction processing is performed to obtain a processed spatial signal. Based on the processed spatial signal, the incident light is then parsed into spectral components of multiple calibration color bands, eliminating the internal noise interference of the sensor and effectively improving the sensing accuracy of spectral imaging.

[0320] Therefore, an embodiment of the present application further provides an imaging system, the imaging system comprising:

[0321] Imaging lens, used to converge the light of the object into an image to form incident light;

[0322] An image sensor configured to focus and diffract incident light based on pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, thereby correspondingly recording n different spatial signals on n photoelectric converters in the pixels; wherein n is a positive integer greater than or equal to 3;

[0323] Where pixels are configured as:

[0324] Determine the sources of the n spatial signals according to the spatial correlation between the n spatial signals in the photoelectric conversion module to eliminate the interference of internal noise;

[0325] Eliminating the internal noise interference, the incident light is decomposed into n calibration bands, and the incident light is analytically calibrated using the pre-calibrated calibration parameters obtained by pre-calibrating the n calibration bands and n spatial signals to obtain the spectral components of the incident light in each calibration band.

[0326] And / or, after eliminating internal noise interference, the n spatial signals are superimposed and summed to obtain a spatial superposition signal, and the brightness change of the incident light is detected based on the spatial superposition signal.

[0327] For example, in some embodiments, Figure 24 As shown, the imaging system includes:

[0328] Imaging lens, which converges the light of the object and forms an image;

[0329] The above-mentioned image sensor performs asymmetric spatial signal acquisition of incident light based on multiple pixels, and determines the source of the four spatial signals in each pixel based on the spatial correlation between the four spatial signals to eliminate the interference of internal noise; after eliminating the interference of internal noise, combined with the pre-calibration of four calibration color bands in the visible spectrum range, the incident light is parsed into spectral components corresponding to the four calibration color bands to perform brightness detection and color detection on the incident light in each pixel; after eliminating the interference of internal noise, the four spatial signals are superimposed and summed to obtain a spatial superposition signal, and the brightness change of the incident light in each pixel is detected based on the spatial superposition signal.

[0330] For example, Figure 24 and Figure 25 As shown, in the embodiment of the present application, since the asymmetric optical module and the subsequent multi-color band calibration technology in the visible spectrum range can effectively replace the microlens and filter structure in the relevant pixels, the image sensor in the embodiment of the present application does not require any structural offset to adapt to high chief ray angles, and the image sensor in the embodiment of the present application can also adapt to imaging lenses with any aperture number F. Figure 25 As shown, experiments have proved that two imaging lenses with different high chief ray angles CRA and different aperture numbers F, namely lens 1 and lens 2, can effectively adapt to the structure of the image sensor in the embodiment of the present application.

[0331] In this way, the assembly of the imaging system can also be simplified. Imaging lenses of different specifications + image sensors of different specifications can be flexibly assembled according to actual needs to obtain an imaging system, and the assembled imaging system can be calibrated based on the pre-calibration of multiple calibration color bands in the visible spectrum range.

[0332] It should be noted that the pixels and image sensors based on the asymmetric optical module are formed, and multi-band calibration is performed based on the spatial correlation of the spatial signal. Even if the asymmetric optical module has a printing error or a process error, resulting in an unsatisfactory structure of the asymmetric optical module, it will not affect the subsequent spectral imaging performance.

[0333] Finally, if Figure 26 As shown, an embodiment of the present application also provides an optoelectronic device 2600.

[0334] For example, Figure 26As shown, optoelectronic device 2600 includes a processor 2601 having one or more processing cores, a memory 2602 having one or more computer-readable storage media, and a computer program stored in memory 2602 and executable on processor 2601. Processor 2601 is electrically connected to memory 2602. Those skilled in the art will appreciate that the optoelectronic device structure shown in the figure does not limit the optoelectronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0335] Processor 2601 is the control center of optoelectronic device 2600. It connects various components of optoelectronic device 2600 using various interfaces and circuits. By running or loading software programs and / or units stored in memory 2602 and accessing data stored in memory 2602, it executes various functions of optoelectronic device 2600 and processes data, thereby providing overall monitoring of optoelectronic device 2600. Processor 2601 can be a CPU, a graphics processor (GPU), a network processor (NP), etc., and can implement or execute the various methods, steps, and logic blocks disclosed in the embodiments of this application.

[0336] In an embodiment of the present application, the processor 2601 in the optoelectronic device 2600 will load the instructions corresponding to the processes of one or more applications into the memory 2602 in accordance with the corresponding steps, and the processor 2601 will run the applications stored in the memory 2602 to implement various functions, such as executing the steps of the above-mentioned image sensor operation method.

[0337] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0338] like Figure 26 As shown, the optoelectronic device 2600 further includes: an image sensor (not shown in the figure), and the processor 2601 is electrically connected to the image sensor. The image sensor can be the image sensor involved in the above embodiment. Those skilled in the art will understand that Figure 26 The optoelectronic device structure shown in the figure does not constitute a limitation to the optoelectronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0339] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0340] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0341] To this end, the embodiments of the present application provide a computer readable storage medium, which stores a plurality of computer programs capable of being loaded by a processor to execute any of the operation methods of the image sensor provided by the embodiments of the present application. The computer program can execute the steps of the operation method of the image sensor as described above.

[0342] The specific implementation of the above operations can refer to the previous embodiments, which will not be repeated here.

[0343] The computer readable storage medium can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0344] Since the computer program stored in the computer readable storage medium can execute any of the operation methods of the image sensor provided by the embodiments of the present application, the beneficial effects of any of the operation methods of the image sensor provided by the embodiments of the present application can be achieved, which will be described in detail in the previous embodiments, and will not be repeated here.

[0345] The above describes in detail the pixel, the image sensor, the operation method of the image sensor, the optoelectronic device, the computer readable storage medium and the imaging system provided by the embodiments of the present application. The principles and implementation manners are described by applying specific examples in the embodiments of the present application. The above embodiment descriptions are only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed; in conclusion, the content of the specification should not be understood as a limitation of the embodiments of the present application.

[0346] It should be noted that each of the embodiments in the present application is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0347] It should also be noted that, in the present application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0348] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein but is intended to be applied in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A pixel, characterized in that: The pixels include: A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3; an asymmetric optical module, the asymmetric optical module being disposed on the photoelectric conversion module, wherein the asymmetric optical module is configured to focus and diffract incident light to generate a three-dimensional asymmetric light intensity pattern, thereby recording n different spatial signals in a one-to-one correspondence on the n photoelectric converters in the photoelectric conversion module; The pixel is configured to determine sources of the n spatial signals according to spatial correlations among the n spatial signals in the photoelectric conversion module, so as to eliminate interference from internal noise.

2. The pixel according to claim 1, wherein The pixel is further configured to: decompose the incident light using n calibration color bands while excluding internal noise interference, and analytically calibrate the incident light using pre-calibration calibration parameters obtained by pre-calibration of the n calibration color bands and the n spatial signals to obtain the spectral components of the incident light under each of the calibration color bands.

3. The pixel according to claim 1 or 2, characterized in that The pixel is further configured to: superimpose and sum the n spatial signals after eliminating internal noise interference to obtain a spatial superposition signal, and detect brightness changes of the incident light based on the spatial superposition signal.

4. The pixel according to claim 1, wherein The asymmetric optical module comprises: A background structure layer, wherein the background structure layer is made of a first material having a first refractive index; A diffraction structure layer is embedded in the background structure layer, wherein the diffraction structure layer includes a plurality of components made of a second material having a second refractive index for focusing and diffracting the incident light, and the first refractive index is lower than the second refractive index.

5. The pixel according to claim 4, wherein: The first material and the second material are inorganic materials.

6. The pixel according to claim 5, wherein: The component includes at least two diffraction cylinders of different sizes and at least two diffraction ring cylinders of different sizes. Multiple diffraction cylinders and multiple diffraction ring cylinders are staggered and asymmetrically arranged so that the diffraction structure layer diffracts the incident light asymmetrically.

7. An image sensor, characterized in that: The image sensor includes a plurality of pixels arranged in an array; the pixels include: A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3; an asymmetric optical module, the asymmetric optical module being disposed on the photoelectric conversion module, wherein the asymmetric optical module is configured to focus and diffract incident light to generate a three-dimensional asymmetric light intensity pattern, thereby recording n different spatial signals in a one-to-one correspondence on the n photoelectric converters in the photoelectric conversion module; The pixel is configured to determine sources of the n spatial signals according to spatial correlations among the n spatial signals in the photoelectric conversion module, so as to eliminate interference from internal noise.

8. The image sensor according to claim 7, wherein: The image sensor includes pixels of at least two different structures, and the pixels in the image sensor are arranged in a regular array.

9. The image sensor according to claim 7 or 8, characterized in that The pixel is further configured to: decompose the incident light using n calibration color bands while excluding internal noise interference, and analytically calibrate the incident light using pre-calibration calibration parameters obtained by pre-calibration of the n calibration color bands and the n spatial signals to obtain the spectral components of the incident light under each of the calibration color bands.

10. The image sensor according to claim 7 or 8, characterized in that The pixel is further configured to: superimpose and sum the n spatial signals after eliminating internal noise interference to obtain a spatial superposition signal, and detect brightness changes of the incident light based on the spatial superposition signal.

11. The image sensor according to claim 7, wherein: The pixel further includes a back-illuminated silicon substrate, the photoelectric conversion module is arranged on the back-illuminated silicon substrate, and a groove is provided on the back-illuminated silicon substrate surrounding the photoelectric conversion module.

12. An imaging system, characterized in that: The imaging system comprises: Imaging lens, used to converge the light of the object into an image to form incident light; An image sensor configured to focus and diffract the incident light based on pixels in the image sensor to generate a three-dimensional asymmetric light intensity pattern, and then record n ​​different spatial signals on n photoelectric converters in the pixels in a one-to-one correspondence; wherein n is a positive integer greater than or equal to 3; Wherein, the pixels are configured as follows: determining sources of the n spatial signals according to spatial correlations between the n spatial signals to eliminate interference from internal noise; Under the condition of eliminating internal noise interference, the incident light is decomposed by using n calibration color bands, and the incident light is analytically calibrated using pre-calibrated calibration parameters obtained by pre-calibrating the n calibration color bands and the n spatial signals to obtain the spectral components of the incident light in each of the calibration color bands; And / or, when internal noise interference is eliminated, the n spatial signals are superimposed and summed to obtain a spatial superposition signal, and the brightness change of the incident light is detected based on the spatial superposition signal.

13. A method for operating an image sensor, characterized in that: Applied to an image sensor composed of multiple pixels, the method includes: Acquiring an imaging mode of the image sensor, wherein the imaging mode includes a spectral imaging mode, an event imaging mode, and a fusion imaging mode; Based on the imaging mode, controlling the image sensor to collect corresponding incident light in units of the pixel, to obtain n different spatial signals corresponding to each pixel; Determining the source of the spatial signal using the pixel as a unit to eliminate interference from internal noise; When interference from the internal noise is eliminated, based on the imaging mode, detection is performed in units of the pixels, and the color of the incident light is detected according to the n spatial signals within the pixels, and / or the brightness change of the incident light is detected according to the n spatial signals within the pixels; The pixels include: A photoelectric conversion module, comprising n photoelectric converters arranged adjacent to each other in an array, wherein n is a positive integer greater than or equal to 3; An asymmetric optical module is arranged on the photoelectric conversion module, wherein the asymmetric optical module is used to focus and diffract the incident light to produce a three-dimensional asymmetric light intensity pattern, and then record the n different spatial signals on the n photoelectric converters in the photoelectric conversion module in a one-to-one correspondence.

14. The method according to claim 13, wherein: The determining of the spatial signal source in units of pixels to eliminate interference from internal noise includes: For each pixel, obtaining corresponding n different spatial signals; For each of the pixels, analyzing the spatial correlation between the n spatial signals according to the signal values ​​of the n spatial signals; For each of the pixels, determining sources of the n spatial signals according to the spatial correlation; For each of the pixels, when the n spatial signals corresponding to the pixel are derived from the internal noise, the n spatial signals corresponding to the pixel are removed, and / or the n spatial signals corresponding to the pixel are compensated and corrected to eliminate the interference of the internal noise.

15. The method according to claim 14, characterized in that The spatial correlation includes a first correlation, a second correlation, and a third correlation. For each pixel, analyzing the spatial correlation between the n spatial signals according to the signal values ​​of the n spatial signals includes: For each of the pixels, constructing a spatial signal matrix based on the signal values ​​of the n spatial signals; For each of the pixels, analyzing the spatial correlation between the n spatial signals based on the values ​​and distribution patterns of the signal values ​​in the spatial signal matrix; Wherein, when the n signal values ​​in the spatial signal matrix are different and all are non-zero values, the n spatial signals exhibit the first correlation; When the signal values ​​of at least one row or column in the spatial signal matrix are the same non-zero value, the n spatial signals present the second correlation; In the case that some of the n signal values ​​in the spatial signal matrix are zero and some are non-zero values, and the non-zero values ​​are randomly distributed, the third correlation exists between the n spatial signals.

16. The method according to claim 15, characterized in that The determining, for each pixel, sources of the n spatial signals according to the spatial correlation includes: In a case where the first correlation exists between the n spatial signals, determining that the n spatial signals corresponding to the pixel originate from the incident light; When the second correlation is present between the n spatial signals, determining that the n spatial signals corresponding to the pixel are derived from the internal noise, and the internal noise is row stripe noise or column stripe noise; In a case where the third correlation exists between the n spatial signals, it is determined that the n spatial signals corresponding to the pixel are derived from the internal noise, and the internal noise is random noise.

17. The method according to claim 13, wherein The detecting the color of the incident light according to the n spatial signals in the pixel includes: For each pixel, selecting n calibration color bands to decompose the incident light; For each of the pixels, pre-calibration is performed based on the n calibration ribbons to obtain pre-calibration parameters; For each pixel, obtaining n different spatial signals obtained by the pixel performing photoelectric collection on the incident light; For each of the pixels, the incident light is analytically calibrated using the pre-calibrated calibration parameters and the n spatial signals, and the spectral components of the incident light in each of the calibration color bands are calculated.

18. The method according to claim 13, characterized in that The detecting the brightness change of the incident light according to the n spatial signals in the pixel includes: Superimposing and summing the n spatial signals to obtain a spatial superposition signal; A brightness change of the incident light is detected based on the spatial superposition signal.

19. A photoelectric device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 13 to 18.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 13 to 18 are implemented.

Images