Method and apparatus for eliminating cross-talk between image sensor pixels
By establishing a crosstalk convolution mathematical model and performing regular filtering calculations, crosstalk between image sensor pixels is eliminated, solving the problem of ranging error at high resolution, and making it applicable to various image sensor types.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SIGMASTAR TECH LTD
- Filing Date
- 2023-03-24
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies struggle to effectively eliminate crosstalk between pixels in image sensors, especially at high resolutions, which affects ranging accuracy. Furthermore, existing methods, such as deep trench isolation structures, cannot completely eliminate optical crosstalk and are not applicable to existing image sensors.
By establishing a mathematical model of crosstalk convolution between the actual image function and the ideal image function of the image sensor pixels, the regularized filtering coefficients are obtained and convolution calculation is performed to obtain the ideal image function that eliminates crosstalk between pixels, thus achieving crosstalk elimination.
Without changing the pixel structure, it effectively eliminates crosstalk between pixels, is applicable to existing image sensors, and can further eliminate crosstalk after adopting a deep trench isolation structure. It has a wide range of applications and reduces the impact of computation.
Smart Images
Figure CN116228591B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of ranging technology, and in particular to a method and apparatus for eliminating crosstalk between pixels of an image sensor. Background Technology
[0002] Time-of-flight (ToF) technology is a method for accurately measuring the distance to a reference object. There are two types: direct-ToF (dToF) ranging technology, which directly measures the flight time of light to calculate the distance to the reference object, and indirect-ToF (iToF) ranging technology, which calculates the distance to the reference object by periodically modulating and demodulating the light intensity and using phase information.
[0003] Please see Figure 1 This is a schematic diagram illustrating the working principle of an IToF image sensor. Figure 1 As shown, the working principle of the IToF image sensor is as follows: the modulation module 11 controls the transmitter 12 of the image sensor to actively generate emitted light 13; the emitted light 13 is emitted onto the surface of the target object 19, and reflected light 14 is formed after reflection by the target object 19; the signal of the reflected light 14 is sampled and acquired by the detector 15 of the image sensor; and then the relative phase difference between the emitted light 13 and the reflected light 14 is used to determine the signal. The depth information of the target 19 is calculated. The emitter 12 typically uses an active light source in the near-infrared wavelength range to generate emitted light; the detector is typically a pixel array of an image sensor.
[0004] Please see Figure 2 This is a schematic diagram of the pixel structure of an iToF image sensor in the prior art. In the diagram, solid straight lines with arrows represent incident light, and dashed curves with arrows represent the electron motion path. For example... Figure 2 As shown, the pixel structure of the ToF image sensor includes: a photodiode 22 formed in a silicon substrate 21, a dielectric layer 23 covering the silicon substrate 21, and a microlens 24 formed on the dielectric layer 23. Reflected light from the reference position of the pixel enters the pixel as incident light 29. The incident light 29 generates photogenerated electrons 28 within the pixel and outputs them as a voltage signal. Ideally, the reflected light from the reference position of the pixel would only be collected by that pixel. However, in practical applications, because complete optical and electrical isolation between pixels is not possible, due to light incidence or electron diffusion, some photons or electrons are collected by surrounding pixels, resulting in crosstalk (CTK) between pixels.
[0005] With the development of iToF image sensors and consumer demand, high-resolution iToF image sensors have become a goal pursued by many manufacturers and consumers. High resolution means smaller pixel sizes, which increases crosstalk between pixels, causing ranging errors in the iToF image sensor and affecting ranging accuracy. Therefore, eliminating crosstalk between pixels is crucial for high-quality iToF image sensors.
[0006] Existing technologies for eliminating crosstalk between pixels typically involve adding a deep trench isolation (DTI) structure between pixels during the pixel design process to physically isolate adjacent pixels. The DTI structure is generally made of an insulator, which can block the diffusion of photogenerated electrons between pixels, eliminating electrical crosstalk and thus reducing overall crosstalk. However, the DTI structure has limited effectiveness in blocking incident light from entering surrounding pixels and cannot completely eliminate optical crosstalk. Furthermore, the DTI structure needs to be introduced during pixel design and cannot eliminate crosstalk in existing image sensors.
[0007] Therefore, how to eliminate crosstalk in existing image sensors is a technical problem that urgently needs to be solved. Summary of the Invention
[0008] The purpose of this invention is to provide a method and apparatus for eliminating crosstalk between pixels of an image sensor, which can effectively eliminate crosstalk between pixels without designing an additional deep trench isolation structure, and can also eliminate crosstalk in existing image sensors.
[0009] To achieve the above objectives, the present invention provides a method for eliminating crosstalk between pixels of an image sensor, comprising: establishing a crosstalk convolution mathematical model between the actual image function and the ideal image function of the image sensor pixels, and obtaining regularized filtering coefficients based on the convolution kernel in advance to obtain and store crosstalk coefficients; obtaining the actual image function acquired by the image sensor; and performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficients to obtain and output the ideal image function with crosstalk between pixels eliminated.
[0010] To achieve the above objectives, the present invention also provides an apparatus for eliminating crosstalk between pixels of an image sensor, comprising: a model building module, used to build a crosstalk convolution mathematical model between the actual image function and the ideal image function of the image sensor pixels, and to pre-obtain regularization filtering coefficients to obtain and store crosstalk coefficients; an acquisition module, used to acquire the actual image function acquired by the image sensor; and a processing module, used to perform convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficients, to obtain and output the ideal image function with crosstalk eliminated between pixels.
[0011] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor; when the processor executes the computer-executable program, it implements the steps of the method for eliminating crosstalk between image sensor pixels as described in the present invention.
[0012] This invention transforms the crosstalk problem between pixels into a convolution problem by establishing a crosstalk convolution mathematical model. For image sensors with the same manufacturing process, when eliminating crosstalk between pixels, a two-step calculation is performed using pre-stored regularized filter coefficients and the Fourier transform function corresponding to the actual image function acquired by the image sensor. This approximate solution of the ideal image function is obtained without changing the pixel structure, thus eliminating crosstalk between pixels and restoring the ideal image. Alternatively, the image crosstalk elimination algorithm can be further optimized by obtaining a pre-stored target sequence and the actual image function acquired by the image sensor. A one-step calculation is then performed to obtain an approximate solution of the ideal image function, restoring the ideal image while reducing the computational load on the image sensor. This invention can effectively eliminate crosstalk between pixels without designing additional deep-channel isolation structures. Furthermore, this invention can be applied to existing image sensors for crosstalk elimination, making it widely applicable. It can also be applied to image sensors that have already adopted deep-channel isolation structures to eliminate crosstalk between pixels, achieving further crosstalk elimination. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] Figure 1 This is a schematic diagram illustrating the working principle of an iToF image sensor.
[0015] Figure 2This is a schematic diagram of the pixel structure of an iToF image sensor in the prior art;
[0016] Figure 3 A schematic diagram illustrating the steps of the method for eliminating crosstalk between pixels in an image sensor provided by the present invention;
[0017] Figure 4 This is a schematic diagram illustrating the relationship between the ideal signal and the output signal according to an embodiment of the present invention;
[0018] Figure 5 This is a structural block diagram of the image sensor pixel crosstalk elimination device provided by the present invention. Detailed Implementation
[0019] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] One embodiment of the present invention provides a method for eliminating crosstalk between pixels of an image sensor.
[0021] Please refer to the following: Figures 3-4 ,in, Figure 3 This is a schematic diagram illustrating the steps of the method for eliminating crosstalk between pixels in an image sensor provided by the present invention. Figure 4 This is a schematic diagram illustrating the relationship between the ideal signal and the output signal according to an embodiment of the present invention.
[0022] like Figure 3 As shown in this embodiment, the method for eliminating crosstalk between image sensor pixels includes the following steps: S1, establishing a crosstalk convolution mathematical model between the actual image function and the ideal image function of the image sensor pixels, and pre-obtaining regularization filtering coefficients to obtain crosstalk coefficients and storing them; S2, obtaining the actual image function acquired by the image sensor; and S3, performing convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficients to obtain and output the ideal image function with eliminated crosstalk between pixels. Detailed explanations are given below.
[0023] Regarding step S1, a crosstalk convolution mathematical model is established between the actual image function and the ideal image function of the image sensor pixels. Regularized filtering coefficients are pre-obtained to obtain crosstalk coefficients, which are then stored. Specifically, by establishing the crosstalk convolution mathematical model, the crosstalk problem between pixels can be transformed into a convolution problem. Based on the ideal image, and through a certain convolution kernel, the actual image output by the image sensor can be obtained.
[0024] like Figure 4 As shown, taking the iToF image sensor as an example, the principle of establishing the crosstalk convolution mathematical model is explained. Figure 4 A 3x3 pixel array is used to simulate a real-world pixel array. The upper part represents the information of each pixel in an ideal situation, and the lower part represents the surrounding pixels affected by the information of that pixel in the actual application. Figure 4 The middle (a) portion represents the pixel range 421 affected by the first pixel 411; Figure 4 The middle (b) portion of the image represents the pixel range 422 affected by the second pixel 412; Figure 4 The middle (c) portion of the diagram represents the pixel range 429 affected by the last pixel 419. During the operation of the iToF image sensor, not just one pixel works at a time, but the entire pixel array works simultaneously. This results in each pixel not only collecting its own signal but also collecting crosstalk signals from surrounding pixels, ultimately measuring the signal output by the pixel array.
[0025] Therefore, the relationship between the ideal signal and the actual output signal of each pixel can be described by the following formula:
[0026]
[0027] Among them, A mn_output B is the output signal of the pixel at coordinates (m, n) in the pixel array; ij_ideal It is the ideal signal of the pixel at coordinate (i, j) in the pixel array; a mn This represents the influence coefficient of all pixels in the pixel array on the pixel at coordinates (m, n). The final pixel output signal can be understood as the superposition of all pixel signal components in space; therefore, crosstalk between pixels can be represented by convolution.
[0028] The above analysis shows that by establishing a crosstalk convolution mathematical model, the crosstalk problem between pixels can be transformed into a convolution problem. Based on an ideal image, and through a certain convolution kernel, the actual image output by the iToF image sensor can be obtained.
[0029] In some embodiments, the crosstalk convolution mathematical model is expressed by the following formula:
[0030] g(x,y)=h(x,y)*f(x,y)+n(x,y) (Formula 1)
[0031] Wherein, g(x,y) is the actual image function, f(x,y) is the ideal image function, h(x,y) is the convolution kernel, and n(x,y) is additive noise.
[0032] In some embodiments, the regularization filtering coefficients are further obtained by: obtaining a convolution kernel in advance through an imaging experiment on the center pixel of the image sensor pixel array using a point light source, and obtaining the regularization filtering coefficients based on the convolution kernel; wherein, when eliminating crosstalk between pixels, the regularization filtering coefficients are the same for image sensors with the same process conditions.
[0033] Specifically, the convolution kernel h(x, y) is obtained by: (1) measuring the crosstalk sequence CTK(x, y) of the center pixel of the pixel array to the whole image; (2) dividing the crosstalk sequence CTK(x, y) by the sum of all elements in the crosstalk sequence to obtain the convolution kernel h(x, y).
[0034] The crosstalk CTK(x,y) divided by the sum of all elements in crosstalk CTK(x,y) is expressed by the following formula:
[0035]
[0036] Where M and N are the number of rows and columns of the crosstalk sequence CTK(x,y), respectively.
[0037] In some embodiments, the regularization filter coefficients are expressed by the following formula:
[0038]
[0039] Where HP represents the regularization filter coefficients, H(u, v) represents the Fourier transform of the convolution kernel, and H... * (u, v) is the conjugate of H(u, v), P(u, v) is the Fourier transform of the Laplace operator, γ is the boundary fuzziness adjustment coefficient, and γ·|P(u, v)| 2 Used to reduce the impact of additive noise on an ideal image.
[0040] Since the boundary fuzziness adjustment coefficient γ can be manually adjusted to obtain a suitable value, the Fourier transform P(u, v) of the Laplace operator is a constant value; the convolution kernel h(x, y) can be obtained through testing, and its frequency domain expression H(u, v) can be obtained through Fourier transform. Taking the conjugate of H(u, v) yields Hu. * (u, v), and iToF image sensors with the same process conditions have the same convolution kernel h(x, y); therefore, when eliminating crosstalk between pixels, the regularization filter coefficient HP is the same for iToF image sensors with the same process conditions.
[0041] Specifically, the boundary blur adjustment coefficient γ can be obtained as follows: adjust the value of γ from small to large and observe whether there is boundary blur in the image on the test interface; when γ increases to a certain first critical value, the image boundary is clear and there is no boundary blur; when γ increases to a certain second critical value, the image boundary blur reappears; then the value range of γ is between the first critical value and the second critical value (including the critical value itself).
[0042] The Laplacian operator is a two-dimensional isotropic measure of the second spatial derivative of an image. It can highlight regions in an image where the intensity changes rapidly, and is therefore commonly used in edge detection tasks. In some embodiments, the Laplacian operator p(x, y) is expressed by the following formula:
[0043]
[0044] Regarding step S2, obtaining the actual image function acquired by the image sensor. Specifically, the actual image function can be directly extracted from the image sensor. The captured image of the target object is an m*n pixel array. Each pixel in the pixel array corresponds to a digital signal. The combination of digital signals and its display by software is the image we see. Therefore, this combination of digital signals can be directly extracted from the image sensor as the actual image function. That is, the actual image function is a discrete sequence.
[0045] In step S3, the crosstalk convolution mathematical model is convolved based on the actual image function and the crosstalk coefficient to obtain an ideal image function that eliminates crosstalk between pixels and output it.
[0046] In some embodiments, the regularized filtering coefficient HP is used as the crosstalk coefficient; then, step S3 further includes: (1) converting the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform, and converting the crosstalk product mathematical model into an approximate solution model for the Fourier transform function corresponding to the ideal image function according to regularized filtering; (2) obtaining the Fourier transform function corresponding to the actual image function; (3) calculating the approximate solution model according to the Fourier transform function corresponding to the actual image function and the regularized filtering coefficient to obtain an approximate solution for the Fourier transform function corresponding to the ideal image function; (4) performing an inverse Fourier transform on the approximate solution for the Fourier transform function corresponding to the ideal image function to obtain an approximate solution for the ideal image function.
[0047] Specifically, the crosstalk convolution mathematical model is expressed using Equation 1 above. Since spatial convolution is equivalent to the product of the frequency domain elements of its Fourier transform, that is:
[0048]
[0049] Wherein, H(u,v) is the Fourier transform of the convolution kernel h(x,y), and F(u,v) is the Fourier transform of the ideal image function f(x,y).
[0050] Therefore, the crosstalk convolution mathematical model shown in Formula 1 can be converted into a crosstalk product mathematical model in the frequency domain, and the crosstalk product mathematical model is expressed by the following formula:
[0051] G(u,v)=H(u,v)·F(u,v)+N(u,v) (Formula 4)
[0052] Wherein, G(u,v) is the Fourier transform of the actual image function g(x,y), H(u,v) is the Fourier transform of the convolution kernel h(x,y), F(u,v) is the Fourier transform of the ideal image function f(x,y), and N(u,v) is the Fourier transform of the additive noise n(x,y).
[0053] The crosstalk product mathematical model is converted into an approximate solution model for the Fourier transform function corresponding to the ideal image function by regular filtering. Specifically, from Equation 4, we can obtain:
[0054] F(u,v)=[(G(u,v)-N(u,v))] / H(u,v) (Formula 5)
[0055] Using regularized filtering, an approximate solution model for the Fourier transform function corresponding to the ideal image function can be obtained according to Formula 5. The approximate solution model is expressed by the following formula:
[0056]
[0057] in, HP is the approximate solution of the Fourier transform function corresponding to the ideal image function, and HP is the regular filter coefficient, the expression of which is shown in Formula 2.
[0058] In this embodiment, for image sensors with the same process conditions, when eliminating crosstalk between pixels, by obtaining the pre-stored regularization filter coefficients HP and the Fourier transform function G(u, v) corresponding to the actual image function acquired by the image sensor, and substituting the regularization filter coefficients HP and the Fourier transform function G(u, v) corresponding to the actual image function into the above formula 6, an approximate solution of the Fourier transform of the ideal image function can be calculated. Then the approximate solution The inverse Fourier transform is used to obtain an approximate solution to the ideal image function f(x, y). This can eliminate crosstalk and restore the ideal image.
[0059] The above embodiments of the present invention can eliminate crosstalk and restore the ideal image. However, the restoration process of the ideal image, in addition to obtaining and storing the regularized filter coefficients HP, also requires calculating the approximate solution of the Fourier transform of the ideal image function according to Formula 6. approximate solution The inverse Fourier transform is used to obtain an approximate solution to the ideal image function f(x, y). In other words, two more calculations are required. The calculation steps are numerous, and the amount of computation is related to the size of the pixel array. The larger the pixel array, the larger the sequence of the above parameters, and the greater the amount of computation, which will affect the frame rate of the image sensor.
[0060] To further optimize the image crosstalk cancellation algorithm, this invention further performs an inverse Fourier transform on the regularized filter coefficients to obtain the target sequence as the crosstalk coefficients.
[0061] In some embodiments, the target sequence is further obtained by: (1) performing an inverse Fourier transform on the regularized filter coefficients HP to obtain an initial sequence d(x, y); (2) taking a submatrix at the center position of the initial sequence d(x, y) according to the calculation requirements. The target sequence is defined as follows: When eliminating crosstalk between pixels, for image sensors with the same process conditions, the initial sequence d(x, y) is the same, and the target sequence (i.e., the submatrix) is defined as follows: The larger the value of ), the closer the approximate solution of the ideal image function is to the ideal image. Image sensors under the same manufacturing conditions have the same initial sequence d(x, y), thus obtaining the target sequence... Since the values are the same, all target sequences can be pre-calculated and obtained from other computing software. This data is then stored in the image sensor to reduce the computational load on the sensor. Target sequence The larger the value, the greater the computational load, and the smaller the approximate solution. The closer the target sequence is to the ideal image function f(x, y), the greater the computational cost and the lower the frame rate of the image sensor. Therefore, the appropriate target sequence can be selected based on the actual crosstalk cancellation accuracy and frame rate requirements.
[0062] When the target sequence obtained by performing an inverse Fourier transform on the regularized filter coefficients is used as the crosstalk coefficients, step S3 further includes: converting the crosstalk convolution mathematical model into a spatial convolution mathematical model according to the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform and regularized filtering; and then calculating the spatial convolution mathematical model according to the actual image function and the target sequence to obtain an approximate solution of the ideal image function.
[0063] In some embodiments, the step of converting the crosstalk convolution mathematical model into a spatial convolution mathematical model based on the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform and regular filtering further includes: (1) converting the crosstalk convolution mathematical model into a crosstalk product mathematical model based on the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform, and converting the crosstalk product mathematical model into an approximate solution model for the Fourier transform function corresponding to the ideal image function based on regular filtering; (2) converting the approximate solution model into the spatial convolution mathematical model based on the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform.
[0064] In this embodiment, the crosstalk convolution mathematical model is represented by Equation 1, the crosstalk product mathematical model is represented by Equation 4, the approximate solution model is represented by Equation 6, and the regularized filter coefficients HP are represented by Equation 2. Based on the equivalence relationship between spatial convolution and the frequency domain element-wise product of its Fourier transform as shown in Equation 3, the frequency domain element-wise product represented by Equation 6 can be converted into the corresponding spatial convolution, i.e., the spatial convolution mathematical model.
[0065] The spatial convolution mathematical model is expressed by the following formula:
[0066]
[0067] in, This is an approximate solution to the ideal image function. Let g(x, y) be the target sequence and g(x, y) be the actual image function.
[0068] In this embodiment, for image sensors with the same process conditions, crosstalk between pixels is eliminated by acquiring a pre-stored target sequence. And the function g(x, y) for obtaining the actual image acquired by the image sensor, and the target sequence By substituting the actual image function g(x, y) into Equation 7 above, an approximate solution for the ideal image function can be calculated. This can eliminate crosstalk and create a complex image. In other words, the target sequence... The values can be pre-calculated from other computing software and stored in the image sensor. Equation 7 only requires one step of calculation to eliminate crosstalk and restore the ideal image, reducing the computational load of the image sensor and avoiding affecting the frame rate of the image sensor.
[0069] Compared to existing technologies that use deep trench isolation structures to eliminate crosstalk between pixels, which require considering the impact of crosstalk during the image sensor design process and are not applicable to existing image sensors, the crosstalk elimination method for image sensors provided by this invention can eliminate crosstalk in existing image sensors and has a wider range of applications. Furthermore, the crosstalk elimination method for image sensors provided by this invention can also be applied to image sensors that have adopted deep trench isolation structures to eliminate crosstalk between pixels, achieving further elimination of crosstalk. In addition to being applicable to crosstalk elimination in iToF image sensors, the crosstalk elimination method for image sensors provided by this invention is also applicable to other solid-state image sensors such as CMOS image sensors and CDD image sensors.
[0070] As can be seen from the above, this invention transforms the crosstalk problem between pixels into a convolution problem by establishing a crosstalk convolution mathematical model. For image sensors with the same process conditions, when eliminating crosstalk between pixels, the approximate solution of the ideal image function can be obtained by acquiring pre-stored regularized filter coefficients and the Fourier transform function corresponding to the actual image function acquired by the image sensor in two steps. Without changing the pixel structure, crosstalk between pixels is eliminated by regularized filtering, restoring the ideal image. Alternatively, the image crosstalk elimination algorithm can be further optimized by acquiring pre-stored target sequences and the actual image function acquired by the image sensor in one step, restoring the ideal image while reducing the computational load of the image sensor.
[0071] Based on the same inventive concept, the present invention also provides a device for eliminating crosstalk between pixels of an image sensor. The provided device for eliminating crosstalk between pixels of an image sensor can employ, for example... Figure 3 The method shown for eliminating crosstalk between image sensor pixels completes the elimination of crosstalk between image sensor pixels.
[0072] Please see Figure 5 This is a structural block diagram of an image sensor pixel crosstalk elimination device provided in an embodiment of the present invention. Figure 5 As shown, the crosstalk elimination device between pixels of the image sensor includes: a model building module 51, an acquisition module 52, and a processing module 53.
[0073] Specifically, the model building module 51 is used to build a crosstalk convolution mathematical model between the actual image function and the ideal image function of the image sensor pixels, and to obtain the regularization filtering coefficients in advance to obtain the crosstalk coefficients and store them; the acquisition module 52 is used to acquire the actual image function acquired by the image sensor; the processing module 53 is used to perform convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficients, to obtain the ideal image function that eliminates crosstalk between pixels and output it.
[0074] In some embodiments, regularized filtering coefficients are used as the crosstalk coefficients; the processing module 53 is further configured to: convert the crosstalk convolution mathematical model into a crosstalk product mathematical model according to the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform, and convert the crosstalk product mathematical model into an approximate solution model for the Fourier transform function corresponding to the ideal image function according to regularized filtering; obtain the Fourier transform function corresponding to the actual image function; calculate the approximate solution model according to the Fourier transform function corresponding to the actual image function and the regularized filtering coefficients to obtain an approximate solution for the Fourier transform function corresponding to the ideal image function; perform an inverse Fourier transform on the approximate solution for the Fourier transform function corresponding to the ideal image function to obtain an approximate solution for the ideal image function.
[0075] In some embodiments, the inverse Fourier transform of the regularized filtering coefficients is performed to obtain a target sequence as the crosstalk coefficients; the processing module 53 is further configured to: perform convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficients to obtain an ideal image function that eliminates crosstalk between pixels, the step further includes: converting the crosstalk convolution mathematical model into a spatial convolution mathematical model according to the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform and regularized filtering, and then calculating the spatial convolution mathematical model according to the actual image function and the target sequence to obtain an approximate solution of the ideal image function.
[0076] In some embodiments, the image sensor may be an iToF image sensor, a CMOS image sensor, or a CCD image sensor, or other image sensors that achieve imaging through optical imaging principles.
[0077] The working methods of each module can be found in the following references. Figure 3 The descriptions of the corresponding steps in the method for eliminating crosstalk between image sensor pixels shown are not repeated here.
[0078] Based on the same inventive concept, the present invention also provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor; when the processor executes the computer-executable program, it implements as follows: Figure 3 The steps of the method for eliminating crosstalk between pixels of an image sensor are shown.
[0079] Within the scope of this inventive concept, embodiments can be described and illustrated based on modules that perform one or more of the described functions. These modules (also referred to herein as units, etc.) can be physically implemented by analog and / or digital circuitry, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits, etc., and can optionally be driven by firmware and / or software. The circuitry can, for example, be implemented in one or more semiconductor chips. The circuitry constituting a module can be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware performing some functions of the module and a processor performing other functions of the module. Without departing from the scope of this inventive concept, each module of an embodiment can be physically divided into two or more interactive and discrete modules. Similarly, without departing from the scope of this inventive concept, the modules of an embodiment can be physically combined into more complex modules.
[0080] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such use of data can be interchanged where appropriate. The term "based on" can be understood as not necessarily intended to express an exclusive set of factors, but can instead, at least in part, depending on the context, allow for the presence of other factors that are not necessarily explicitly described. Furthermore, embodiments and features in the embodiments of this invention can be combined with each other without conflict. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar / identical parts between embodiments can be referred to mutually.
[0081] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for eliminating crosstalk between pixels of an image sensor, characterized in that, include: A crosstalk convolution mathematical model is established between the actual image function and the ideal image function of the image sensor pixels. Regular filter coefficients are obtained in advance based on the convolution kernel. The regular filter coefficients are subjected to inverse Fourier transform to obtain the target sequence, which is then used as crosstalk coefficients and stored. A function to obtain the actual image acquired by the image sensor; as well as The crosstalk convolution mathematical model is convolved based on the actual image function and the crosstalk coefficient to obtain and output an ideal image function that eliminates crosstalk between pixels. The process of obtaining the target sequence by performing an inverse Fourier transform on the regularized filter coefficients includes: The initial sequence is obtained by performing an inverse Fourier transform on the regularized filter coefficients; According to the calculation requirements, the submatrix at the center position of the initial sequence is taken as the target sequence; In eliminating crosstalk between pixels, for image sensors with the same process conditions, the initial sequence is the same, and the larger the target sequence is, the closer the approximate solution of the ideal image function is to the ideal image.
2. The method according to claim 1, characterized in that, The regularization filter coefficients are further obtained in the following manner: the convolution kernel is obtained in advance through an imaging experiment on the center pixel of the pixel array of the image sensor using a point light source, and the regularization filter coefficients are obtained based on the convolution kernel; wherein, when eliminating crosstalk between pixels, the regularization filter coefficients are the same for image sensors with the same process conditions.
3. The method according to claim 1, characterized in that, The regularization filter coefficients are expressed by the following formula: ； Where HP represents the regularization filter coefficient. The Fourier transform of the convolution kernel for conjugate, For the Fourier transform of the Laplace operator, This is the boundary fuzziness adjustment coefficient. Used to reduce the impact of additive noise on an ideal image.
4. The method according to claim 3, characterized in that, The Laplace operator Expressed using the following formula: 。 5. The method according to claim 3, characterized in that, The step of performing convolution calculation on the crosstalk convolution mathematical model based on the actual image function and the crosstalk coefficient to obtain an ideal image function that eliminates crosstalk between pixels further includes: converting the crosstalk convolution mathematical model into a spatial convolution mathematical model based on the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform and regular filtering; and then calculating the spatial convolution mathematical model based on the actual image function and the target sequence to obtain an approximate solution of the ideal image function.
6. The method according to claim 5, characterized in that, The step of converting the crosstalk convolution mathematical model into a spatial convolution mathematical model based on the equivalence relationship between spatial convolution and the element-wise product of the frequency domain of Fourier transform, and regular filtering, further includes: Based on the equivalence relationship between the frequency domain element-wise product of spatial convolution and Fourier transform, the crosstalk convolution mathematical model is converted into a crosstalk multiplication mathematical model, and the crosstalk multiplication mathematical model is converted into an approximate solution model for the Fourier transform function corresponding to the ideal image function based on regular filtering. Based on the equivalence between spatial convolution and the frequency domain element-wise product of Fourier transform, the approximate solution model is converted into the spatial convolution mathematical model.
7. The method according to claim 6, characterized in that, The crosstalk convolution mathematical model is expressed by the following formula: ， in, For the actual image function, For the ideal image function, For convolution kernel, It is additive noise; The mathematical model for the crosstalk product is expressed by the following formula: ， in, For the actual image function The corresponding Fourier transform, For the convolution kernel Fourier transform, For the ideal image function Fourier transform, The additive noise Fourier transform; The approximate solution model is expressed by the following formula: ； in, This is an approximate solution to the Fourier transform function corresponding to the ideal image function. These are the regularization filter coefficients; The spatial convolution mathematical model is expressed by the following formula: ， in, This is an approximate solution to the ideal image function. Let be the target sequence.
8. A device for eliminating crosstalk between pixels of an image sensor, characterized in that, include: The model building module is used to build a mathematical model of crosstalk convolution between the actual image function and the ideal image function of the image sensor pixels, and to obtain regularized filtering coefficients in advance. The regularized filtering coefficients are then subjected to inverse Fourier transform to obtain the target sequence, which is used as the crosstalk coefficients and stored. The acquisition module is used to acquire the actual image function captured by the image sensor; as well as The processing module is used to perform convolution calculation on the crosstalk convolution mathematical model according to the actual image function and the crosstalk coefficient, obtain the ideal image function that eliminates crosstalk between pixels, and output it. The process of obtaining the target sequence by performing an inverse Fourier transform on the regularized filter coefficients includes: The initial sequence is obtained by performing an inverse Fourier transform on the regularized filter coefficients; According to the calculation requirements, the submatrix at the center position of the initial sequence is taken as the target sequence; In eliminating crosstalk between pixels, for image sensors with the same process conditions, the initial sequence is the same, and the larger the target sequence is, the closer the approximate solution of the ideal image function is to the ideal image.