Image sensor temperature compensation method and system, storage medium and electronic equipment
By combining multi-sensor interpolation and Chebyshev polynomials, a three-stage pipeline system based on FPGA was constructed, which solved the compensation accuracy problem caused by temperature gradients in image sensors and achieved high-precision, real-time temperature compensation.
Patent Information
- Application Number
- CN202511564334.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-02-27
AI Technical Summary
Existing image sensor temperature compensation schemes suffer from spatial temperature perception distortion and insufficient accuracy and efficiency of compensation models. In particular, the accuracy of edge pixel compensation decreases when a temperature gradient exists, and existing technologies cannot meet the high-speed requirements of industrial real-time detection.
Pixel-level temperature information is obtained by interpolation using multiple temperature sensors, and high-precision compensation is performed using Chebyshev polynomials. Efficient temperature compensation is achieved by constructing a three-stage pipeline system architecture in an FPGA.
This achievement represents a leap from chip-level to pixel-level temperature sensing, improving the image quality and measurement accuracy of image sensors under different ambient temperatures, meeting the high-speed requirements of real-time detection, and saving hardware storage resources.
Smart Images
Figure CN121585897A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing, and particularly relates to an image sensor temperature compensation method, system, storage medium, and electronic device. Background Technology
[0002] Industrial cameras are widely used in machine vision, precision measurement, and automated inspection, with their core component being the image sensor (such as CMOS or CCD). The photoelectric characteristics of image sensors are extremely sensitive to temperature, which can cause drift in their output signal. This manifests primarily as follows: increased temperature leads to a significant increase in dark current noise, especially during long exposures, resulting in images filled with noise and a decreased signal-to-noise ratio (SNR); temperature changes also cause drift in the sensor's photoelectric conversion characteristics, altering image brightness and contrast and affecting measurement stability; the sensor's reference black level also changes with temperature, causing an overall image bias and affecting the accurate setting of thresholds.
[0003] To address the aforementioned issues, traditional compensation schemes are mainly divided into two categories: software post-processing: algorithmic compensation is performed on the acquired complete image on a computer (PC). This method has inherent processing latency, cannot meet the high-speed requirements of industrial real-time detection, and consumes a large amount of CPU resources; FPGA-based hardware compensation: the compensation algorithm is integrated into the FPGA inside the camera, utilizing its parallel processing capabilities to achieve real-time, low-latency compensation.
[0004] Currently, FPGA-based image sensor temperature compensation schemes employ a single temperature sensor and lookup table interpolation, which have two inherent drawbacks: 1. Spatial temperature perception distortion: Image sensor chips exhibit temperature gradients during operation, with significant temperature differences between the chip's center and edges. A single temperature sensor cannot reflect this two-dimensional temperature distribution, leading to decreased compensation accuracy for edge pixels. 2. Insufficient accuracy and efficiency of the compensation model: The accuracy of lookup tables is limited by representation modulus density, making it difficult to achieve high-precision compensation, resulting in a trade-off between temperature compensation accuracy and speed.
[0005] Therefore, in order to solve the above problems, the present invention provides an image sensor temperature compensation method, system, storage medium and electronic device, which obtains pixel-level temperature information by interpolation through multiple temperature sensors and uses Chebyshev polynomials for high-precision compensation in hardware implementation, to improve the image quality and measurement accuracy of industrial cameras under different working environmental temperatures. Summary of the Invention
[0006] The purpose of this invention is to overcome the above-mentioned problems in the prior art and to provide an image sensor temperature compensation method, system, storage medium and electronic device.
[0007] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: A temperature compensation method for an image sensor, used to compensate for pixel value changes in an image sensor caused by temperature variations, includes: The temperature of each pixel on the image sensor is linearly mapped to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. The recursive computation of the polynomial is solidified to form a pipeline architecture of hardware with the same order as the preset order, and the output values of each order term corresponding to the input value are calculated in a pipeline. The output value is summed and multiplied by the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different order of the polynomial.
[0008] Furthermore, the linear mapping includes: The input value is obtained by dividing the difference between twice the pixel temperature value and the sum of the maximum and minimum temperatures by the difference between the maximum and minimum temperatures. The maximum temperature value is the maximum value within the preset operating temperature range of the image sensor, and the minimum temperature value is the minimum value within the preset operating temperature range of the image sensor.
[0009] Furthermore, the recursive computation of the solidified polynomial forms a hardware pipeline architecture of the same order of magnitude as the preset order, including: Set up arithmetic units of the same order of magnitude as the preset order of the polynomial, and connect each level according to the recursive relationship of the polynomial; The arithmetic units at different levels have the same structure, each including a multiplier and an adder.
[0010] Further, the step of pipelined calculation of the output values of each order term corresponding to the input value includes: The arithmetic units at each level perform the same multiplication and addition operation on the data pipeline to obtain the output values of the polynomial terms corresponding to different orders of the polynomial step by step. The calculation results of each level are stored and used as input for the next level of calculation.
[0011] Furthermore, the dot product of the output value and the pre-stored temperature compensation coefficient includes: The output value of each order term is multiplied in parallel with the temperature compensation coefficient corresponding to its order.
[0012] Furthermore, the temperature compensation coefficient is obtained through calibration, which includes: The image sensor is calibrated with a number of compensation parameters of the same preset order within its operating temperature range, in order to fit a temperature compensation coefficient that optimally matches the Chebyshev polynomial with the calibration data.
[0013] Furthermore, the temperature compensation coefficient includes a gain compensation coefficient and a black level compensation coefficient.
[0014] Furthermore, the preset point temperature is obtained by a temperature sensor located at the edge of the image sensor.
[0015] An image sensor temperature compensation system, comprising: The mapping module linearly maps the temperature of each pixel on the image sensor to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. The solidification module solidifies the recursive calculation of the polynomial to form a hardware pipeline architecture of the same order as the preset order, and calculates the output value of each order term corresponding to the input value in a pipelined manner. The calculation module accumulates the product of the output value and the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different degree of the polynomial. The mapping module, the solidification module, and the calculation module form a three-stage pipeline system architecture in the FPGA. The calculation result of the mapping module serves as the input to the calculation of the solidification module, and the calculation result of the solidification module serves as the input to the calculation module.
[0016] A storage medium having a computer program stored thereon, which, when executed by a processor, implements an image sensor temperature compensation method as described above.
[0017] The beneficial effects of this invention are: (1) In this invention, the temperature of each pixel on the image sensor is linearly mapped to generate the input value of the first type of Chebyshev polynomial within the standard range of the preset order. In the first-level system architecture, the temperature of each pixel on the image sensor is linearly mapped to generate the standard input value that satisfies the subsequent Chebyshev polynomial calculation. The raw material source of the input value is carried out in a relatively independent pipeline operation in the FPGA, so that the subsequent compensation calculation can make full use of the minimum and maximum error characteristics of the Chebyshev polynomial in the standard range, laying a mathematical foundation for high-precision consistency compensation across the entire temperature range. By solidifying the recursive computation of the polynomial, a pipeline architecture of hardware of the same order as the preset order is formed. The output values of each term corresponding to the input value are calculated in a pipeline. The pipeline architecture is constructed by utilizing the recursive relationship of the Chebyshev polynomial itself. A multi-level recursive pipeline hardware computing unit is formed to perfectly match its recursive characteristics. The high-order computation is transformed into fixed linear operations on the low-order terms, which improves the data throughput. Furthermore, in the second-level system architecture, the high-order polynomial computation of the Chebyshev polynomial is further decomposed into a regular pipeline architecture, which further improves the data throughput. By accumulating the output value and the product of the pre-stored temperature compensation coefficients, compensation parameters are obtained in real time for temperature compensation. Chebyshev polynomial approximation is used to replace piecewise linear interpolation to build a more powerful compensation model, making the compensation curve smoother and more accurate. Moreover, only one set of temperature coefficients needs to be stored in the FPGA, which greatly saves hardware storage resources compared to the need to store a massive temperature-coefficient pair lookup table in the existing technology.
[0018] By interpolating the temperature of a preset point on the image sensor, the temperature estimate of each pixel on the image sensor is obtained, which accurately reflects the two-dimensional temperature gradient on the surface of the image sensor. This overcomes the spatial limitations of the traditional single-point temperature measurement method and solves the problem of inaccurate edge pixel compensation caused by uneven temperature distribution from the source, realizing a leap from chip-level to pixel-level temperature sensing.
[0019] (2) In this invention, by introducing Chebyshev polynomials as a mathematical model in temperature compensation, on the one hand, its minimum and maximum error characteristics are combined with pixel-level temperature acquisition to accurately locate the temperature compensation position and compensation method, and to achieve high compensation accuracy in the pixel space of the image sensor. On the other hand, by combining its recursive characteristics with the pipeline architecture characteristics of FPGA hardware parallelism, the high-order calculation is transformed into fixed linear operation on the low-order terms, thus overcoming the technical contradiction between existing high approximation accuracy and high hardware efficiency.
[0020] (3) In this invention, multi-sensor interpolation is used to achieve a leap from chip-level temperature sensing to pixel-level temperature prediction, which fundamentally solves the compensation error caused by uneven temperature space. Chebyshev polynomials are used, and their minimum and maximum error characteristics in a given interval ensure the consistency of compensation accuracy across the entire temperature range. The three-stage pipeline system architecture optimized for the algorithm decomposes the complex two-dimensional interpolation and high-order polynomial calculations into regular pipeline steps, so as to process a number of parallel pixels in each clock cycle, achieving extremely high data throughput. At the same time, it avoids massive per-pixel data storage and only needs to store a small number of global model coefficients, thus achieving considerable performance improvement at a reasonable hardware cost. Attached Figure Description
[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of the image sensor temperature compensation method in this invention; Figure 2 This is a flowchart of the image sensor temperature compensation system in this invention; Figure 3 This is a schematic diagram showing the positions of each pixel within the rectangular area of the image sensor in this invention; Figure 4 This is a hardware block diagram of the three-stage flow system architecture in this invention; Figure 5 This is a hardware block diagram of the pipeline architecture in this invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] like Figure 1 As shown, the present invention first provides an image sensor temperature compensation method for compensating for pixel value changes in an image sensor caused by temperature variations, comprising: The temperature of each pixel on the image sensor is linearly mapped to generate input values within the standard range of Chebyshev polynomials; The polynomial recursive relation is solidified to form a hardware pipeline architecture, and the output values of each order term corresponding to the input value are calculated in a pipeline. The output value is summed and multiplied by the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different order of the polynomial.
[0024] In this embodiment, in order to obtain the temperature of all pixels on the image sensor, the pixel-level temperature estimate of all pixels is obtained by interpolation of the temperatures of several preset points. Compared with the overall chip-level temperature estimation of the image sensor in the prior art, the accuracy is higher, thus facilitating the acquisition of higher-precision temperature compensation parameters.
[0025] In this embodiment, the image sensor types include, but are not limited to, area array sensors, linear array sensors, and CCD image sensors. All of them can obtain the pixel-level temperature of the image sensor through interpolation algorithms. By applying the pixel value interpolation method in image processing to the acquisition of the temperature of each pixel of the image sensor, a leap from chip-level temperature sensing in the prior art to pixel-level temperature sensing is achieved. The temperature differences of different pixels are characterized to correspond to different temperature compensations, which fundamentally improves the accuracy of temperature compensation of the image sensor.
[0026] The types of interpolation algorithms include, but are not limited to, bilinear interpolation, nearest neighbor interpolation, linear interpolation, cubic polynomial interpolation, and bicubic interpolation.
[0027] In this embodiment, as Figure 3 As shown, taking an area array image sensor and bilinear interpolation algorithm as an example, in order to accurately sense the pixel-level temperature gradient on the surface of the image sensor, a temperature sensor is arranged at each of the four corners of the image sensor edge. For any pixel P(x, y) in the image array, its specific temperature value T(x, y) is calculated in real time by the bilinear interpolation algorithm, rather than being taken from a single sensor.
[0028] Suppose that pixel P(x, y) is located within a rectangular region formed by four temperature sensors S1(T1), S2(T2), S3(T3), and S4(T4). The specific interpolation calculation process for its temperature T(x,y) is as follows:
[0029] Where T1, T2, T3, and T4 represent the real-time temperature values acquired by the four temperature sensors S1, S2, S3, and S4, respectively; (x1, y1) and (x2, y2) are the coordinates of the lower left and upper right boundary points of the rectangular area; T(x,y) is the temperature of the pixel being calculated; and T... top T represents the temperature value at the intersection of the vertical axis of the corresponding pixel and the upper boundary of the rectangular region. bottom The temperature value at the intersection of the vertical axis of the corresponding pixel and the lower boundary of the rectangular region.
[0030] Based on the above interpolation calculation process, the temperature T of any pixel point at the lower boundary of the rectangular region can be obtained from (x1, y1) and (x2, y1). bottom Similarly, the temperature T at any point on the upper boundary of the rectangular region can be obtained. top , among which, T top and T bottom The X-coordinate is the same as the X-coordinate of the pixel to be calculated; by T top and T bottom It can obtain the temperature of any pixel on the vertical axis of the corresponding pixel.
[0031] In this embodiment, the preset order of the first type of Chebyshev polynomial corresponds to different temperature compensation accuracies. The higher the order, the higher the temperature compensation accuracy, but the more hardware computing resources are required. To solve this problem, in this embodiment, the output values of polynomial terms of different orders are calculated in the second-level system architecture by solidifying the polynomial recursion relationship to form a pipeline architecture. This cleverly combines the recursive characteristics of Chebyshev polynomials with the pipeline architecture of FPGA hardware parallelism, saving the calculation time of the Chebyshev polynomial model. Furthermore, the pipeline calculation enables the calculation of the output values corresponding to the number of parallel pixels to be completed in each clock cycle, significantly improving the calculation speed.
[0032] In this invention, by introducing Chebyshev polynomials as a mathematical model in temperature compensation, on the one hand, its minimum and maximum error characteristics are combined with pixel-level temperature acquisition to accurately locate the temperature compensation position and compensation method, thus achieving high compensation accuracy in the pixel space of the image sensor. On the other hand, by combining its recursive characteristics with the pipeline architecture characteristics of FPGA hardware parallelism, high-order calculations are transformed into fixed linear operations on low-order terms, overcoming the technical contradiction between high approximation accuracy and high hardware efficiency in the present invention.
[0033] In this embodiment, the linear mapping, solidification calculation, and cumulative correction steps are respectively formed into a three-level pipeline system architecture in the FPGA. The complex interpolation algorithm and high-order polynomial calculation are decomposed into regular pipeline steps to process a number of pixels in parallel in each clock cycle, thereby achieving an extremely high data throughput.
[0034] In this invention, the temperature of each pixel on the image sensor is linearly mapped to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. In the first-level system architecture, the linear mapping enables the corresponding temperature of each pixel on the image sensor to generate standard input values that satisfy the subsequent Chebyshev polynomial calculations. The raw material source of the input values is carried out in a relatively independent pipeline operation in the FPGA, so that the subsequent compensation calculation can make full use of the minimum and maximum error characteristics of the Chebyshev polynomial within the standard range, laying a mathematical foundation for high-precision consistency compensation across the entire temperature range. By solidifying the recursive computation of the polynomial, a pipeline architecture of hardware of the same order as the preset order is formed. The output values of each term corresponding to the input value are calculated in a pipeline. The pipeline architecture is constructed by utilizing the recursive relationship of the Chebyshev polynomial itself. A multi-level recursive pipeline hardware computing unit is formed to perfectly match its recursive characteristics. The high-order computation is transformed into fixed linear operations on the low-order terms, which improves the data throughput. Furthermore, in the second-level system architecture, the high-order polynomial computation of the Chebyshev polynomial is further decomposed into a regular pipeline architecture, which further improves the data throughput. By accumulating the output value and the product of the pre-stored temperature compensation coefficients, compensation parameters are obtained in real time for temperature compensation. Chebyshev polynomial approximation is used to replace piecewise linear interpolation to build a more powerful compensation model, making the compensation curve smoother and more accurate. Moreover, only one set of temperature coefficients needs to be stored in the FPGA, which greatly saves hardware storage resources compared to the need to store a massive temperature-coefficient pair lookup table in the existing technology.
[0035] By interpolating the temperature of a preset point on the image sensor, the temperature estimate of each pixel on the image sensor is obtained, which accurately reflects the two-dimensional temperature gradient on the surface of the image sensor. This overcomes the spatial limitations of the traditional single-point temperature measurement method and solves the problem of inaccurate edge pixel compensation caused by uneven temperature distribution from the source, realizing a leap from chip-level to pixel-level temperature sensing.
[0036] In some implementations, to map pixel temperatures within a standard range, the linear mapping includes: The input value is obtained by mapping the temperature value of each pixel one by one using a mapping formula. The mapping formula is a linear mapping formula, including but not limited to: ; Where t is the input value, T(x,y) is the temperature value of any pixel, and T max The maximum value of the preset operating temperature range for the image sensor, T min Set the minimum value of the preset operating temperature range for the image sensor.
[0037] The input value is obtained by dividing the difference between twice the pixel temperature value and the sum of the maximum and minimum temperatures by the difference between the maximum and minimum temperatures. The maximum temperature value is the maximum value within the preset operating temperature range of the image sensor, and the minimum temperature value is the minimum value within the preset operating temperature range of the image sensor.
[0038] In addition to the mapping formulas mentioned above, there are other linear mapping formulas that map the temperature values of each pixel within the standard range of the Chebyshev polynomial, all of which fall within the protection scope of this invention.
[0039] In some implementations, such as Figure 5 As shown, the recursive computation of the solidified polynomial forms a hardware pipeline architecture of the same order of magnitude as the preset order, including: Set up arithmetic units of the same order of magnitude as the preset order of the polynomial, and connect each level according to the recursive relationship of the polynomial; The arithmetic units at different levels have the same structure, each including a multiplier and an adder.
[0040] In this embodiment, each stage of the pipeline architecture performs higher-order polynomial operations on the results of the previous two stages of the pipeline architecture to construct the pipeline architecture through the recursive relationship of the Chebyshev polynomial itself. Each stage of the pipeline architecture includes an arithmetic unit with the same structure, which is used to recursively calculate different orders of the Chebyshev polynomial. The arithmetic unit includes a multiplier and an adder to adapt to the recursive formula structure of the polynomial.
[0041] In this embodiment, the Chebyshev polynomial recursive formula is:
[0042] Among them, T n+1 (t), T n-1 (t), T n (t) represents terms of different orders in a polynomial, and n represents the default order of the Chebyshev polynomial.
[0043] In some implementations, the step of pipelined calculation of the output values of each order term corresponding to the input value includes: The arithmetic units at each level perform the same multiplication and addition operation on the data pipeline to obtain the output values of the polynomial terms corresponding to different orders of the polynomial step by step. The calculation results of each level are stored and used as input for the next level of calculation.
[0044] In this embodiment, different levels of pipeline architecture execute polynomial operations of different orders. By combining their recursive relationship with the hardware parallelism of FPGA, parallel and pipelined computation of the Chebyshev polynomial model is achieved, which greatly improves the data throughput.
[0045] In some implementations, the dot product of the output value and the pre-stored temperature compensation coefficient includes: The output value of each order term is multiplied in parallel with the temperature compensation coefficient corresponding to its order.
[0046] In this embodiment, different temperature compensation coefficients correspond to different polynomial terms of different orders. Parallel multiplication means that the process of multiplying the output value of each polynomial term of different orders with its corresponding temperature compensation coefficient is calculated in parallel in different computing units within the FPGA, so as to improve the accumulation speed of compensation parameters and ensure real-time performance.
[0047] In some embodiments, the temperature compensation coefficient is obtained through calibration, the calibration including: The image sensor is calibrated with a number of compensation parameters of the same preset order within its operating temperature range, in order to fit a temperature compensation coefficient that optimally matches the Chebyshev polynomial with the calibration data.
[0048] In this embodiment, the specific calibration process includes: In a laboratory environment, the ideal gain and black level values of the image sensor at different temperatures were measured within its preset operating temperature range. These values were used as calibration compensation parameters. The measured calibration temperature values were mapped to the interval [-1, 1]. The corresponding calibration compensation parameters were considered as the values of the final compensation parameter objective function f(t). Using the Chebyshev fitting algorithm, the temperature compensation coefficients c0, c1, c2, ..., c... were solved to make the Chebyshev polynomial best approximate the objective function f(t). n The coefficients c used to determine the gain compensation function Gain(T) and the black level compensation function BlackLevel(T) respectively. n gain and c n black, where Gain(T) represents the gain compensation function and BlackLevel(T) represents the black level compensation function.
[0049] In some implementations, the temperature compensation coefficient includes a gain compensation coefficient and a black level compensation coefficient.
[0050] In this embodiment, the gain compensation coefficient is c. n gain, black level compensation coefficient is c n The black values were all obtained through prior calibration and fitting.
[0051] In some implementations, the preset point temperature is obtained by a temperature sensor located at the edge of the image sensor.
[0052] In this embodiment, in order to obtain more accurate pixel mapping temperature, the temperature sensor is placed at the edge of the image sensor so that the edge temperature obtained after interpolation is more accurate. Furthermore, placing the temperature sensor at the edge of the image sensor is more conducive to the spatial layout of the temperature sensor and will not affect the image acquisition of the image sensor.
[0053] In this embodiment, obtaining the preset point temperature of the image sensor also includes acquisition methods other than temperature sensors, such as thermal pixel analysis based on dark current, temperature estimation based on active pixel regions, etc., to obtain the temperature of the preset point or preset region.
[0054] In this embodiment, the position and number of preset points on the image sensor can be set arbitrarily. Generally, the more preset points there are and the more uniformly the preset point positions are distributed, the higher the interpolation accuracy of the pixel temperature. Example
[0055] The overall hardware architecture in this embodiment is as follows: Figure 4 The diagram illustrates the data flow and core module functions of a three-tiered pipeline system architecture, specifically: Level 1: Linear Mapping The partitioned temperature interpolation T(x,y) calculates a unique temperature estimate for each pixel P(x,y) in the image array, accurately reflecting the two-dimensional temperature gradient on the surface of the image sensor. It overcomes the spatial limitations of the traditional single-point temperature measurement method, solves the problem of edge pixel compensation inaccuracy caused by uneven temperature distribution from the source, and realizes the leap from chip-level to pixel-level temperature sensing. By linearly mapping the calculated actual temperature T(x, y) to the ideal domain [-1, 1] of the Chebyshev polynomial, we obtain the variable t, as shown in the formula:
[0056] Among them, T max The maximum value of the preset operating temperature range for the image sensor, T min Set the minimum value of the preset operating temperature range for the image sensor.
[0057] This embodiment obtains the variable t by linearly mapping the actual working temperature T to the optimal definition interval [-1, 1] of the Chebyshev polynomial. This allows subsequent compensation calculations to fully utilize the minimum and maximum error characteristics of the Chebyshev polynomial within the standard interval, laying a mathematical foundation for high-precision and consistent compensation across the entire temperature range.
[0058] Level 2: Solidification Calculation To further achieve efficient computing, such as Figure 5 As shown, a pipelined architecture is further designed in this level of system architecture, which utilizes the recursive relationship of Chebyshev polynomials: To build an assembly line.
[0059] Among them, T n+1 (t), T n-1 (t), T n (t) represents terms of different orders in a polynomial, and n represents the default order of the Chebyshev polynomial.
[0060] like Figure 5 As shown, the computation process is parallel and pipelined. Within the same clock cycle, the results of the previous stage register are used to synchronously advance the calculation of higher-order terms through a fixed array of multipliers and adders. By utilizing the recursive relationship of polynomials, the calculation of higher-order terms is transformed into fixed linear operations on lower-order terms.
[0061] This recursive structure is extremely suitable for FPGA hardware implementation. It can construct a regular multiplier-adder array to achieve parallel pipelined computation. Compared with the traditional lookup table (LUT) interpolation method, it avoids the contradiction between accuracy and storage resources. The computation path is fixed, the latency is lower, and the resource utilization is higher, which greatly improves the utilization of hardware resources and computational throughput. It is very suitable for FPGA implementation and realizes real-time high-precision temperature compensation.
[0062] Level 3: Cumulative Correction The function of the compensation parameter can be expressed as:
[0063] Where Gain(T) represents the gain compensation function, BlackLevel(T) represents the black level compensation function, and c n gain and c n black is the polynomial coefficient obtained through prior calibration and fitting, which is stored in the FPGA's memory space.
[0064] Specifically, calibration fitting includes: In a laboratory environment, the preset operating temperature range of the image sensor was traversed, and its ideal gain and black level values at different temperatures were measured as calibration compensation parameters.
[0065] Using a numerical fitting tool, based on the measurement calibration data, Chebyshev fitting is performed to map the measured calibration temperature values to the interval [-1, 1]. The corresponding calibration compensation parameters are considered as the values of the final compensation parameter objective function f(t). Using the Chebyshev fitting algorithm, the temperature compensation coefficients that best approximate the Chebyshev polynomial to the objective function f(t) are solved: c0, c1, c2, ..., c n The coefficients c of the gain compensation function Gain(T) and the black level compensation function BlackLevel(T) are used to determine these functions, respectively. n gain and c n black.
[0066] The compensation coefficient synthesis formula calculates the Chebyshev polynomials T for each term. n (t) and the pre-calibrated coefficient c n By performing dot product summation, the final gain and black level compensation parameters are obtained. Chebyshev polynomial approximation is used to replace piecewise linear interpolation to build a more powerful compensation model, which makes the compensation curve smoother and more accurate. Moreover, only one set of temperature coefficients needs to be stored in the FPGA, which greatly saves hardware storage resources compared to the need to store a massive number of temperature-coefficient pairs in the existing technology.
[0067] Temperature compensation is performed using compensation parameters; the specific formula is as follows:
[0068] Among them, Pixel corrected To compensate for the pixel value, Pixel raw These are the original pixel values.
[0069] The pixel correction formula applies the compensation coefficient to each original pixel in real time to complete the correction. The entire calculation is completed in a dedicated hardware pipeline, and one corrected pixel can be output every clock cycle, realizing true real-time processing and completely eliminating the CPU overhead and latency problems of software compensation schemes.
[0070] The complete hardware implementation steps and details, based on the FPGA three-stage pipeline architecture, are as follows: 1. System initialization (calibration and programming) In a laboratory environment, the ideal gain and black level values of the image sensor were measured across its entire operating temperature range at different temperatures. Using numerical fitting tools and based on measurement data, the coefficients c of the gain compensation function Gain(T) and the black level compensation function BlackLevel(T) were determined using the Chebyshev fitting algorithm. n gain and c n black.
[0071] These coefficients are compiled into an initialization file, which is then burned into the specified BRAM during FPGA project synthesis.
[0072] 2. Real-time compensation pipeline operation Level 1: Temperature Interpolation and Mapping: Input readings from multiple temperature sensors and the current pixel coordinates (x, y); the coordinate generator produces coordinates, the temperature sensor interface synchronously reads data, and the bilinear interpolation unit (composed of a multiply-accumulate array) calculates the pixel temperature T(x, y) in real time based on the coordinates and sensor values. Subsequently, the mapping unit executes t=[2×T(x, y)-(T max +T min )] / (T max -T min Perform the operation; output the mapped variable t.
[0073] Second stage: Parallel computation of Chebyshev terms: Input the t value output from the first stage; feed the t value into the Chebyshev recursive computation core, and according to the recursive formula, the computation is pipelined in consecutive clock cycles, with each computation being performed in parallel on the hardware; output the polynomial terms T0(t), T1(t), T2(t), ..., T n (t).
[0074] Level 3: Coefficient Synthesis and Pixel Correction: Input the outputs of Level 2, T_n(t), pre-stored coefficients c_n, and the original pixel value Pixel. raw The dot multiplication adder executes in parallel. and Coefficient synthesis, On the other path, the raw pixel values are read from the line buffer, and the correction arithmetic unit (one multiplier and one subtractor) is executed: Output the corrected pixel values.
[0075] The entire process is carried out in a pipeline. Once the pipeline is full, precise temperature compensation for the number of parallel pixels can be completed in each clock cycle, perfectly meeting the real-time processing requirements of high-speed image sensors.
[0076] In summary, this invention achieves a leap from chip-level temperature sensing to pixel-level temperature prediction through multi-sensor interpolation, fundamentally solving the compensation error caused by temperature spatial unevenness. The use of Chebyshev polynomials, with their minimum-maximum error characteristics within a given interval, ensures consistent compensation accuracy across the entire temperature range. A three-stage pipeline system architecture optimized for the algorithm decomposes complex two-dimensional interpolation and high-order polynomial calculations into regular pipeline steps, processing a number of pixels in parallel per clock cycle, achieving extremely high data throughput. Simultaneously, it avoids massive per-pixel data storage, requiring only the storage of a small number of global model coefficients, achieving considerable performance improvement at a reasonable hardware cost.
[0077] Based on the same inventive concept, such as Figure 2 As shown, the present invention also provides an image sensor temperature compensation system, comprising: The mapping module linearly maps the temperature of each pixel on the image sensor to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. The solidification module solidifies the recursive calculation of the polynomial to form a hardware pipeline architecture of the same order as the preset order, and calculates the output value of each order term corresponding to the input value in a pipelined manner. The calculation module accumulates the product of the output value and the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different degree of the polynomial. The mapping module, the solidification module, and the calculation module form a three-stage pipeline system architecture in the FPGA. The calculation result of the mapping module serves as the input to the calculation of the solidification module, and the calculation result of the solidification module serves as the input to the calculation module.
[0078] Based on the same inventive concept, the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements an image sensor temperature compensation method as described above.
[0079] Based on the same inventive concept, the present invention finally provides an electronic device, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is used to execute the executable instructions to implement the image sensor temperature compensation method as described above.
[0080] In the description of this specification, references to terms such as "an embodiment," "example," and "specific example" indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A method for temperature compensation of an image sensor, used to compensate for pixel value changes in an image sensor caused by temperature variations, characterized in that, include: The temperature of each pixel on the image sensor is linearly mapped to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. The recursive computation of the polynomial is solidified to form a pipeline architecture of hardware with the same order as the preset order, and the output values of each order term corresponding to the input value are calculated in a pipeline. The output value is summed and multiplied by the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different order of the polynomial.
2. The image sensor temperature compensation method according to claim 1, characterized in that, The linear mapping includes: The input value is obtained by dividing the difference between twice the pixel temperature value and the sum of the maximum and minimum temperatures by the difference between the maximum and minimum temperatures. The maximum temperature value is the maximum value within the preset operating temperature range of the image sensor, and the minimum temperature value is the minimum value within the preset operating temperature range of the image sensor.
3. The image sensor temperature compensation method according to claim 1, characterized in that, The recursive computation of the solidified polynomial forms a pipelined architecture of hardware with the same order of magnitude as the preset order, including: Set up arithmetic units of the same order of magnitude as the preset order of the polynomial, and connect each level according to the recursive relationship of the polynomial; The arithmetic units at different levels have the same structure, each including a multiplier and an adder.
4. The image sensor temperature compensation method according to claim 3, characterized in that, The step of pipelined calculation of the output values of each order term corresponding to the input value includes: The arithmetic units at each level perform the same multiplication and addition operation on the data pipeline to obtain the output values of the polynomial terms corresponding to different orders of the polynomial step by step. The calculation results of each level are stored and used as input for the next level of calculation.
5. A method for temperature compensation of an image sensor according to any one of claims 1-4, characterized in that, The dot product of the output value and the pre-stored temperature compensation coefficient includes: The output value of each order term is multiplied in parallel with the temperature compensation coefficient corresponding to its order.
6. A method for temperature compensation of an image sensor according to any one of claims 1-4, characterized in that, The temperature compensation coefficient is obtained through calibration, and the calibration includes: The image sensor is calibrated with a number of compensation parameters of the same preset order within its operating temperature range, in order to fit a temperature compensation coefficient that optimally matches the Chebyshev polynomial with the calibration data.
7. A method for temperature compensation of an image sensor according to any one of claims 1-4, characterized in that, The temperature compensation coefficient includes a gain compensation coefficient and a black level compensation coefficient.
8. A method for temperature compensation of an image sensor according to any one of claims 1-4, characterized in that, The preset point temperature is obtained by a temperature sensor located at the edge of the image sensor.
9. A temperature compensation system for an image sensor, characterized in that, include: The mapping module linearly maps the temperature of each pixel on the image sensor to generate input values within the standard range of a first-order Chebyshev polynomial of a preset order. The solidification module solidifies the recursive calculation of the polynomial to form a hardware pipeline architecture of the same order as the preset order, and calculates the output value of each order term corresponding to the input value in a pipelined manner. The calculation module accumulates the product of the output value and the pre-stored temperature compensation coefficient to obtain compensation parameters in real time for temperature compensation. The temperature of each pixel is obtained by interpolation of the temperatures of several preset points on the image sensor; The standard interval is [-1, 1]; Each term is a polynomial term corresponding to a different degree of the polynomial. The mapping module, the solidification module, and the calculation module form a three-stage pipeline system architecture in the FPGA. The calculation result of the mapping module serves as the input to the calculation of the solidification module, and the calculation result of the solidification module serves as the input to the calculation module.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements an image sensor temperature compensation method as described in any one of claims 1-8.